Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
beans_and_eggs	2000	0.001	13.795	246.216	13732.5	50000	50000	21963.2	1.59936
glucose_kiel	2000	0.001	9.912	143.93	12464.9	4365.97	50000	21285.8	1.70765
maple_glucose	2000	0.001	16.705	268.748	12350.7	3899.07	50000	21038.4	1.70342
maplecms	2000	0.004	21.75	244.946	12876.7	16245.4	50000	21454.3	1.66613
maplecomsps	2000	0.001	16.654	210.084	10213.3	2678.8	50000	19576.9	1.91681
maplecomsps_lrb	2000	0.001	13.866	127.674	10126.8	2154.78	50000	19676.3	1.94299
scavel_sat	2000	0.001	20.532	485.202	16257.7	50000	50000	23035.5	1.4169
splatz	2000	0.001	23.787	273.456	15632.8	50000	50000	22865.1	1.46264

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout	memout	crash	other
beans_and_eggs	73.200	26.450	0.000	0.000	0.350
cominisatps	78.100	21.500	0.050	0.000	0.350
glucose_kiel	75.700	24.050	0.000	0.000	0.250
maple_glucose	76.250	23.750	0.000	0.000	0.000
maplecms	75.000	21.650	0.000	3.250	0.100
maplecomsps	80.550	19.450	0.000	0.000	0.000
maplecomsps_lrb	80.450	19.500	0.000	0.000	0.050
riss5	76.300	23.300	0.000	0.000	0.400
scavel_sat	68.250	31.000	0.000	0.500	0.250
splatz	69.350	27.050	0.000	3.550	0.050

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 4937 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.