Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
bcghostq	1254	0	22.4034	900	534.515	900	900	421.354	0.788292
brareqs	1254	0	0.782825	36.5041	372.389	900	900	425.276	1.14202
cbdepqbf	1254	0	2.57655	900	505.422	900	900	432.451	0.855624
cghostq	1254	0	20.1689	900	533.703	900	900	422.083	0.790857
depqbf	1254	0.007693	8.39419	900	577.206	900	900	418.007	0.72419
dual_ooq13	1254	0	29.0729	900	592.39	900	900	407.648	0.688141
ghostq	1254	0	20.8132	900	559.729	900	900	417.991	0.746774
hiqqer1	1254	0	2.77165	401.957	460.575	900	900	432.051	0.938069
hiqqer3	1254	0	4.16478	315.344	454.903	900	900	428.378	0.94169
ooq13	1254	0.007596	161.58	900	654.448	900	900	385.58	0.589167
pre_dual_ooq13	1254	0	23.4936	900	574.427	900	900	413.34	0.719569
rareqs	1254	0	1.9115	900	489.179	900	900	437.028	0.893391
sqube	1254	0	172.272	900	660.716	900	900	384.596	0.582089
xbdepqbf	1254	0	5.18408	293.351	453.15	900	900	430.226	0.949412

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
bcghostq	44.338	55.662
brareqs	62.600	37.400
cbdepqbf	46.810	53.190
cghostq	44.338	55.662
depqbf	38.517	61.483
dual_ooq13	37.959	62.041
ghostq	41.388	58.612
hiqqer1	52.233	47.767
hiqqer3	53.668	46.332
ooq13	30.144	69.856
pre_dual_ooq13	39.713	60.287
rareqs	47.847	52.153
sqube	29.027	70.973
xbdepqbf	52.951	47.049

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 9762 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.