Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
clasp1	1167	0	21.565	311.53	560.263	1200	1201	537.312	0.959035
clasp2	1167	0	22.795	281.24	570.703	1200	1201	544.269	0.953681
cryptominisat2011	1167	0	26.105	149.51	452.841	1200	1201	506.447	1.11838
eagleup	1167	0.02	1200	1200	1190.75	1200	1200	104.997	0.0881774
ebglucose	1167	0.03	13.065	166.69	505.491	1200	1201	543.069	1.07434
ebminisat	1167	0	14.245	139.54	481.151	1200	1201	531.157	1.10393
glucose2	1167	0.03	10.17	116.21	463.446	1200	1201	529.717	1.14299
glueminisat	1167	0	11.345	104.43	457.392	1200	1201	528.961	1.15647
gnoveltyp2	1167	0	1200	1200	1187.55	1200	1200	119.734	0.100824
lingeling	1167	0	18.355	91.05	415.695	1200	1201	505.897	1.21699
lrglshr	1167	0	14.8	158.49	521.14	1200	1201	546.983	1.04959
marchrw	1167	0	1200	1200	1130.38	1200	1200	269.188	0.238139
minisatpsm	1167	0.03	11.5	114.9	462.666	1200	1201	532.637	1.15123
mphaseSAT	1167	0	18.085	146.73	460.619	1200	1201	516.686	1.12172
mphaseSAT64	1167	0	21.21	141.52	456.434	1200	1201	515.847	1.13017
mphaseSATm	1167	0	18.195	152.62	467.868	1200	1201	517.871	1.10687
mxc09	1167	0	15.245	167.67	512.106	1200	1201	541.698	1.05779
picosat	1167	0	23.75	339.83	566.77	1200	1201	544.376	0.960488
precosat	1167	0	16.735	130.01	471.7	1200	1201	528.542	1.1205
qutersat	1167	0	23.78	134.7	435.504	1200	1201	502.435	1.15369
rcl	1167	0.04	15.12	209.74	525.668	1200	1201	544.603	1.03602
restartsat	1167	0	12.93	142.58	481.115	1200	1201	528.709	1.09892
sapperlot	1167	0	21.33	216.02	522.333	1200	1201	536.445	1.02702
satime11	1167	0	1200	1200	1107.07	1200	1200	311.698	0.281553
sattime	1167	0	1200	1200	1112.88	1200	1200	304.707	0.273801
sattimep	1167	0	1200	1200	1118.17	1200	1201	289.892	0.259255
sol	1167	0.07	102.975	1081.02	722.906	1200	1200	519.468	0.718583
sparrow	1167	0	1200	1200	1166.68	1200	1200	188.538	0.161602
spear.hw	1167	0	63.14	1200	702.144	1200	1201	541.473	0.77117
spear.sw	1167	0	51.2	1200	707.257	1200	1201	546.732	0.773031
tnm	1167	0	1200	1200	1161.44	1200	1200	207.808	0.178922

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
clasp1	62.639	37.361
clasp2	60.583	39.417
cryptominisat2011	72.494	27.506
eagleup	0.771	99.229
ebglucose	64.953	35.047
ebminisat	68.123	31.877
glucose2	69.066	30.934
glueminisat	69.152	30.848
gnoveltyp2	1.114	98.886
lingeling	73.608	26.392
lrglshr	63.668	36.332
marchrw	6.684	93.316
minisatpsm	68.380	31.620
mphaseSAT	70.523	29.477
mphaseSAT64	70.780	29.220
mphaseSATm	70.523	29.477
mxc09	65.381	34.619
picosat	60.326	39.674
precosat	69.152	30.848
qutersat	73.179	26.821
rcl	63.753	36.247
restartsat	68.638	31.362
sapperlot	64.953	35.047
satime11	8.569	91.431
sattime	7.969	92.031
sattimep	7.883	92.117
sol	51.157	48.843
sparrow	3.171	96.829
spear.hw	48.672	51.328
spear.sw	47.558	52.442
tnm	3.342	96.658

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:

	eagleup	marchrw
eagleup	-	better
marchrw	worse	-

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 18243 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.