Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
CCEHC2akms	601	0.315	1801	1801	1714.18	1801	1801	365.54	0.213245
CCLS2akms	601	0.297	1801	1801	1737.01	1801	1801	313.893	0.180708
ILP.2015	601	0.009	27.857	1801	1034.73	1801	1801	846.773	0.818356
LMHS.C	601	0.012	8.609	110.246	632.591	1801	1801	787.09	1.24423
LMHS.I	601	0.005	4.967	86.749	574.683	1801	1801	765.622	1.33225
MaxHS	601	0.01	7.74	49.217	429.67	391.833	1801	691.341	1.609
Open.WBO	601	0.001	0.545	12.174	348.622	215.467	1801	648.444	1.86002
Open.WBO.L	601	0.009	3.545	43.757	429.76	388.657	1801	700.285	1.62948
Open.WBO.R	601	0.001	3.911	23.388	304.628	154.005	1801	601.059	1.97309
QMSAT14	601	0.01	3.986	46.243	449.435	470.089	1801	712.695	1.58576
QMSAT15UC	601	0	2.807	34.965	468.247	715.358	1801	724.545	1.54736
WMaxSatz.	601	0.091	1801	1801	1634.18	1801	1801	473.534	0.289769
WMaxSatz09	601	0.089	1801	1801	1634.02	1801	1801	473.523	0.289791
WPM3.2015.co	601	0.002	1.113	9.394	290.401	94.706	1801	599.944	2.06591
ahms.1.55	601	0.013	1801	1801	1709.13	1801	1801	384.293	0.224847
ahms.1.68	601	0.014	1801	1801	1700.76	1801	1801	400.691	0.235595
ahms.ls.1.55	601	0.011	1801	1801	1710.77	1801	1801	382.779	0.223747
ahms.ls.1.68	601	0.011	1801	1801	1700.04	1801	1801	403.036	0.237075
maxino.k16	601	0	0.737	7.807	300.666	97.132	1801	617.814	2.05482
maxino.kdyn	601	0	0.665	7.309	300.942	109.012	1801	618.89	2.05651
msUZK.nopp	601	0.025	108.506	1801	1097.63	1801	1801	808.578	0.736657
msUZK.pp	601	0.034	99.778	1801	1104.76	1801	1801	813.302	0.736179
mscg2015a	601	0	0.922	7.801	313.325	84.961	1801	639.329	2.04047
mscg2015b	601	0.002	1.535	8.094	312.308	97.706	1801	630.777	2.01973
optiriss.def	601	0.001	1.887	22.517	382.239	242.704	1801	676.662	1.77026
optiriss.sel	601	0.002	2.008	116.107	801.972	1801	1801	867.041	1.08114
ratselfax.cnf	601	0.172	1801	1801	1531.95	1801	1801	584.434	0.381497
toysat	601	0.03	791.526	1801	1360.45	1801	1801	715.238	0.525737
toysat_ls	601	0.673	1801	1801	1529.21	1801	1801	571.888	0.373975

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
ahms.1.55	5.491	94.509
ahms.1.68	5.990	94.010
ahms.ls.1.55	5.324	94.676
ahms.ls.1.68	5.990	94.010
CCEHC2akms	5.491	94.509
CCLS2akms	4.160	95.840
ILP.2015	47.088	52.912
LMHS.C	71.547	28.453
LMHS.I	74.542	25.458
MaxHS	81.364	18.636
maxino.k16	87.022	12.978
maxino.kdyn	86.855	13.145
mscg2015a	85.691	14.309
mscg2015b	86.023	13.977
msUZK.nopp	45.923	54.077
msUZK.pp	44.426	55.574
Open.WBO	85.025	14.975
Open.WBO.L	81.032	18.968
Open.WBO.R	87.521	12.479
optiriss.def	82.862	17.138
optiriss.sel	58.403	41.597
QMSAT14	80.033	19.967
QMSAT15UC	79.035	20.965
ratselfax.cnf	20.965	79.035
toysat	30.116	69.884
toysat_ls	21.131	78.869
WMaxSatz.	12.146	87.854
WMaxSatz09	12.146	87.854
WPM3.2015.co	88.020	11.980

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 8526 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.