Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
CryptoMiniSat_Strange.Night2.st_fixed_	296	0.001998	328.722	5000	3347.71	5000	5000	2241.35	0.669517
MPhaseSAT_2011.02.15	296	0.001999	95.4812	5000	3074.28	5000	5000	2301.06	0.748487
PicoSAT_941	296	0.000999	398.757	5000	3318.98	5000	5000	2214.13	0.667112
QuteRSat_2011.05.12_fixed_	296	0.033994	357.553	5000	3368.53	5000	5000	2230.9	0.662277
RestartSAT_B95	296	0.001999	357.771	5000	3353.7	5000	5000	2239.18	0.667676
SAT07referencesolverminisat_SAT2007	296	0.002999	352.099	5000	3326.16	5000	5000	2206.4	0.663347
SAT09referencesolverclasp_1.2.0.SAT09.32	296	0.003999	343.978	4957.31	3089.27	5000	5000	2166.04	0.70115
SApperloT2010_2011.05.15_fixed_	296	0.001999	342.034	5000	3370.59	5000	5000	2236.34	0.663485
Sol_2011.04.04	296	0.06199	36.6904	5000	3195.55	5000	5000	2329.74	0.72906
clasp_2.0.R4092.crafted	296	0.002998	176.913	5000	2997.07	5000	5000	2227.31	0.743164
glucose_2	296	0.006998	339.607	5000	3266.98	5000	5000	2213.57	0.677558
jMiniSat_2011	296	0.12498	816.456	5000	3542.41	5000	5000	2169.38	0.612403
sathys_2011.04.01	296	0.008997	350.969	5000	3486.45	5000	5000	2241.23	0.64284
sattime._2011.03.02	296	0.000999	51.6829	5000	3299.18	5000	5000	2336.21	0.708118
sattime_2011.03.02	296	0.001999	17.0434	5000	3269.91	5000	5000	2341.4	0.716044

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
clasp_2.0.R4092.crafted	49.662	50.338
CryptoMiniSat_Strange.Night2.st_fixed_	36.824	63.176
glucose_2	41.554	58.446
jMiniSat_2011	32.770	67.230
MPhaseSAT_2011.02.15	44.257	55.743
PicoSAT_941	40.541	59.459
QuteRSat_2011.05.12_fixed_	36.824	63.176
RestartSAT_B95	37.500	62.500
SApperloT2010_2011.05.15_fixed_	36.486	63.514
SAT07referencesolverminisat_SAT2007	40.878	59.122
SAT09referencesolverclasp_1.2.0.SAT09.32	50.000	50.000
sathys_2011.04.01	32.095	67.905
sattime_2011.03.02	36.149	63.851
sattime._2011.03.02	35.135	64.865
Sol_2011.04.04	38.851	61.149

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 2695 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.