Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
EagleUP_1.565.350 600 0 0.005998 0.802627 384.17 2344.23 5000 5000 2448.82 1.04462
MPhaseSAT_M_2011.02.16 600 0 0.623904 1.9777 360.482 2138.31 5000 5000 2390.04 1.11773
SAT09referencesolverTNM_2009.03.22 600 0 0.004998 1.25931 1820.01 2489.04 5000 5000 2406.37 0.966786
SAT09referencesolvergnovelty.2_2009.03.22 600 0 0.002999 1.47227 5000 2660.03 5000 5000 2434.68 0.915283
SAT09referencesolvermarch_hi_hi 600 0 0.015997 139.673 5000 3134.39 5000 5000 2305.4 0.735519
adaptg2wsat2011_2011.03.02 600 0 0.004998 1.56451 868.514 2414.56 5000 5000 2402.8 0.995126
march_rw_2011.03.02 600 0 0.013997 127.562 5000 3125.33 5000 5000 2315.05 0.740737
sattime2011_2011.03.02 600 0 0.004998 1.70324 1072.05 2416.63 5000 5000 2392.32 0.989942
sparrow2011_sparrow2011_ubcsat1.2_2011.03.02 600 0 0.002998 1.08383 167.271 2066.36 5000 5000 2401.95 1.16241

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
adaptg2wsat2011_2011.03.02 54.667 45.333 0.000 0.000 0.000 0.000
EagleUP_1.565.350 54.667 45.333 0.000 0.000 0.000 0.000
march_rw_2011.03.02 41.833 58.167 0.000 0.000 0.000 0.000
MPhaseSAT_M_2011.02.16 59.833 40.167 0.000 0.000 0.000 0.000
SAT09referencesolvergnovelty.2_2009.03.22 49.667 50.333 0.000 0.000 0.000 0.000
SAT09referencesolvermarch_hi_hi 41.833 58.167 0.000 0.000 0.000 0.000
SAT09referencesolverTNM_2009.03.22 54.000 46.000 0.000 0.000 0.000 0.000
sattime2011_2011.03.02 55.667 44.333 0.000 0.000 0.000 0.000
sparrow2011_sparrow2011_ubcsat1.2_2011.03.02 60.333 39.667 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 2565 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11