Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
clasp1 1362 0 0.11 1200 1200 1041.32 1200 1200 378.518 0.363497
clasp2 1362 0 0.07 1200 1200 1041.81 1200 1201 375.089 0.360034
cryptominisat2011 1362 0 0.13 1200 1200 1135.17 1200 1200 247.705 0.218209
eagleup 1362 0 0 0.23 22.795 428.86 1200 1200 551.428 1.2858
ebglucose 1362 0 1.28 1200 1200 1137.28 1200 1200 244.195 0.214719
ebminisat 1362 0 0.02 1200 1200 1086.71 1200 1200 324.482 0.298591
glucose2 1362 0 0.39 1200 1200 1125.08 1200 1200 269.337 0.239395
glueminisat 1362 0 0.48 1200 1200 1160.49 1200 1200 195.002 0.168034
gnoveltyp2 1362 0 0 0.28 55.1 486.509 1200 1200 559.934 1.15092
lingeling 1362 0 0.5 1200 1200 1126.45 1200 1200 267.977 0.237895
lrglshr 1362 0 0.14 1200 1200 1119.16 1200 1200 276.371 0.246945
marchrw 1362 0 0 310.415 1200 885.711 1200 1201 503.315 0.56826
minisatpsm 1362 0 0.41 1200 1200 1151.84 1200 1200 215.874 0.187417
mphaseSAT 1362 0 0.56 1.22 430.155 602.078 1200 1200 576.15 0.956936
mphaseSAT64 1362 0 0.35 1.2025 356.425 593.353 1200 1200 575.864 0.970525
mphaseSATm 1362 0 0.56 1.24 520.845 605.673 1200 1200 576.153 0.951262
mxc09 1362 0 0.58 1200 1200 1063.6 1200 1201 349.769 0.328854
picosat 1362 0 0.01 1200 1200 1059.43 1200 1200 358.998 0.338859
precosat 1362 0 0.19 1200 1200 1128.55 1200 1200 265.951 0.235656
qutersat 1362 0 0.33 1200 1200 1134 1200 1200 251.087 0.221417
rcl 1362 0 0.33 1200 1200 1155.52 1200 1200 207.564 0.179629
restartsat 1362 0 0.09 1200 1200 1105.41 1200 1200 300.744 0.272064
sapperlot 1362 0 0.56 1200 1200 1077.91 1200 1200 335.223 0.310992
satime11 1362 0 0 0.46 90.825 460.706 1200 1200 534.789 1.1608
sattime 1362 0 0 0.49 179.325 561.255 1200 1201 575.689 1.02572
sattimep 1362 0 0 1.77 1200 668.83 1200 1201 579.45 0.866363
sol 1362 0 0.18 1200 1200 1147.1 1200 1200 230.677 0.201096
sparrow 1362 0 0 0.23 13.39 368.934 1200 1200 524.778 1.42242
spear.hw 1362 0 0.2 1200 1200 1089 1200 1200 317.775 0.291804
spear.sw 1362 0 4.71 1200 1200 1184.01 1200 1200 126.718 0.107024
tnm 1362 0 0 0.3025 42.925 448.966 1200 1200 545.646 1.21534

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
clasp1 16.006 83.994 0.000 0.000 0.000 0.000
clasp2 16.960 83.040 0.000 0.000 0.000 0.000
cryptominisat2011 7.195 92.805 0.000 0.000 0.000 0.000
eagleup 67.254 32.746 0.000 0.000 0.000 0.000
ebglucose 7.122 92.878 0.000 0.000 0.000 0.000
ebminisat 11.968 88.032 0.000 0.000 0.000 0.000
glucose2 7.783 92.217 0.000 0.000 0.000 0.000
glueminisat 4.332 95.668 0.000 0.000 0.000 0.000
gnoveltyp2 64.464 35.536 0.000 0.000 0.000 0.000
lingeling 7.783 92.217 0.000 0.000 0.000 0.000
lrglshr 8.811 91.189 0.000 0.000 0.000 0.000
marchrw 29.515 70.485 0.000 0.000 0.000 0.000
minisatpsm 5.433 94.567 0.000 0.000 0.000 0.000
mphaseSAT 53.671 46.329 0.000 0.000 0.000 0.000
mphaseSAT64 54.479 45.521 0.000 0.000 0.000 0.000
mphaseSATm 53.524 46.476 0.000 0.000 0.000 0.000
mxc09 14.537 85.463 0.000 0.000 0.000 0.000
picosat 14.537 85.463 0.000 0.000 0.000 0.000
precosat 7.489 92.511 0.000 0.000 0.000 0.000
qutersat 7.269 92.731 0.000 0.000 0.000 0.000
rcl 5.140 94.860 0.000 0.000 0.000 0.000
restartsat 9.912 90.088 0.000 0.000 0.000 0.000
sapperlot 12.775 87.225 0.000 0.000 0.000 0.000
satime11 68.796 31.204 0.000 0.000 0.000 0.000
sattime 57.489 42.511 0.000 0.000 0.000 0.000
sattimep 47.210 52.790 0.000 0.000 0.000 0.000
sol 5.507 94.493 0.000 0.000 0.000 0.000
sparrow 73.128 26.872 0.000 0.000 0.000 0.000
spear.hw 12.555 87.445 0.000 0.000 0.000 0.000
spear.sw 1.762 98.238 0.000 0.000 0.000 0.000
tnm 68.062 31.938 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 31020 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11