Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
clasp1 1614 0 0 40.47 842.32 664.355 1200 1201 543.385 0.817914
clasp2 1614 0 0 41.9825 734.415 649.774 1200 1201 544.044 0.837281
cryptominisat2011 1614 0 0 45.2775 892.205 668.57 1200 1201 545.228 0.815515
eagleup 1614 0 0 417.925 1200 902.13 1200 1200 508.803 0.564002
ebglucose 1614 0 0.04 41.7475 1200 720.336 1200 1201 547.994 0.760748
ebminisat 1614 0 0 36.3375 1086.5 675.155 1200 1201 551.183 0.81638
glucose2 1614 0 0.04 28.615 1200 678.497 1200 1201 553.755 0.81615
glueminisat 1614 0 0 32.255 1200 698.474 1200 1201 557.054 0.79753
gnoveltyp2 1614 0 0 187.952 1200 870.547 1200 1200 518.818 0.595968
lingeling 1614 0 0 35.6775 1065.75 667.019 1200 1201 556.459 0.834248
lrglshr 1614 0 0 51.2975 1200 736.569 1200 1201 550.09 0.746827
marchrw 1614 0 0 358.07 1200 882.262 1200 1201 500.572 0.567374
minisatpsm 1614 0 0.03 34.82 1200 706.185 1200 1201 553.969 0.784454
mphaseSAT 1614 0 0 5.4175 95.235 430.679 1200 1201 516.632 1.19958
mphaseSAT64 1614 0 0 7.745 105.94 446.75 1200 1201 520.87 1.16591
mphaseSATm 1614 0 0 5.495 84.93 409.998 1167.88 1201 506.556 1.23551
mxc09 1614 0 0 32.3925 845.27 657.87 1200 1201 549.902 0.835882
picosat 1614 0 0 42.9175 1200 691.522 1200 1201 545.962 0.789508
precosat 1614 0 0 26.5325 1200 673.872 1200 1201 558.832 0.829284
qutersat 1614 0 0 46.62 890.785 666.37 1200 1201 547.648 0.821838
rcl 1614 0 0.04 44.8425 1200 749.209 1200 1201 545.078 0.727539
restartsat 1614 0 0 43.2575 1102.25 680.226 1200 1201 548.194 0.8059
sapperlot 1614 0 0 48.36 1200 705.47 1200 1201 542.694 0.769266
satime11 1614 0 0 36.0075 1200 794.86 1200 1200 540.925 0.68053
sattime 1614 0 0 45.1625 1200 831.131 1200 1200 538.143 0.647483
sattimep 1614 0 0 174.925 1200 870.62 1200 1201 519.307 0.59648
sol 1614 0 0.06 114.693 1200 799.974 1200 1200 523.042 0.653824
sparrow 1614 0 0 18.335 1200 784.524 1200 1200 553.642 0.705705
spear.hw 1614 0 0 169.235 1200 806.39 1200 1201 517.925 0.642275
spear.sw 1614 0 0 336.298 1200 884.227 1200 1201 495.879 0.560805
tnm 1614 0 0 45.735 1200 822.444 1200 1200 539.227 0.655639

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
clasp1 53.470 46.530 0.000 0.000 0.000 0.000
clasp2 54.399 45.601 0.000 0.000 0.000 0.000
cryptominisat2011 51.797 48.203 0.000 0.000 0.000 0.000
eagleup 26.022 73.978 0.000 0.000 0.000 0.000
ebglucose 46.097 53.903 0.000 0.000 0.000 0.000
ebminisat 50.805 49.195 0.000 0.000 0.000 0.000
glucose2 49.628 50.372 0.000 0.000 0.000 0.000
glueminisat 47.026 52.974 0.000 0.000 0.000 0.000
gnoveltyp2 29.988 70.012 0.000 0.000 0.000 0.000
lingeling 50.248 49.752 0.000 0.000 0.000 0.000
lrglshr 44.052 55.948 0.000 0.000 0.000 0.000
marchrw 30.855 69.145 0.000 0.000 0.000 0.000
minisatpsm 46.716 53.284 0.000 0.000 0.000 0.000
mphaseSAT 72.862 27.138 0.000 0.000 0.000 0.000
mphaseSAT64 71.933 28.067 0.000 0.000 0.000 0.000
mphaseSATm 75.279 24.721 0.000 0.000 0.000 0.000
mxc09 53.098 46.902 0.000 0.000 0.000 0.000
picosat 49.628 50.372 0.000 0.000 0.000 0.000
precosat 49.876 50.124 0.000 0.000 0.000 0.000
qutersat 51.115 48.885 0.000 0.000 0.000 0.000
rcl 43.185 56.815 0.000 0.000 0.000 0.000
restartsat 50.558 49.442 0.000 0.000 0.000 0.000
sapperlot 49.009 50.991 0.000 0.000 0.000 0.000
satime11 37.794 62.206 0.000 0.000 0.000 0.000
sattime 33.147 66.853 0.000 0.000 0.000 0.000
sattimep 29.678 70.322 0.000 0.000 0.000 0.000
sol 40.087 59.913 0.000 0.000 0.000 0.000
sparrow 37.237 62.763 0.000 0.000 0.000 0.000
spear.hw 39.591 60.409 0.000 0.000 0.000 0.000
spear.sw 30.979 69.021 0.000 0.000 0.000 0.000
tnm 34.201 65.799 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 26948 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11