Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
CCEHC2akms 601 0 0.35 18000 18000 17024.1 18000 18000 4052.46 0.238042
CCLS2akms 601 0 0.3 18000 18000 17262.4 18000 18000 3543.81 0.20529
LMHS.2016 601 0 0.01 8.74 120.56 5045.01 18000 18000 7948.25 1.57547
Naps.1.02.ms 601 0 0.01 19.49 136.64 5750.37 18000 18000 8275.99 1.43921
Open.WBO15 601 0 0 3.76 23.29 2321.91 144.27 18000 5928.09 2.55311
Open.WBO16 601 0 0 4.57 28 2441.88 161.34 18000 6060.59 2.48194
Optiriss6 601 0 0 1.64 21.8 3063.83 209.06 18000 6680.95 2.18059
QMaxSAT14 601 0 0.01 3.89 45.16 3684.36 476.63 18000 7159.99 1.94335
QMaxSAT16UC 601 0 0.01 3.03 39.05 3590.04 378.57 18000 7094.67 1.97621
WMaxSatz. 601 0 0.09 18000 18000 15865.9 18000 18000 5746.22 0.362174
WMaxSatz09 601 0 0.09 18000 18000 15865.5 18000 18000 5747.21 0.362245
WPM3.2015.co 601 0 0 1.13 9.14 2231.82 102.77 18000 5826.39 2.6106
ahms.1.70 601 0 0.02 18000 18000 14254.8 18000 18000 7208.91 0.50572
ahms.ls.1.70 601 0 0.01 18000 18000 14170.4 18000 18000 7263.42 0.512576
maxhs.b 601 0 0.01 4.09 31.77 3063.84 335.67 18000 6642.75 2.16811
maxino16.c10 601 0 0 1.07 9.9 2274.31 94.96 18000 5900.04 2.59421
maxino16.dis 601 0 0 0.95 10.39 2964.78 110.91 18000 6605.15 2.22787
mscg2015a 601 0 0 0.92 7.96 2631.86 86.45 18000 6288.43 2.38934
mscg2015b 601 0 0 1.61 8.07 2548.69 103.76 18000 6193.27 2.42998

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
ahms.1.70 21.298 78.702 0.000 0.000 0.000 0.000
ahms.ls.1.70 21.797 78.203 0.000 0.000 0.000 0.000
CCEHC2akms 5.491 94.509 0.000 0.000 0.000 0.000
CCLS2akms 4.160 95.840 0.000 0.000 0.000 0.000
LMHS.2016 72.712 27.288 0.000 0.000 0.000 0.000
maxhs.b 83.527 16.473 0.000 0.000 0.000 0.000
maxino16.c10 87.687 12.313 0.000 0.000 0.000 0.000
maxino16.dis 83.860 16.140 0.000 0.000 0.000 0.000
mscg2015a 85.691 14.309 0.000 0.000 0.000 0.000
mscg2015b 86.190 13.810 0.000 0.000 0.000 0.000
Naps.1.02.ms 68.719 31.281 0.000 0.000 0.000 0.000
Open.WBO15 87.521 12.479 0.000 0.000 0.000 0.000
Open.WBO16 86.855 13.145 0.000 0.000 0.000 0.000
Optiriss6 83.361 16.639 0.000 0.000 0.000 0.000
QMaxSAT14 80.033 19.967 0.000 0.000 0.000 0.000
QMaxSAT16UC 80.532 19.468 0.000 0.000 0.000 0.000
WMaxSatz. 12.146 87.854 0.000 0.000 0.000 0.000
WMaxSatz09 12.146 87.854 0.000 0.000 0.000 0.000
WPM3.2015.co 88.020 11.980 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 4497 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11