Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
CCEHC2akms 630 0 0.01 18000 18000 17013.7 18000 18000 4070.73 0.239261
CCLS2akms 630 0 0.01 18000 18000 17291.9 18000 18000 3487.03 0.201657
LMHS.2016 630 0 0 1.88 124.675 6118.08 18000 18000 8413.72 1.37522
Naps.1.02.ms 630 0 0 18000 18000 14397.9 18000 18000 7175.56 0.498374
Open.WBO16 630 0 0 1.745 7.725 6737.64 18000 18000 8694.91 1.2905
Optiriss6 630 0 0 8.5025 18000 10289.3 18000 18000 8882.83 0.86331
QMaxSAT14 630 0 0 35.965 1049.35 8434.22 18000 18000 8874.5 1.0522
QMaxSAT16UC 630 0 0 35.025 940.61 8349.72 18000 18000 8867.36 1.06199
WMaxSatz. 630 0 0.01 18000 18000 15801.2 18000 18000 5855.84 0.370595
WMaxSatz09 630 0 0.01 18000 18000 15826.8 18000 18000 5830.12 0.36837
WPM3.2015.co 630 0 0 3.3525 13.98 6914.71 18000 18000 8733.37 1.26301
ahms.1.70 630 0 0 18000 18000 16484.5 18000 18000 4955.78 0.300633
ahms.ls.1.70 630 0 0 18000 18000 16564.3 18000 18000 4842.8 0.292363
maxhs.b 630 0 0 1.9125 59.95 3892.78 779.135 18000 7273.97 1.86858
maxino16.c10 630 0 0 0.97 21.285 6124.67 18000 18000 8495.27 1.38706
maxino16.dis 630 0 0 1.03 7.54 6054 18000 18000 8484.68 1.4015
mscg2015a 630 0 0 0.88 6.36 6228.38 18000 18000 8540.72 1.37126
mscg2015b 630 0 0 1.2225 15.635 5902.53 18000 18000 8410.8 1.42495

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
ahms.1.70 8.571 91.429 0.000 0.000 0.000 0.000
ahms.ls.1.70 8.095 91.905 0.000 0.000 0.000 0.000
CCEHC2akms 5.556 94.444 0.000 0.000 0.000 0.000
CCLS2akms 3.968 96.032 0.000 0.000 0.000 0.000
LMHS.2016 66.667 33.333 0.000 0.000 0.000 0.000
maxhs.b 79.048 20.952 0.000 0.000 0.000 0.000
maxino16.c10 66.190 33.810 0.000 0.000 0.000 0.000
maxino16.dis 66.508 33.492 0.000 0.000 0.000 0.000
mscg2015a 65.556 34.444 0.000 0.000 0.000 0.000
mscg2015b 67.460 32.540 0.000 0.000 0.000 0.000
Naps.1.02.ms 20.159 79.841 0.000 0.000 0.000 0.000
Open.WBO16 62.698 37.302 0.000 0.000 0.000 0.000
Optiriss6 43.016 56.984 0.000 0.000 0.000 0.000
QMaxSAT14 53.810 46.190 0.000 0.000 0.000 0.000
QMaxSAT16UC 54.286 45.714 0.000 0.000 0.000 0.000
WMaxSatz. 12.381 87.619 0.000 0.000 0.000 0.000
WMaxSatz09 12.222 87.778 0.000 0.000 0.000 0.000
WPM3.2015.co 61.746 38.254 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
CCEHC2akms CCLS2akms Open.WBO16 Optiriss6 QMaxSAT14 QMaxSAT16UC WPM3.2015.co ahms.ls.1.70 maxino16.c10 maxino16.dis mscg2015a mscg2015b
CCEHC2akms - - better better better better better - better better better better
CCLS2akms - - better better better better better - better better better better
Open.WBO16 worse worse - - - - - - - - - -
Optiriss6 worse worse - - - - - - - - - -
QMaxSAT14 worse worse - - - - - - - - - -
QMaxSAT16UC worse worse - - - - - - - - - -
WPM3.2015.co worse worse - - - - - - - - - -
ahms.ls.1.70 - - - - - - - - better better - -
maxino16.c10 worse worse - - - - - worse - - - -
maxino16.dis worse worse - - - - - worse - - - -
mscg2015a worse worse - - - - - - - - - -
mscg2015b worse worse - - - - - - - - - -


An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 6565 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11