Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
astar.ec 1179 0 0.01 60.01 7200 4127.84 7200 7200 3484.87 0.844235
astar.ed3 1179 0 0 92.32 7200 4379.83 7200 7200 3432.51 0.783708
astar.comp 1179 0 1.02 17.79 185.79 2674.14 7200 7200 3347.05 1.25164
cpbayes 1179 0 0.12 1.955 85.83 2055.51 5949.65 7200 3062.08 1.48969
ilp.141 1179 0 0 4.005 36.39 1157.54 448.745 7200 2395.07 2.06911
ilp.141.nc 1179 0 0.01 7.06 41.83 1173.84 455.775 7200 2404.06 2.04803
ilp.162 1179 0 0.02 4.44 29.56 1233.13 508.51 7200 2455.57 1.99133
ilp.162.nc 1179 0 0.01 5.725 32.18 1267.88 527.83 7200 2493 1.96627

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout
astar.comp 65.140 0.594 34.266
astar.ec 44.020 0.085 55.895
astar.ed3 40.543 0.170 59.288
cpbayes 75.997 24.003 0.000
ilp.141 87.871 11.790 0.339
ilp.141.nc 87.701 11.790 0.509
ilp.162 87.277 12.638 0.085
ilp.162.nc 87.023 12.807 0.170

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 2646 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11