Overview of performance values

The following statistics were calculated from the performance values of each algorithm:
obs nas min qu_1st med mean qu_3rd max sd coeff_var
AIGSolve 825 0 0 0.14 5 5168.46 18000 18000 8127.47 1.57251
aqua.f3v 825 0 0 0.07 13.99 7493.27 18000 18000 8868.71 1.18356
aqua.s2v 825 0 0 0.07 11.49 7449.54 18000 18000 8861.25 1.1895
aqua.s3o 825 0 0 0.07 15.01 7557.3 18000 18000 8880.78 1.17513
caqe.minisat 825 0 0 0.19 4.41 5451.17 18000 18000 8256.02 1.51454
caqe.picosat 825 0 0 0.19 4.39 5148.1 18000 18000 8116.24 1.57655
depqbf.v1 825 0 0.02 0.21 45.81 8063.03 18000 18000 8944.51 1.10932
depqbf.v2 825 0 0.02 0.21 2.41 4860.7 18000 18000 7977.53 1.64123
depqbf.v3 825 0 0 0.14 9.11 6521.44 18000 18000 8637.14 1.32442
ghostq.cegar 825 0 0.06 0.52 8.2 5253.99 18000 18000 8169.1 1.55484
ghostq.plain 825 0 0.05 0.39 6.95 5623.91 18000 18000 8330.12 1.4812
hiqqer1 825 0 0.03 0.06 1.24 5381.75 18000 18000 8230.07 1.52926
hiqqer1ldsq 825 0 0.02 0.05 1.16 5489.64 18000 18000 8278 1.50793
hiqqer3 825 0 0.03 0.06 1.45 5535.28 18000 18000 8295.1 1.49859
iprover.QBF 825 0 0.04 11.36 18000 10422.9 18000 18000 8876.46 0.851628
iprover.QBF.bloqqer 825 0 0.04 3.19 18000 10942.3 18000 18000 8781.78 0.802555
qesto 825 0 0 0.03 1.4 5320.67 18000 18000 8198.2 1.54082
qestos 825 0 0 0.03 0.68 6507.1 18000 18000 8647.68 1.32896
qsts 825 0 0 0.04 0.56 5722.74 18000 18000 8380.4 1.4644
rareqs 825 0 0 0.02 0.54 4053.54 212.79 18000 7503.13 1.85101
squeezebf.struqs 825 0 0 3.18 18000 10269.9 18000 18000 8900.05 0.866615
struqs.10 825 0 0 1.94 18000 10204.6 18000 18000 8908.96 0.873031
xb.bid.qsts 825 0 0 0.19 3.95 3666.71 131.6 18000 7225.59 1.9706
xb.qsts 825 0 0 0.18 3.12 4644 18000 18000 7859.44 1.69239

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

ok timeout memout not_applicable crash other
AIGSolve 71.394 28.606 0.000 0.000 0.000 0.000
aqua.f3v 58.424 41.576 0.000 0.000 0.000 0.000
aqua.s2v 58.667 41.333 0.000 0.000 0.000 0.000
aqua.s3o 58.061 41.939 0.000 0.000 0.000 0.000
caqe.minisat 69.818 30.182 0.000 0.000 0.000 0.000
caqe.picosat 71.515 28.485 0.000 0.000 0.000 0.000
depqbf.v1 55.273 44.727 0.000 0.000 0.000 0.000
depqbf.v2 73.091 26.909 0.000 0.000 0.000 0.000
depqbf.v3 63.879 36.121 0.000 0.000 0.000 0.000
ghostq.cegar 70.909 29.091 0.000 0.000 0.000 0.000
ghostq.plain 68.848 31.152 0.000 0.000 0.000 0.000
hiqqer1 70.182 29.818 0.000 0.000 0.000 0.000
hiqqer1ldsq 69.576 30.424 0.000 0.000 0.000 0.000
hiqqer3 69.333 30.667 0.000 0.000 0.000 0.000
iprover.QBF 42.182 57.818 0.000 0.000 0.000 0.000
iprover.QBF.bloqqer 39.273 60.727 0.000 0.000 0.000 0.000
qesto 70.545 29.455 0.000 0.000 0.000 0.000
qestos 63.879 36.121 0.000 0.000 0.000 0.000
qsts 68.242 31.758 0.000 0.000 0.000 0.000
rareqs 77.576 22.424 0.000 0.000 0.000 0.000
squeezebf.struqs 43.030 56.970 0.000 0.000 0.000 0.000
struqs.10 43.394 56.606 0.000 0.000 0.000 0.000
xb.bid.qsts 79.758 20.242 0.000 0.000 0.000 0.000
xb.qsts 74.303 25.697 0.000 0.000 0.000 0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.


Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-4

Discarding the performance values of failed or censored runs:
## Warning: Removed 7168 rows containing non-finite values (stat_boxplot).
plot of chunk unnamed-chunk-5

Estimated densitities of performance values


Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-6

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-7

Estimated cumulative distribution functions of performance values


Imputing the performance values of failed runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-8

Discarding the performance values of failed or censored runs:
plot of chunk unnamed-chunk-9

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):
plot of chunk unnamed-chunk-10

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.

plot of chunk unnamed-chunk-11