Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
Choco.free	100	0	0.166234	0.411776	0.9	1	0.437917	1.06348
Chuffed.free	100	0	0	0.300961	0.9	1	0.432091	1.4357
Concrete.free	100	0.109562	0.9	0.581231	1	1	0.434662	0.747831
G12FD.free	100	0	0.539284	0.495805	1	1	0.442611	0.892712
Gecode.free	100	0	0.235714	0.434368	0.9	1	0.442049	1.01768
HaifaCSP.free	100	0	0	0.336321	0.9	1	0.447955	1.33193
JaCoP.fd	100	0	0.159453	0.413044	0.9	1	0.444695	1.07663
LCG.Glucose.UC.free	100	0	0	0.43	1	1	0.488349	1.1357
LCG.Glucose.free	100	0	0	0.282567	0.9	1	0.429608	1.52038
MZN.Cbc.free	100	0	0.454443	0.500385	1	1	0.468417	0.936113
MZN.CPLEX.free	100	0	0	0.366926	1	1	0.463174	1.26231
MZN.Gurobi.free	100	0	0	0.326295	0.9	1	0.456041	1.39764
MZN.SCIP.free	100	0	0.0140583	0.431279	1	1	0.476795	1.10554
MinisatID.free	100	0	0.380747	0.471968	1	1	0.459282	0.973121
Mistral.free	100	0	0.152299	0.432955	1	1	0.458928	1.05999
OR.Tools.free	100	0	0.481852	0.483913	1	1	0.443082	0.915623
OscaR.CBLS.free	100	0	0.685118	0.534991	1	1	0.459867	0.85958
Picat.CP.fd	100	0.111665	0.9	0.598213	1	1	0.426639	0.713188
Picat.SAT.free	100	0	0.00219681	0.375436	0.9	1	0.455132	1.21228
SICStus.Prolog.fd	100	0	0.644254	0.51297	1	1	0.447546	0.87246
Yuck.free	100	0.0221889	0.9	0.603013	1	1	0.449341	0.74516
iZplus.free	100	0	0	0.314971	0.9	1	0.452849	1.43775

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
Choco.free	94.000	6.000
Chuffed.free	99.000	1.000
Concrete.free	65.000	35.000
G12FD.free	80.000	20.000
Gecode.free	94.000	6.000
HaifaCSP.free	94.000	6.000
iZplus.free	88.000	12.000
JaCoP.fd	89.000	11.000
LCG.Glucose.free	99.000	1.000
LCG.Glucose.UC.free	82.000	18.000
MinisatID.free	86.000	14.000
Mistral.free	82.000	18.000
MZN.Cbc.free	68.000	32.000
MZN.CPLEX.free	84.000	16.000
MZN.Gurobi.free	85.000	15.000
MZN.SCIP.free	76.000	24.000
OR.Tools.free	88.000	12.000
OscaR.CBLS.free	61.000	39.000
Picat.CP.fd	77.000	23.000
Picat.SAT.free	92.000	8.000
SICStus.Prolog.fd	83.000	17.000
Yuck.free	56.000	44.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:

	MZN.Cbc.free	MZN.Gurobi.free
MZN.Cbc.free	-	better
MZN.Gurobi.free	worse	-

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 378 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.