Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
Choco.free	100	0.497	99.0828	12000	6445.17	12000	12000	5931.15	0.920247
Chuffed.free	100	0.075	0.84025	26.635	3992.48	12000	12000	5649.08	1.41493
Concrete.free	100	4.07	12000	12000	9370.28	12000	12000	4976.72	0.531117
G12FD.free	100	0.127	183.705	12000	7990.27	12000	12000	5617.92	0.703095
Gecode.free	100	14.492	172.587	12000	6357.34	12000	12000	5905.26	0.928889
HaifaCSP.free	100	0.036	4.416	54.452	4127.87	12000	12000	5680.24	1.37607
JaCoP.fd	100	0.24	117.799	12000	6222.56	12000	12000	5926.85	0.952478
LCG.Glucose.UC.free	100	0.043	0.4595	5.0825	3372.45	12000	12000	5407.46	1.60342
LCG.Glucose.free	100	0.036	0.5815	7.3735	3388.72	12000	12000	5397.84	1.59289
MZN.Cbc.free	100	0.034	40.5495	12000	6780.48	12000	12000	5919.99	0.873093
MZN.CPLEX.free	100	0.044	1.86775	65.7735	4724.19	12000	12000	5848.32	1.23795
MZN.Gurobi.free	100	0.036	1.6665	37.2935	4135.42	12000	12000	5675.89	1.37251
MZN.SCIP.free	100	0.044	13.572	236.918	4983.24	12000	12000	5880.97	1.18015
MinisatID.free	100	0.229	18.334	12000	6177.92	12000	12000	5971.3	0.966556
Mistral.free	100	0.126	19.9522	12000	7706.25	12000	12000	5754.9	0.746784
OR.Tools.free	100	0.052	265.041	12000	7416.84	12000	12000	5763.14	0.777034
Picat.CP.fd	100	18.35	12000	12000	9895.58	12000	12000	4515.87	0.456352
Picat.SAT.free	100	0.587	8.54725	63.644	4235.94	12000	12000	5727.06	1.35202
SICStus.Prolog.fd	100	0.495	178.082	12000	7188.88	12000	12000	5803.39	0.807273
iZplus.free	100	0.108	9.597	12000	6543.61	12000	12000	5944.81	0.90849

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
Choco.free	47.000	53.000
Chuffed.free	67.000	33.000
Concrete.free	22.000	78.000
G12FD.free	34.000	66.000
Gecode.free	48.000	52.000
HaifaCSP.free	66.000	34.000
iZplus.free	46.000	54.000
JaCoP.fd	49.000	51.000
LCG.Glucose.free	72.000	28.000
LCG.Glucose.UC.free	72.000	28.000
MinisatID.free	49.000	51.000
Mistral.free	36.000	64.000
MZN.Cbc.free	44.000	56.000
MZN.CPLEX.free	61.000	39.000
MZN.Gurobi.free	66.000	34.000
MZN.SCIP.free	59.000	41.000
OR.Tools.free	39.000	61.000
Picat.CP.fd	18.000	82.000
Picat.SAT.free	65.000	35.000
SICStus.Prolog.fd	41.000	59.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:

	Picat.CP.fd	Picat.SAT.free
Picat.CP.fd	-	better
Picat.SAT.free	worse	-

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 999 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.