Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
SH	9720	0	0	0	0.0654365	0	1	0.189658	2.89836
DH	9720	0	0.176171	0.526597	0.463617	0.733267	1	0.302214	0.651863
S1	9720	0	0.825559	0.92781	0.862848	0.971526	1	0.175589	0.203499
S2	9720	0	0.827812	0.928269	0.869213	0.972663	1	0.164272	0.188989
S3	9720	0	0.823114	0.924814	0.861145	0.971168	1	0.176032	0.204417
S4	9720	0	0.822936	0.928488	0.863616	0.972174	1	0.1728	0.200088
S5	9720	0.0912738	0.957948	0.991551	0.958651	0.999892	1	0.0803643	0.0838305
C1	9720	0	0.824994	0.929119	0.864831	0.972397	1	0.173337	0.200429
C2	9720	0	0.823229	0.92707	0.863586	0.973199	1	0.175737	0.203497
C3	9720	0.0937109	0.945044	0.980686	0.950617	0.995428	1	0.0805107	0.0846931
C4	9720	0.238806	0.946774	0.981723	0.952247	0.996085	1	0.0788972	0.0828537
C5	9720	0	0.911212	0.967027	0.929154	0.989967	1	0.101949	0.109722
C6	9720	0	0.915644	0.968252	0.934383	0.990156	1	0.0920129	0.0984745
MATLS	9720	-1	0.921055	0.97606	0.901995	0.993368	1	0.29003	0.321543
M3	9720	-1	0.9188	0.971976	0.798366	0.996757	1	0.520406	0.651839
M4	9720	-1	0.916371	0.969609	0.794719	0.994387	1	0.521044	0.655634
M3B	9720	-1	0.875342	0.962477	0.66952	0.996041	1	0.677517	1.01195
M4B	9720	-1	0.873767	0.961336	0.671609	0.996559	1	0.672626	1.00151
CS2SA	9720	-1	-1	-1	-0.329502	0.882458	1	0.91136	-2.76587
RLS	9720	0	0.738949	0.888945	0.805317	0.956102	1	0.220471	0.27377
EA	9720	0	0.746093	0.885765	0.811733	0.954539	1	0.208075	0.256334

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	crash
C1	100.000	0.000
C2	100.000	0.000
C3	100.000	0.000
C4	100.000	0.000
C5	100.000	0.000
C6	100.000	0.000
CS2SA	35.350	64.650
DH	100.000	0.000
EA	100.000	0.000
M3	92.582	7.418
M3B	86.193	13.807
M4	92.593	7.407
M4B	86.461	13.539
MATLS	97.901	2.099
RLS	100.000	0.000
S1	100.000	0.000
S2	100.000	0.000
S3	100.000	0.000
S4	100.000	0.000
S5	100.000	0.000
SH	100.000	0.000

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 10587 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.