Overview of performance values

The following statistics were calculated from the performance values of each algorithm:

obs (number of observations within performance values)
nas (number of NAs, i.e., missing values, within performance values)
min (minimum), mean (arithmetic mean), max (maximum), sd (standard deviation)
qu_1st (1st quartile = lower quartile = 25%-quantile)
med (median = 50%-quantile)
qu_3rd (3rd quartile = upper quartile = 75%-quantile)
coeff_var (coefficient of variation = standard deviation / arithmetic mean)

	obs	min	qu_1st	med	mean	qu_3rd	max	sd	coeff_var
MaxHS	572	0.010687	12.6982	142.842	1346.92	3600	3600	1658.68	1.23146
Open.WBO.g	572	0.011054	9.52032	78.7937	1367.92	3600	3600	1685.66	1.23227
Open.WBO.ms	572	0.010212	27.0548	143.653	1319.87	3600	3600	1648.58	1.24904
Open.WBO.ms.pre	572	0.011531	26.2469	149.399	1412.46	3600	3600	1685.53	1.19333
QMaxSAT2018	572	0.052555	34.7664	3600	1974.5	3600	3600	1721.32	0.871775
UWrMaxSAT	572	0.010753	3.93165	41.1688	1224.76	3600	3600	1665.33	1.35972
maxino2018	572	0.011318	4.21974	86.4189	1349.91	3600	3600	1684	1.24749

Summary of the runstatus per algorithm

The following table summarizes the runstatus of each algorithm over all instances (in %).

	ok	timeout
MaxHS	65.909	34.091
maxino2018	65.035	34.965
Open.WBO.g	65.035	34.965
Open.WBO.ms	66.783	33.217
Open.WBO.ms.pre	63.636	36.364
QMaxSAT2018	48.601	51.399
UWrMaxSAT	67.657	32.343

Dominated Algorithms

Here, you'll find an overview of dominating/dominated algorithms:
None of the algorithms was superior to any of the other.

An algorithm (A) is considered to be superior to an other algorithm (B), if it has at least an equal performance on all instances (compared to B) and if it is better on at least one of them. A missing value is automatically a worse performance. However, instances which could not be solved by either one of the algorithms, were not considered for the dominance relation.

Visualisations

Important note w.r.t. some of the following plots:
If appropriate, we imputed performance values for failed or censored runs. We used max + 0.3 * (max - min), in case of minimization problems, or min - 0.3 * (max - min), in case of maximization problems.
In addition, a small noise is added to the imputed values (except for the cluster matrix, based on correlations, which is shown at the end of this page).

Boxplots of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

## Warning: Removed 1472 rows containing non-finite values (stat_boxplot).

Estimated densitities of performance values

Imputing the performance values of failed or censored runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Estimated cumulative distribution functions of performance values

Imputing the performance values of failed runs (as described in the red note at the beginning of this section):

Discarding the performance values of failed or censored runs:

Scatterplot matrix of the performance values

The figure underneath shows pairwise scatterplots of the performance values.

Imputing the performance values of failed and censored runs (as described in the red note at the beginning of this section):

Clustering algorithms based on their correlations

The following figure shows the correlations of the ranks of the performance values. Per default it will show the correlation coefficient of spearman. Missing values were imputed prior to computing the correlation coefficients. The algorithms are ordered in a way that similar (highly correlated) algorithms are close to each other. Per default the clustering is based on hierarchical clustering, using Ward's method.