LLAMA results
All results were produced by using the cross-validation splits in the repository with
10 folds and 1 repetitions.
The best values within a type (i.e., baseline (except for vbs), classif, regr and cluster) and performance measure (i.e., Percentage solved, PAR10, MCP) are colored green. Furthermore, the three best values over all groups within a performance measure are colored pink, the absolute best one is red.
The performance is measured in three different ways.
- Percentage solved records the percentage of problem
instances in the data set for which the selector selected an algorithm that was
able to solve it with runstatus "ok" and the algorithm time plus the feature
computation time was at most the timeout.
- The penalized average runtime score (PAR10) measures the time required to
run on all problem instances. If an instance was solved within the timeout by
the algorithm the selector chose, the
actual runtime is taken. If a timeout occurred, the timeout value was multiplied
by 10.
- The misclassification penalty (mcp) measures the additional time required to run
on all problems if sub-optimal algorithms were used. That is, if an algorithm is
run on a problem instance that is not the best, a performance loss is incurred.
There are no additional penalties or factors for timeouts. The virtual best
solver always has a misclassification penalty of zero.
| algo | model | succ | par10 | mcp |
| baseline | vbs | 0.925 | 1383.196 | 0.000 |
| baseline | singleBest | 0.877 | 2274.311 | 109.418 |
| baseline | singleBestByPar | 0.880 | 2231.818 | 120.835 |
| baseline | singleBestBySuccesses | 0.880 | 2231.818 | 120.835 |
| classif | rpart | 0.894 | 1960.712 | 64.854 |
| classif | randomForest | 0.902 | 1805.768 | 44.686 |
| classif | ksvm | 0.900 | 1839.721 | 51.680 |
| cluster | XMeans | 0.905 | 1771.350 | 64.180 |
| regr | lm | 0.895 | 1945.607 | 76.698 |
| regr | rpart | 0.887 | 2083.447 | 79.787 |
| regr | randomForest | 0.902 | 1816.559 | 55.495 |
The following default feature steps were used for model building:
base
Number of presolved instances: 0
The cost for using the feature steps (adapted for presolving) is: 0
or on average: 0
The feature steps correspond to the following 37 / 37 instance features:
f_1, f_2, f_3, f_4, f_5,
f_6, f_7, f_8, f_9, f_10,
f_11, f_12, f_13, f_14, f_15,
f_16, f_17, f_18, f_19, f_20,
f_21, f_22, f_23, f_24, f_25,
f_26, f_27, f_28, f_29, f_30,
f_31, f_32, f_33, f_34, f_35,
f_36, f_37
Algorithm and Feature Subset Selection
In order to get a better insight of the scenarios, forward selections have been applied to the solvers and features to determine whether small subsets achieve comparable performances. Following this approach, we reduced the number of solvers from 19 to 4, resulting in a PAR10 score of 1638.493 for the reduced model. Analogously, the model that was generated based on 4 of the originally 29 features resulted in a PAR10 score of 1646.627.
Below, you can find the list of the selected features and solvers:
Selected Features:
f_1, f_9, f_10, f_30
Selected Solvers:
Open.WBO16, WMaxSatz., WPM3.2015.co, maxhs.b