Conditional problems
Generated by python -m hone.benchmarks with hone 0.10.0: 8 problems, 10 seeds, budgets of 10, 20, 50, 100 and 1,000 evaluations per dimension.
Overview
Problems on which each optimizer has the lowest median error, at each budget. Ties count for each optimizer in them, and errors below 1e-08 count as equal.
| Evaluations per dimension |
genetic |
bayesian |
random |
cma-es |
| 10 |
2 |
6 |
0 |
0 |
| 20 |
3 |
5 |
0 |
0 |
| 50 |
7 |
0 |
1 |
0 |
| 100 |
7 |
0 |
0 |
1 |
| 1000 |
6 |
0 |
0 |
2 |
10 evaluations per dimension
| Problem |
genetic median |
genetic solved |
bayesian median |
bayesian solved |
random median |
random solved |
cma-es median |
cma-es solved |
| branches-3x2d |
1 |
0% |
1.21e-05 |
70% |
1.01 |
0% |
n/a |
n/a |
| branches-3x2d-flat |
1 |
0% |
0.865 |
0% |
1.01 |
0% |
n/a |
n/a |
| branches-3x5d |
1.29 |
0% |
0.00309 |
0% |
1.59 |
0% |
n/a |
n/a |
| branches-3x5d-flat |
1.11 |
0% |
1.62 |
0% |
1.59 |
0% |
n/a |
n/a |
| nested-9 |
0.263 |
0% |
0.00062 |
0% |
0.413 |
0% |
n/a |
n/a |
| nested-9-flat |
0.26 |
0% |
0.475 |
0% |
0.413 |
0% |
n/a |
n/a |
| layers-7 |
1.02 |
0% |
0.000521 |
10% |
2.28 |
0% |
1.29 |
0% |
| layers-7-flat |
1.04 |
0% |
0.77 |
0% |
2.28 |
0% |
1.29 |
0% |
20 evaluations per dimension
| Problem |
genetic median |
genetic solved |
bayesian median |
bayesian solved |
random median |
random solved |
cma-es median |
cma-es solved |
| branches-3x2d |
0.5 |
10% |
7.8e-06 |
90% |
0.964 |
0% |
n/a |
n/a |
| branches-3x2d-flat |
0.911 |
0% |
0.148 |
0% |
0.964 |
0% |
n/a |
n/a |
| branches-3x5d |
1.02 |
0% |
n/a |
n/a |
1.4 |
0% |
n/a |
n/a |
| branches-3x5d-flat |
1.01 |
0% |
n/a |
n/a |
1.4 |
0% |
n/a |
n/a |
| nested-9 |
0.251 |
0% |
0.000398 |
0% |
0.308 |
0% |
n/a |
n/a |
| nested-9-flat |
0.251 |
0% |
0.373 |
0% |
0.308 |
0% |
n/a |
n/a |
| layers-7 |
0.594 |
0% |
0.000258 |
20% |
1.6 |
0% |
0.725 |
0% |
| layers-7-flat |
0.598 |
0% |
0.501 |
0% |
1.6 |
0% |
0.725 |
0% |
50 evaluations per dimension
| Problem |
genetic median |
genetic solved |
bayesian median |
bayesian solved |
random median |
random solved |
cma-es median |
cma-es solved |
| branches-3x2d |
0.00044 |
10% |
n/a |
n/a |
0.661 |
0% |
n/a |
n/a |
| branches-3x2d-flat |
0.951 |
10% |
n/a |
n/a |
0.661 |
0% |
n/a |
n/a |
| branches-3x5d |
1 |
0% |
n/a |
n/a |
1.27 |
0% |
n/a |
n/a |
| branches-3x5d-flat |
1 |
0% |
n/a |
n/a |
1.27 |
0% |
n/a |
n/a |
| nested-9 |
0.25 |
0% |
n/a |
n/a |
0.289 |
0% |
n/a |
n/a |
| nested-9-flat |
0.25 |
0% |
n/a |
n/a |
0.289 |
0% |
n/a |
n/a |
| layers-7 |
0.00651 |
0% |
n/a |
n/a |
1.11 |
0% |
0.504 |
0% |
| layers-7-flat |
0.503 |
0% |
n/a |
n/a |
1.11 |
0% |
0.504 |
0% |
100 evaluations per dimension
| Problem |
genetic median |
genetic solved |
bayesian median |
bayesian solved |
random median |
random solved |
cma-es median |
cma-es solved |
| branches-3x2d |
3.83e-05 |
70% |
n/a |
n/a |
0.351 |
0% |
n/a |
n/a |
| branches-3x2d-flat |
0.333 |
20% |
n/a |
n/a |
0.351 |
0% |
n/a |
n/a |
| branches-3x5d |
0.00859 |
0% |
n/a |
n/a |
1.15 |
0% |
n/a |
n/a |
| branches-3x5d-flat |
1 |
0% |
n/a |
n/a |
1.15 |
0% |
n/a |
n/a |
| nested-9 |
0.0639 |
0% |
n/a |
n/a |
0.267 |
0% |
n/a |
n/a |
| nested-9-flat |
0.25 |
0% |
n/a |
n/a |
0.267 |
0% |
n/a |
n/a |
| layers-7 |
0.5 |
0% |
n/a |
n/a |
1.07 |
0% |
0.5 |
20% |
| layers-7-flat |
0.00255 |
0% |
n/a |
n/a |
1.07 |
0% |
0.5 |
20% |
1000 evaluations per dimension
| Problem |
genetic median |
genetic solved |
bayesian median |
bayesian solved |
random median |
random solved |
cma-es median |
cma-es solved |
| branches-3x2d |
1.79e-13 |
100% |
n/a |
n/a |
0.0305 |
0% |
n/a |
n/a |
| branches-3x2d-flat |
3.46e-07 |
70% |
n/a |
n/a |
0.0305 |
0% |
n/a |
n/a |
| branches-3x5d |
5.3e-09 |
100% |
n/a |
n/a |
1.06 |
0% |
n/a |
n/a |
| branches-3x5d-flat |
1 |
0% |
n/a |
n/a |
1.06 |
0% |
n/a |
n/a |
| nested-9 |
1.13e-08 |
100% |
n/a |
n/a |
0.254 |
0% |
n/a |
n/a |
| nested-9-flat |
0.25 |
0% |
n/a |
n/a |
0.254 |
0% |
n/a |
n/a |
| layers-7 |
2.74e-08 |
80% |
n/a |
n/a |
0.592 |
0% |
1.05e-15 |
100% |
| layers-7-flat |
1.26e-07 |
80% |
n/a |
n/a |
0.592 |
0% |
1.05e-15 |
100% |
Median error over the runs (best value found minus the optimum); solved means error ≤ 0.0001. n/a: the optimizer does not support the problem's parameters, or isn't run at this budget.