Skip to content

Full suite

Generated by python -m hone.benchmarks with hone 0.8.2: 36 problems, 10 seeds, budgets of 10, 20, 50, 100 and 1,000 evaluations per dimension.

Overview

Problems on which each optimizer has the lowest median error, at each budget. Ties count for each optimizer in them, and errors below 1e-08 count as equal.

Evaluations per dimension genetic cma-es bayesian random
10 3 4 29 0
20 5 3 30 0
50 16 14 10 0
100 14 18 8 1
1000 12 28 0 1

10 evaluations per dimension

Problem genetic median genetic solved cma-es median cma-es solved bayesian median bayesian solved random median random solved
sphere-2d 0.962 0% 0.153 0% 8.97e-05 50% 1.02 0%
ellipsoid-2d 2.91e+04 0% 4.36e+03 0% 4.3 0% 2.37e+04 0%
rosenbrock-2d 9.51 0% 7.61 0% 2.88 0% 44.2 0%
rastrigin-2d 10.7 0% 6.18 0% 11 0% 10.7 0%
ackley-2d 4 0% 2.8 0% 0.792 0% 4.39 0%
sphere-5d 4.32 0% 2.11 0% 0.00137 0% 8.06 0%
ellipsoid-5d 1.38e+04 0% 2.35e+04 0% 111 0% 9.61e+04 0%
rosenbrock-5d 646 0% 471 0% 368 0% 2.55e+03 0%
rastrigin-5d 31.7 0% 37 0% 20.3 0% 50.8 0%
ackley-5d 4.42 0% 4.51 0% 2.24 0% 6.6 0%
sphere-10d 6.3 0% 8.52 0% 0.00582 0% 33.8 0%
ellipsoid-10d 6.72e+04 0% 2.35e+05 0% 666 0% 3.62e+05 0%
rosenbrock-10d 2.86e+03 0% 2.12e+03 0% 919 0% 2.59e+04 0%
rastrigin-10d 70.6 0% 79 0% 64.4 0% 116 0%
ackley-10d 4.48 0% 4.81 0% 2.58 0% 7.4 0%
rotated-ellipsoid-2d 3.62e+04 0% 1.32e+04 0% 126 0% 4.25e+04 0%
rotated-rosenbrock-2d 8.88 0% 7.19 0% 8.73 0% 14 0%
rotated-rastrigin-2d 9.22 0% 6.68 0% 8.81 0% 9.57 0%
rotated-ackley-2d 3.4 0% 2.84 0% 0.51 0% 3.69 0%
rotated-ellipsoid-5d 2.58e+04 0% 5.73e+04 0% 4.89e+04 0% 3.03e+04 0%
rotated-rosenbrock-5d 333 0% 225 0% 291 0% 2.41e+03 0%
rotated-rastrigin-5d 32.1 0% 31.9 0% 29.4 0% 45.9 0%
rotated-ackley-5d 3.74 0% 5.1 0% 2.44 0% 6.05 0%
rotated-ellipsoid-10d 5.52e+04 0% 1.63e+05 0% 1.96e+05 0% 3.7e+05 0%
rotated-rosenbrock-10d 2.19e+03 0% 1.93e+03 0% 1.39e+03 0% 1.91e+04 0%
rotated-rastrigin-10d 71.8 0% 84.5 0% 59.4 0% 110 0%
rotated-ackley-10d 4.7 0% 5.12 0% 2.51 0% 7.59 0%
integer-sphere-2d 1 20% 0.5 50% 0 100% 1 10%
mixed-rastrigin-2d 5.01 0% 3.8 0% 2.75 0% 5.05 0%
material-selection-2d 1.24 0% n/a n/a 0.00683 0% 1.17 0%
integer-sphere-5d 2.5 10% 3 0% 0 100% 11 0%
mixed-rastrigin-5d 16.1 0% 17 0% 16 0% 32.6 0%
material-selection-5d 3.61 0% n/a n/a 0.32 0% 9.24 0%
integer-sphere-10d 10.5 0% 11 0% 0 100% 37.5 0%
mixed-rastrigin-10d 36.5 0% 42.5 0% 37.5 0% 70.3 0%
material-selection-10d 11.3 0% n/a n/a 1.01 0% 26.4 0%

20 evaluations per dimension

Problem genetic median genetic solved cma-es median cma-es solved bayesian median bayesian solved random median random solved
sphere-2d 0.126 0% 0.0706 0% 5.44e-05 90% 0.277 0%
ellipsoid-2d 922 0% 813 0% 2.35 0% 5.71e+03 0%
rosenbrock-2d 2 0% 3.63 0% 0.0915 0% 8.12 0%
rastrigin-2d 5.05 0% 3.34 0% 1.76 0% 9.29 0%
ackley-2d 2.55 0% 1.98 0% 0.255 0% 3.76 0%
sphere-5d 0.921 0% 0.624 0% 0.000569 0% 5.84 0%
ellipsoid-5d 2.14e+03 0% 1.25e+04 0% 24.8 0% 6.81e+04 0%
rosenbrock-5d 66.8 0% 150 0% 41.7 0% 1e+03 0%
rastrigin-5d 20.8 0% 25.4 0% 13.2 0% 41 0%
ackley-5d 2.39 0% 2.9 0% 2.22 0% 6.37 0%
sphere-10d 1.85 0% 1.83 0% 0.00305 0% 31 0%
ellipsoid-10d 2.55e+04 0% 6.6e+04 0% 103 0% 2.7e+05 0%
rosenbrock-10d 487 0% 364 0% 198 0% 1.95e+04 0%
rastrigin-10d 46.1 0% 63.1 0% 58.4 0% 114 0%
ackley-10d 3.47 0% 3.38 0% 2.54 0% 7.09 0%
rotated-ellipsoid-2d 502 0% 350 0% 3.75 0% 3.39e+03 0%
rotated-rosenbrock-2d 2.28 0% 2.06 0% 1.41 0% 2.78 0%
rotated-rastrigin-2d 3.82 0% 4.69 0% 3.19 0% 8.34 0%
rotated-ackley-2d 2 0% 2.23 0% 0.293 0% 2.61 0%
rotated-ellipsoid-5d 5.2e+03 0% 1.61e+04 0% 4.71e+03 0% 3.03e+04 0%
rotated-rosenbrock-5d 135 0% 87.2 0% 135 0% 1.15e+03 0%
rotated-rastrigin-5d 16.5 0% 27.3 0% 15.2 0% 42.4 0%
rotated-ackley-5d 2.91 0% 2.89 0% 2.19 0% 5.96 0%
rotated-ellipsoid-10d 2.41e+04 0% 7.14e+04 0% 1.99e+04 0% 3.2e+05 0%
rotated-rosenbrock-10d 447 0% 243 0% 784 0% 7.71e+03 0%
rotated-rastrigin-10d 51.7 0% 65.1 0% 40.9 0% 102 0%
rotated-ackley-10d 3.83 0% 3.41 0% 2.48 0% 7.14 0%
integer-sphere-2d 0 60% 0 60% 0 100% 1 20%
mixed-rastrigin-2d 1.2 0% 1.25 0% 1.01 20% 4.98 0%
material-selection-2d 0.915 0% n/a n/a 3.54e-05 80% 0.875 0%
integer-sphere-5d 0.5 50% 0.5 50% 0 100% 10.5 0%
mixed-rastrigin-5d 10.5 0% 13.2 0% 12.1 0% 21.1 0%
material-selection-5d 1.73 0% n/a n/a 0.000661 0% 6.31 0%
integer-sphere-10d 2 0% 3 0% 0 100% 33.5 0%
mixed-rastrigin-10d 19.3 0% 29.6 0% 30 0% 65.9 0%
material-selection-10d 4.49 0% n/a n/a n/a n/a 25.6 0%

50 evaluations per dimension

Problem genetic median genetic solved cma-es median cma-es solved bayesian median bayesian solved random median random solved
sphere-2d 0.00822 0% 0.000828 10% 3.87e-05 90% 0.184 0%
ellipsoid-2d 9.74 0% 17.3 0% 0.324 0% 867 0%
rosenbrock-2d 0.426 0% 0.473 0% 0.0178 0% 2.68 0%
rastrigin-2d 2.36 0% 2.46 0% 1.43 0% 8.39 0%
ackley-2d 0.192 0% 0.0532 0% 0.0844 0% 2.47 0%
sphere-5d 0.0248 0% 0.00572 0% n/a n/a 4.3 0%
ellipsoid-5d 105 0% 1.88e+03 0% n/a n/a 3.4e+04 0%
rosenbrock-5d 8.91 0% 13.2 0% n/a n/a 623 0%
rastrigin-5d 9.18 0% 16.1 0% n/a n/a 33.8 0%
ackley-5d 0.528 0% 0.316 0% n/a n/a 5.12 0%
sphere-10d 0.104 0% 0.0245 0% n/a n/a 21.8 0%
ellipsoid-10d 2.11e+03 0% 1.95e+04 0% n/a n/a 2.2e+05 0%
rosenbrock-10d 78.7 0% 13.9 0% n/a n/a 1.15e+04 0%
rastrigin-10d 22.7 0% 50.5 0% n/a n/a 93.4 0%
ackley-10d 1.6 0% 0.47 0% n/a n/a 6.72 0%
rotated-ellipsoid-2d 71.1 0% 10.8 0% 0.445 0% 1.31e+03 0%
rotated-rosenbrock-2d 0.0917 0% 0.249 0% 0.0586 0% 0.734 0%
rotated-rastrigin-2d 1.6 0% 2.4 0% 2.02 0% 4.42 0%
rotated-ackley-2d 0.19 0% 0.116 0% 0.078 0% 2.13 0%
rotated-ellipsoid-5d 3.49e+03 0% 3.76e+03 0% n/a n/a 2.22e+04 0%
rotated-rosenbrock-5d 10.7 0% 4.49 0% n/a n/a 503 0%
rotated-rastrigin-5d 9 0% 19.2 0% n/a n/a 32.3 0%
rotated-ackley-5d 0.646 0% 0.215 0% n/a n/a 4.76 0%
rotated-ellipsoid-10d 1.09e+04 0% 9.6e+03 0% n/a n/a 1.87e+05 0%
rotated-rosenbrock-10d 51.3 0% 13.4 0% n/a n/a 4.87e+03 0%
rotated-rastrigin-10d 30.9 0% 53.3 0% n/a n/a 90.6 0%
rotated-ackley-10d 2.36 0% 0.624 0% n/a n/a 6.39 0%
integer-sphere-2d 0 100% 0 100% 0 100% 1 40%
mixed-rastrigin-2d 0.398 0% 0.657 0% 2.85e-07 100% 2.05 0%
material-selection-2d 0.188 10% n/a n/a 5.41e-06 100% 0.875 0%
integer-sphere-5d 0 100% 0 100% n/a n/a 6.5 0%
mixed-rastrigin-5d 2.94 0% 6.14 0% n/a n/a 17 0%
material-selection-5d 1.07 0% n/a n/a n/a n/a 5.07 0%
integer-sphere-10d 0 100% 0 100% n/a n/a 30.5 0%
mixed-rastrigin-10d 6.19 0% 19.2 0% n/a n/a 62.7 0%
material-selection-10d 2.15 0% n/a n/a n/a n/a 21.1 0%

100 evaluations per dimension

Problem genetic median genetic solved cma-es median cma-es solved bayesian median bayesian solved random median random solved
sphere-2d 0.00017 30% 7.61e-07 100% 2.08e-05 100% 0.154 0%
ellipsoid-2d 1.95 0% 1 0% 0.0667 0% 455 0%
rosenbrock-2d 0.156 0% 0.0515 0% 0.00656 0% 1.58 0%
rastrigin-2d 0.833 0% 1.65 0% 0.502 0% 6.45 0%
ackley-2d 0.0594 0% 0.002 0% 0.0236 0% 2.17 0%
sphere-5d 0.0029 0% 2.32e-06 100% n/a n/a 3.54 0%
ellipsoid-5d 14.3 0% 187 0% n/a n/a 2.44e+04 0%
rosenbrock-5d 4.05 0% 3.21 0% n/a n/a 588 0%
rastrigin-5d 5.1 0% 10.2 0% n/a n/a 29.5 0%
ackley-5d 0.123 0% 0.00501 0% n/a n/a 5.12 0%
sphere-10d 0.0302 0% 9.69e-06 100% n/a n/a 20.5 0%
ellipsoid-10d 325 0% 3.74e+03 0% n/a n/a 1.78e+05 0%
rosenbrock-10d 39.2 0% 7.31 0% n/a n/a 7.61e+03 0%
rastrigin-10d 15.4 0% 20.2 0% n/a n/a 88.3 0%
ackley-10d 0.467 0% 0.00751 0% n/a n/a 6.45 0%
rotated-ellipsoid-2d 15.4 0% 5.08 0% 0.275 0% 176 0%
rotated-rosenbrock-2d 0.0687 0% 0.162 0% 0.0116 0% 0.344 0%
rotated-rastrigin-2d 1.19 0% 1.06 0% 0.0259 10% 3.37 0%
rotated-ackley-2d 0.0464 0% 0.0022 0% 0.012 0% 2.06 0%
rotated-ellipsoid-5d 2.56e+03 0% 528 0% n/a n/a 9.2e+03 0%
rotated-rosenbrock-5d 4.65 0% 2.32 0% n/a n/a 338 0%
rotated-rastrigin-5d 6.88 0% 13 0% n/a n/a 29.7 0%
rotated-ackley-5d 0.223 0% 0.00459 0% n/a n/a 4.76 0%
rotated-ellipsoid-10d 5.97e+03 0% 1.58e+03 0% n/a n/a 1.25e+05 0%
rotated-rosenbrock-10d 36.4 0% 8.19 0% n/a n/a 4.84e+03 0%
rotated-rastrigin-10d 24.2 0% 29 0% n/a n/a 84.7 0%
rotated-ackley-10d 0.908 0% 0.00718 0% n/a n/a 6.38 0%
integer-sphere-2d 0 100% 0 100% 0 100% 0 90%
mixed-rastrigin-2d 0.00187 20% 0.00153 30% 9.67e-08 100% 1.25 0%
material-selection-2d 0.000604 20% n/a n/a n/a n/a 0.437 0%
integer-sphere-5d 0 100% 0 100% n/a n/a 6 0%
mixed-rastrigin-5d 1.18 0% 4.03 0% n/a n/a 16.7 0%
material-selection-5d 1 0% n/a n/a n/a n/a 3.98 0%
integer-sphere-10d 0 100% 0 100% n/a n/a 27 0%
mixed-rastrigin-10d 4.4 0% 6.54 0% n/a n/a 58.5 0%
material-selection-10d 0.191 0% n/a n/a n/a n/a 19.9 0%

1000 evaluations per dimension

Problem genetic median genetic solved cma-es median cma-es solved bayesian median bayesian solved random median random solved
sphere-2d 1.48e-08 100% 1.64e-17 100% n/a n/a 0.00477 0%
ellipsoid-2d 3.6e-05 60% 2.64e-17 100% n/a n/a 12.9 0%
rosenbrock-2d 0.000954 30% 8.51e-18 100% n/a n/a 0.223 0%
rastrigin-2d 7.13e-05 70% 0 60% n/a n/a 1.44 0%
ackley-2d 7.76e-05 60% 5.61e-12 100% n/a n/a 0.464 0%
sphere-5d 1.36e-07 100% 6.17e-16 100% n/a n/a 1.62 0%
ellipsoid-5d 0.00922 10% 6.81e-16 100% n/a n/a 5.01e+03 0%
rosenbrock-5d 1.77 0% 5.43e-16 90% n/a n/a 115 0%
rastrigin-5d 0.000173 40% 1.99 0% n/a n/a 19.2 0%
ackley-5d 0.000629 10% 1.41e-11 100% n/a n/a 3.58 0%
sphere-10d 7.85e-07 100% 1.46e-15 100% n/a n/a 12.9 0%
ellipsoid-10d 0.276 0% 2.12e-15 100% n/a n/a 7.39e+04 0%
rosenbrock-10d 2.17 0% 2.81e-15 90% n/a n/a 2.67e+03 0%
rastrigin-10d 0.00144 10% 5.47 0% n/a n/a 78.1 0%
ackley-10d 0.00111 0% 2.19e-11 100% n/a n/a 5.51 0%
rotated-ellipsoid-2d 0.679 0% 1.64e-17 100% n/a n/a 10.3 0%
rotated-rosenbrock-2d 7.46e-05 70% 1.1e-17 100% n/a n/a 0.09 0%
rotated-rastrigin-2d 1.7e-05 90% 0.515 30% n/a n/a 1.41 0%
rotated-ackley-2d 0.000393 30% 6.23e-12 100% n/a n/a 0.447 0%
rotated-ellipsoid-5d 209 0% 4.66e-16 100% n/a n/a 2.02e+03 0%
rotated-rosenbrock-5d 3.05 0% 8.5e-16 100% n/a n/a 149 0%
rotated-rastrigin-5d 4.48 0% 1.99 10% n/a n/a 19.3 0%
rotated-ackley-5d 0.000438 0% 1.29e-11 100% n/a n/a 3.64 0%
rotated-ellipsoid-10d 1.6e+03 0% 1.88e-15 100% n/a n/a 5.18e+04 0%
rotated-rosenbrock-10d 8.38 0% 2.09e-15 100% n/a n/a 3.14e+03 0%
rotated-rastrigin-10d 9.45 0% 6.47 0% n/a n/a 65.9 0%
rotated-ackley-10d 0.00145 0% 2.11e-11 100% n/a n/a 5.58 0%
integer-sphere-2d 0 100% 0 100% n/a n/a 0 100%
mixed-rastrigin-2d 0 100% 0 100% n/a n/a 0.142 0%
material-selection-2d 4.62e-09 100% n/a n/a n/a n/a 0.0163 0%
integer-sphere-5d 0 100% 0 100% n/a n/a 2.5 0%
mixed-rastrigin-5d 5.81e-07 100% 1.1 10% n/a n/a 10.4 0%
material-selection-5d 1 40% n/a n/a n/a n/a 1.98 0%
integer-sphere-10d 0 100% 0 100% n/a n/a 15.5 0%
mixed-rastrigin-10d 1.15e-05 90% 1.99 0% n/a n/a 43.8 0%
material-selection-10d 2.92e-06 60% n/a n/a n/a n/a 12.8 0%

Median error over the runs (best value found minus the optimum); solved means error ≤ 0.0001. n/a: the optimizer does not support the problem's parameters, or isn't run at this budget.