Choosing an optimizer¶
Optimizers¶
| Optimizer | Config | Approach |
|---|---|---|
BayesianOptimizer |
BayesianConfig |
Models the objective with a Gaussian process and evaluates where the model expects the most improvement. The first choice when evaluations are expensive and the budget is a few hundred evaluations or fewer. Supports range and choice parameters |
CMAES |
CMAESConfig |
Learns the scale of each variable and how variables interact. The first choice for continuous and integer problems with larger budgets; doesn't support choice parameters |
GeneticOptimizer |
GeneticOptimizerConfig |
Selection, crossover and mutation. Handles every kind of parameter, so it's the choice for problems with choice parameters and larger budgets |
RandomSearch |
OptimizerConfig |
Evaluates random points; useful as a baseline |
from hone import CMAES, CMAESConfig
result = CMAES(search_space, CMAESConfig(max_evaluations=5000, seed=0)).optimize(run_simulation)
CMA-ES chooses its own population size from the number of variables and workers, and by default restarts with a larger population when a run converges or stalls. Giving it a budget with max_evaluations is usually the only setting it needs.
Bayesian optimization spends a fraction of a second choosing each candidate, rising to a second or so after a few hundred evaluations. That is time well spent when an evaluation is a simulation or a training run that takes minutes, and wasted on a function that takes microseconds. Its first 2 × (variables + 1) candidates are spread over the search space; after that it proposes one candidate at a time, or one per worker with optimize(workers=n), and population_size sets that batch size directly. benchmarks/results.md compares the optimizers.
choose_optimizer picks one for you, by what the benchmark shows works best for the budget and the kinds of parameter:
from hone import choose_optimizer
optimizer = choose_optimizer(search_space, max_evaluations=150, seed=0) # BayesianOptimizer
result = optimizer.optimize(run_simulation)
| Budget | Parameters | Chosen |
|---|---|---|
| 200 evaluations or fewer | Any, unless a choice parameter has choose_n |
BayesianOptimizer |
| More than 200 | Any choice or integer parameter | GeneticOptimizer |
| More than 200 | Continuous numbers only | CMAES |
Other settings, such as seed, maximize, noise or threshold, are passed on to the chosen optimizer's config. On the benchmark's 180 problem and budget pairs, its pick had a median error 1.37 times the best optimizer's, on geometric mean, and was the best on 146; always using CMA-ES gives 7.9 times the best, and it can't run the 15 problems with choice parameters. Its biggest misses are plain Rastrigin with 1,000 evaluations per dimension and plain ellipsoid with 50 to 100, where the genetic optimizer wins because the variables don't interact, which a rule can't know in advance; on the rotated versions CMA-ES wins.
All four take the same arguments and support optimize(), ask() and tell(). A new optimizer subclasses Optimizer and implements _next_population, which returns each generation's individuals; the base class handles handing out candidates, recording results and failures, tracking the best result, stopping and the Result.
Configuration¶
GeneticOptimizerConfig validates its values when created. It can also be loaded from a JSON or YAML file:
config = GeneticOptimizerConfig.from_file("config.yaml")
Set seed to make a run repeatable. max_evaluations sets a budget: the run stops after that many evaluations, even partway through a generation, which makes runs of different optimizers directly comparable. Given a budget, the genetic optimizer chooses its own population size, about a twentieth of the budget between 16 and 50, and runs until the budget is spent, unless you set population_size or generations yourself. Without a budget they default to 100 and 25.
Parents are chosen by tournament: tournament_size (default 4) individuals are drawn and the fittest wins.
When a choice parameter's best option is found late, the genetic optimizer can settle on another one: once most of the population uses one option, a child with a different option inherits numbers tuned for the wrong one and loses. niche_share=0.5 breeds half of each generation's children within niches of parents that share their choices, so every option keeps tuning its own numbers. On material-selection-10d with 1,000 evaluations per dimension it found the right material in 10 of 10 runs, against 6 of 10 without. It costs some progress on small budgets, so it's off by default, except in a search space with conditions: there each branch is a niche, and 0.5 is the default, since a branch with a better optimum is often harder to find.
Each value of a new individual mutates with probability mutation_rate: by default 1 over the number of variables, and at least 0.1, so in two variables half of them mutate. A range value moves by a random step whose typical size is a fraction of the width of its range. A choice switches to another option, and a fixed value never changes.
The step size adapts as the run goes (adaptive_mutation, on by default). Each individual carries its own step, starting at mutation_scale (default 0.1); a child takes the geometric mean of its parents' steps, varied at random, so individuals whose steps suit the problem's current scale breed more, and steps shrink as the population closes in. Together with the mutation rate, this cut the genetic optimizer's geometric-mean error on the benchmark by half or more at 20 to 1,000 evaluations per dimension (×0.34 to ×0.50), and by 8% at 10. With noise=True both stay as before (a rate of 0.1 and a fixed step), because on the noisy problems they helped at small budgets but cost more at large ones; set adaptive_mutation=True or a mutation_rate to choose otherwise.
A child that would repeat a point already asked for, such as a copy of a parent that no mutation touched, is mutated again, one value at a time, until it is new. About one child in four used to be such a copy, and evaluating it again only spent the budget; avoiding them cut the geometric-mean error on the benchmark by 10% to 18% at 10 to 100 evaluations per dimension, and by 59% at 1,000. In a small integer space, where the points near the best run out, some repeats remain. With noise=True repeats are kept, because each one refines the point's average.
A child's numbers come from blend crossover (blend_crossover, on by default): each range value is drawn from between its two parents' values, and up to blend_alpha (default 0.1) times their distance beyond either, rather than copied from one parent. Choices and fixed values are still copied, each from the first parent with probability crossover_rate. On the benchmark this cut the genetic optimizer's geometric-mean error to ×0.69, ×0.62, ×0.51, ×0.38 and ×0.08 at 10 to 1,000 evaluations per dimension, and from 50 evaluations per dimension up it now beats random search on every problem. It helps with noise too (×0.67 and ×0.84 at 10 and 40 evaluations per dimension on the noisy problems, unchanged at 200), so it stays on with noise=True.
Every setting is listed in the configuration reference.