Running an optimization

Slow evaluations

When each evaluation takes a while, run them in parallel. workers evaluates that many candidates at a time in background processes; the objective function must be defined at the top level of a module, and on macOS and Windows the calling script needs an if __name__ == "__main__": guard.

result = optimizer.optimize(run_simulation, workers=8, timeout=600)

timeout limits each evaluation, in seconds. An evaluation that runs longer is stopped and counts as failed, as does one whose process crashes, so one hung simulation can't stall the run. A timeout also runs evaluations in a background process when workers is 1.

Each generation waits for its slowest evaluation before the next one starts, so workers beyond the population size sit idle, and so do workers that finish early while one slow evaluation runs on. When evaluation times vary, set min_finished in the config to let the next generation start once that share of the current one has finished:

config = CMAESConfig(max_evaluations=2000, min_finished=0.75)

The slowest quarter of each generation then finishes in the background while the workers move on, and each late result joins the generation under way when it arrives: the genetic optimizer can breed from it, Bayesian optimization adds it to its model, and every optimizer counts it toward the best result. At the end, optimize() waits for the last of them, so the whole budget is used; stopping early by callback or threshold doesn't wait. Because results then depend on which evaluations finish first, runs with min_finished below 1 can't be repeated exactly.

Measured with 8 workers on evaluations of uneven length (one in ten taking five times as long), min_finished=0.75 gets the same number of evaluations done in 23% to 43% less time. Per evaluation it does a little worse for CMA-ES and Bayesian optimization, whose updates prefer complete generations, and better for the genetic optimizer; but in the same time, with the evaluations it saves, its median error was ×0.76 for the genetic optimizer, ×0.25 for CMA-ES and ×0.12 for Bayesian optimization. Lower values, such as 0.5, save little more time and cost more per evaluation.

Workers beyond the population size still sit idle. CMA-ES allows for this: unless you set population_size, it uses at least twice as many candidates per generation as there are workers. Bayesian optimization proposes one candidate per worker at a time. Their results, unlike the other optimizers', therefore depend on the number of workers as well as the seed. For the genetic optimizer and random search, keep population_size at least two or three times workers.

When evaluations run somewhere optimize() can't reach, such as jobs on a compute cluster or tests on a physical rig, ask the optimizer for candidates and tell it the results yourself, in any order:

while not optimizer.done:
    for trial in optimizer.ask():
        submit_job(trial.id, trial.params)
    for job in wait_for_finished_jobs():
        optimizer.tell(job.trial_id, job.value)

result = optimizer.result()

ask() hands out the candidates of the current generation; ask(4) hands out at most four. It returns an empty list while the generation still waits for results, because the next generation is bred from them. result() can be called at any point for the best result so far.

Following a run

callback is called after each generation with a Progress: the generation's statistics, the evaluations so far, the best value and parameters so far, and the seconds elapsed. print_progress prints one line per generation:

from hone import print_progress

result = optimizer.optimize(run_simulation, workers=8, callback=print_progress)
# generation 0: 50 evaluations, best 12.4, 61.3 s

Return True from a callback to stop the run, for a stopping rule of your own:

def good_enough_or_overnight(progress):
    good_enough = progress.best_fitness is not None and progress.best_fitness < 0.5
    return good_enough or progress.elapsed > 8 * 3600

Pressing Ctrl+C stops the run too. Either way, optimize() stops the workers and returns the best result so far, with stopped set.

Resuming a run

A run of days shouldn't be lost to a crash, a reboot or a killed job. Give optimize() a checkpoint file:

result = optimizer.optimize(run_simulation, workers=8, checkpoint="design-study.checkpoint")

After each generation the run's whole state is saved to the file. Run the same script again and the run carries on from the last generation saved, instead of starting over; the result is the same as if it had never stopped. Once the run has finished, running it again returns its result at once. The optimizer must be set up as before, with the same config, search space and initial points; a checkpoint from a different setup is refused.

With ask and tell, call optimizer.save_checkpoint(path) whenever you like, and optimizer.load_checkpoint(path) on a new optimizer set up the same way. Trials that were handed out but not yet told are handed out again by the next ask(), and can still be told by their ids.

A checkpoint is a pickle, which can run code when it is loaded, so only load checkpoints you saved yourself.

Failed evaluations

Simulations crash, time out or fail to converge. Report such a run as failed rather than inventing a penalty value:

optimizer.tell(trial, failed=True)

Inside optimize(), an objective function reports a failure by raising hone.EvaluationFailed or returning NaN or infinity; any other exception stops the run, so bugs in the function aren't hidden. Failed evaluations count toward evaluations and failures in the result, rank below every successful one, and are never used to breed new candidates.

Noisy objectives

When the same parameters can give different values, as with a stochastic simulation, a measurement on a physical rig or a model trained with random initialization, set noise=True:

config = GeneticOptimizerConfig(max_evaluations=2000, noise=True, final_evaluations=100)

Otherwise the optimizer believes every value it sees, and its best result is likely to be a point that got lucky. With noise=True:

• Every evaluation of the same point is averaged, and a point's fitness is that average. The genetic optimizer often breeds a point again, which re-evaluates it for free, and ranks its parents by their latest averages.
• Bayesian optimization measures improvement against the best value its model predicts at any evaluated point, rather than against the luckiest single value.
• final_evaluations keeps that many evaluations back for the end, to evaluate the most promising points again (up to 5 of them, each at least twice) and return the one with the best average. Its best_fitness is then an average too, and so a fairer estimate of what the result will achieve. It's off by default, because on small budgets the evaluations are worth more to the search.

python -m hone.benchmarks --noisy measures this: it adds noise to every evaluation and scores each run by the true, noise-free value of the point it returns.