Benchmarks¶
hone.benchmarks runs optimizers on test problems with known optima. See Benchmarks for what the suites measure and the latest results.
run_benchmark(problems: Iterable[Problem], optimizers: dict[str, OptimizerFactory] | None = None, seeds: Iterable[int] = range(10), budgets_per_dimension: Iterable[int] = BUDGETS, jobs: int = 1) -> list[Run]
Run every optimizer on every problem with every seed and every budget.
A budget of b evaluations per dimension gives a run b × dimensions evaluations. Each budget is a separate run, so an optimizer can choose its settings from the budget. An optimizer that cannot search a problem's parameters is skipped for that problem, and one whose factory raises SkipBudget is skipped at that budget. With jobs > 1, runs are shared among that many processes where the platform can fork them; the results are the same.
One Summary per problem, optimizer and budget, in the order the runs were made.
to_markdown(summaries: list[Summary], tolerance: float = 0.0001) -> str
An overview of wins at each budget, then a table per budget with a row per problem.
standard_problems(dimensions: tuple[int, ...] = (2, 5, 10)) -> list[Problem]
The full suite: every continuous function, plain and rotated, and every mixed problem, in each dimension.
quick_problems() -> list[Problem]
A small subset for CI: one dimension, and a problem of each kind.
noisy_problems() -> list[Problem]
Problems with Gaussian noise added to every evaluation, scored by the true value of the result.
constrained_problems(dimensions: tuple[int, ...] = (2, 5, 10)) -> list[Problem]
The constrained suite: boundary sphere and ellipsoid in each dimension, Rosenbrock in a disk, and three engineering designs.
conditional_problems() -> list[Problem]
Each conditional problem, followed by its flat version.
ml_problems() -> list[Problem]
The machine learning tuning problems. Needs scikit-learn.
model-selection-breast-cancer comes twice: with conditions, and as model-selection-breast-cancer-flat with every setting always searched, as before conditions, to show what they are worth.
pydantic-model
¶
Bases: BaseModel
An objective function, its search space, and the lowest value it can take.
Config:
frozen:True
Fields:
-
name(str) -
function(Callable[..., float]) -
search_space(SearchSpace) -
optimum(float) -
tags(frozenset[str]) -
noise(float)
pydantic-model
¶
Bases: BaseModel
One optimizer on one problem with one seed and one budget.
Config:
frozen:True
Fields:
-
problem(str) -
optimizer(str) -
budget_per_dimension(int) -
seed(int) -
error(float) -
evaluations(int) -
trace(list[tuple[int, float]])
pydantic-model
¶
Bases: BaseModel
All runs of one optimizer on one problem with one budget.
Config:
frozen:True
Fields:
-
problem(str) -
optimizer(str) -
budget_per_dimension(int) -
runs(int) -
median_error(float) -
lower_quartile(float) -
upper_quartile(float) -
solved(float)