Search spaces

Parameters

Class	Values	Example
RangeParameter	A number between minimum and maximum; integer=True restricts it to whole numbers, and log=True searches it on a log scale	RangeParameter("rate", 0, 1)
ChoiceParameter	One of a list of choices, or a list of choose_n distinct choices	ChoiceParameter("kernel", ["linear", "rbf"])
FixedParameter	A single value that never changes	FixedParameter("mode", "fast")

Use log=True for settings that span orders of magnitude, such as learning rates, regularization strengths or tree counts. Values are then spread evenly in their logarithm, so 0.001 to 0.01 is searched as thoroughly as 100 to 1000, and every optimizer searches the range that way. A log range needs a positive minimum, and works with integer=True:

RangeParameter("learning_rate", 1e-5, 1e-1, log=True)
RangeParameter("n_estimators", 10, 1000, integer=True, log=True)

Every parameter the objective function requires must be in the search space. The optimizer checks this before it starts.

Constraints

Limits across parameters are written with the parameters themselves and passed to SearchSpace. Every optimizer then only asks for points that satisfy every constraint, so the objective is never evaluated outside them:

from hone import RangeParameter, SearchSpace

width = RangeParameter("width", 1, 10)
height = RangeParameter("height", 1, 10)
search_space = SearchSpace([width, height], constraints=[width * height <= 40, width >= 2 * height])

Range parameters and numbers combine with +, -, *, /, ** and abs(), and <= or >= makes the result a constraint; < and > mean the same. A point where a constraint can't be computed, such as one that divides by zero, counts as breaking it. There are no equality constraints: an equality leaves nothing to sample, so remove a variable instead and compute it inside the objective. Write a range as two constraints, since Python reads 0 <= width - height <= 5 as two comparisons joined by and, and hone raises an error rather than keep one half.

For anything an expression can't say, such as a square root or a choice parameter, use a function, which takes parameters by name and is satisfied when it returns zero or less:

from hone import Constraint

def stress(width, height, material):
    return bending_stress(width, height) - ALLOWED[material]

search_space = SearchSpace([width, height, material], constraints=[Constraint.function(stress)])

Constraints should be quick to check: hone checks them for every candidate it considers, which can be many times the number of evaluations. Expression constraints are saved with a result; a function constraint saves only its name, so after Result.load pass the function again with rebuild_optimizer(functions={"stress": stress}).

Starting points must satisfy the constraints. If the constraints leave nothing to search, or a region too small to find in 100,000 random draws, the optimizer raises InfeasibleSearchSpace when it's created, before any evaluation. Each optimizer keeps to constraints in its own way: random search and Bayesian optimization only consider points that satisfy them, CMA-ES draws a candidate again while it breaks one, and the genetic optimizer moves such a child back toward its fitter parent until it is feasible, which tends to land it near the boundary, where constrained optima usually are. On the benchmark's constrained problems the choice between optimizers follows the same pattern as without constraints; see benchmarks/README.md. examples/pressure_vessel.py designs a pressure vessel of least cost under three constraints and finds the best published design. examples/maintenance_schedule.py chooses a maintenance schedule for a production line, scored by a noisy simulation of a year, with a constraint on technician hours and a function constraint that depends on the chosen shift; it halves the expected yearly cost of a sensible schedule chosen by hand.

Conditional parameters

Some settings only exist when another takes certain values: a kernel's settings only for the model that has kernels, a third layer's size only with three layers or more. Write each condition as test >> parameters, read "then", and pass them to SearchSpace:

from hone import ChoiceParameter, RangeParameter, SearchSpace

model = ChoiceParameter("model", ["svm", "forest"])
kernel = ChoiceParameter("kernel", ["linear", "rbf"])
C = RangeParameter("C", 1e-3, 1e3, log=True)
gamma = RangeParameter("gamma", 1e-5, 1, log=True)
trees = RangeParameter("trees", 10, 500, integer=True)

search_space = SearchSpace(
    [model, kernel, C, gamma, trees],
    conditions=[
        model["svm"] >> [kernel, C],
        kernel["rbf"] >> gamma,  # only with an svm too, since kernel is
        model["forest"] >> trees,
    ],
)

def train(model, kernel=None, C=None, gamma=None, trees=None):
    ...

The left of >> is a test: a choice parameter indexed by the values that switch the parameters on, such as model["svm"] or optimizer["sgd", "nesterov"], or a comparison of range parameters in brackets, such as (layers >= 3) >> units_3. The brackets are needed, since Python reads layers >= 3 >> units_3 as layers >= (3 >> units_3); hone raises an error saying so. The right is a parameter or a list of them, and each parameter can have one condition. Conditions nest: a parameter whose test uses a parameter that is switched off is off too.

A parameter that is switched off isn't part of the point. The objective isn't passed it, so give it a default, which hone checks before the run starts; results, progress and best_params leave it out; and a starting point must leave it out too. Two points that differ only in a parameter switched off are the same point, so they are averaged together with noise=True and the genetic optimizer doesn't count one as new.

A constraint applies only while every parameter it uses is switched on, and test >> constraint makes a constraint apply only while the test holds:

depth = RangeParameter("depth", 2, 20, integer=True)
search_space = SearchSpace(
    [model, trees, depth],
    constraints=[model["forest"] >> (trees * depth <= 2000)],
    conditions=[model["forest"] >> [trees, depth]],
)

Every optimizer supports conditions; CMA-ES only conditions on range parameters, since it doesn't search choices. Bayesian optimization gains the most: a parameter switched off looks the same to its model whatever value it holds, so values that don't matter add no noise. On the benchmark's conditional problems its error is ×0.0004 that of searching every parameter all the time, at 10 evaluations per dimension. With conditions the genetic optimizer breeds each branch within its own niche by default (niche_share of 0.5, below), so a wide, easy branch doesn't crowd out a narrow, better one; see benchmarks/README.md. examples/choose_a_model.py chooses a classifier, among an SVM with three kernels, a random forest and logistic regression, together with its settings.

Starting points

If you already have a good design, or the best parameters from an earlier run, pass it as a starting point so the search doesn't begin from scratch:

optimizer = CMAES(search_space, CMAESConfig(max_evaluations=500), initial_points={"thickness": 2.5, "width": 40})

# or continue from an earlier result
optimizer = GeneticOptimizer(search_space, config, initial_points=[earlier.best_params])

initial_points takes one dict of parameter values or a list of them; fixed parameters may be left out, and every value is checked against the search space. The points are evaluated first and count toward the budget. The genetic optimizer and random search put them in their first generation, filling the rest with random candidates. CMA-ES evaluates them, then centres its search on the best one; lower initial_step to keep the search close to it.