How do I set a threshold on my metrics? 🤔

A key component of creating a SigOpt experiment is thinking about what you want to see :thinking:

For some experiments the goal is simply to maximize or minimize some a number of metrics. When defining such an experiment the metric-object will look similar to the following:

  metrics=[
    dict(
      name="holdout_accuracy [%]",
      strategy="optimize",
      objective="maximize"
    ),
    dict(
      name="inference_time [ms]",
      strategy="optimize",
      objective="minimize"
    )
  ],

However, a lot of modelers has a pretty good understanding of what they want to see, or at least some minimum criteria for success. This can be things like a minimum accuracy or a maximum inference time determined by the business.

When creating a SigOpt experiment you have the option to provide such information in the form of metric thresholds, something which will help guide the optimizer and hopefully lead you to good results faster.

When creating the experiment, you’ll just need to add two lines of code to the metric-object:

  metrics=[
    dict(
      name="holdout_accuracy [%]",
      strategy="optimize",
      objective="maximize",
      threshold=75
    ),
    dict(
      name="inference_time [ms]",
      strategy="optimize",
      objective="minimize",
      threshold=3
    )
  ],

And now the optimizer knows what you are looking for :sunglasses:, and of course you can update the thresholds throughout the experiment!

If you like metric thresholds, make sure to also check out metric constraints!

1 Like