Bayesian Optimization Tutorial

Bayesian optimization is a powerful tool for optimizing time-consuming-to-evaluate non-convex derivative-free objective functions. It is widely used for tuning hyperparameters in deep neural networks and has wider applications including engineering design, materials and drug discovery, and tuning e-commerce systems and online markets with A/B testing. It uses a machine learning predictive method, often Gaussian process regression, to model the objective function and an acquisition function to guide the choice of points at which to evaluate the objective.

The tutorials and short course below introduce these methods in python using the BoTorch BayesOpt library.

If you'd like a basic introduction to Bayesian optimization, read the tutorial article, watch the video or look at the slides that go with the video.
If you'd like a more hands introduction, go through days 1-3 of the short course.
If you'd like to learn about grey-box Bayesian optimization, which accelerates performance by "looking inside the black box", read the grey-box tutorial and go through day 4 of the short course.

Tutorial Articles:

Bayesian Optimization: article with slides and video
Grey-box Bayesian Optimization: article

4-Day Short Course

Day 1, Gaussian Process Regression:

Lecture slides [pdf]
In-class lab [ipython notebook, pdf] with data file [csv]
Solution [ipython notebook, pdf]

Day 2, Gaussian Process Regression in BoTorch

Lecture slides [pdf]
In-class lab [ipython notebook, pdf] using the same data file as Day 1
Solution [ipython notebook, pdf]

Day 3, Expected Improvement (in pure python & BoTorch)

Lecture slides [pdf]
In-class lab [ipython notebook, pdf]
Solution [ipython notebook, pdf]

Day 4, Grey-box Bayesian Optimization

Lecture slides [pdf]
In-class lab [ipython notebook, pdf]

Packages to install for labs:

BoTorch:

To install via Anaconda: conda install botorch -c pytorch -c gpytorch
To install via pip: pip install botorch

NumPy, SciPy, Pandas, Matplotlib