Marcel Aach1,2, Xin Liu1
1 JSC, Forschungszentrum Jülich
2 University of Iceland
When: 13:15 - 18:00
Where: JSC meeting room 1, building 16.3, room 350 [coordinates]
The performance of machine learning models is highly dependent on their hyperparameters that are set by the user before the training. The hyperparameters define the general architecture of the model (e.g., via the number of layers or the neurons per layer in a neural network) and control the learning process (e.g., via the learning rate of the optimizer or the weight decay). However, searching for optimal hyperparameter values is a long and resource-intensive process, as many different combinations need to be evaluated and the final performance of a combination can usually only be measured after a machine learning model is fully trained.
This tutorial presents a systematic introduction to the field of Hyperparameter Optimization (HPO) and demonstrates how to make use of resource-efficient methods to reduce the runtime of HPO in small and large settings on High-Performance Computing systems. Two HPO optimization libraries (Ray Tune and DeepHyper) are introduced, making use of evolutionary, Bayesian, and early stopping-based algorithms. As HPO is a general method and can be adapted to any machine learning model, it is useful for scientists from many different domains.
Agenda
- 13:30 - 14:15 General Introduction to Hyperparameter Optimization and Algorithms
- 14:15 - 15:45 Hands-On with Ray Tune Library
- 15:45-16:00 Break
- 16:00 - 16:30 Hands-On with Weights and Biases Library for Logging and Visualization of HPO
- 16:30-16:45 Discussion of Results
Requirements
Participants should
- bring their own Laptops and be familiar with running code on a cluster
- should have access to Google Colab (https://colab.research.google.com/), as well as an account with Weights and Biases (https://wandb.ai/) for the hands-on part.
Latest details on github [link]
