AML Assignment 1 Description
Hyperparameter optimization
1 Introduction
The tech company LeTech has millions of customers generating large volumes of data every day.
Because of this, the company has large data lakes that can be used for (predictive) analysis using
machine learning methods.
The developers are well aware that one size does not fit all and that different problems require
different machine learning techniques. Every technique, however, also has a set of hyperparameters
that can highly influence its performance. The developer team lacks the technical expertise to
properly tune the hyperparameters of different machine learning algorithms and ask their manager
to hire an external force for this.
The manager has contacted you and together you have defined a research project. He is interested
in finding the most efficient way to perform hyperparameter optimization and has asked you to
investigate this. You have narrowed down the scope of the project to the Support Vector Machine
and have constrained the hyperparameter optimization techniques that you investigate to random
search, grid search, and vanilla Bayesian optimization using the Expected Improvement. Your task is
to investigate the efficiency and quality of these three hyperparameter optimization techniques.
The company has provided you with 5 data sets that you should use for your investigations. These
data sets are representative of problems that they face. Every data set presents a classification
problem containing a number of input-output examples. These data sets together with the code to
load a given data set can be found in the provided ZIP file (a1.zip).
2 Technical requirements
In a brainstorming session with some of your colleagues, the following points were brought up:
• You should use the sklearn.svm.SVC as Support Vector Machine (not sklearn.svm.LinearSVC)
• For the implementation of Bayesian optimization, you decided to use the GaussianProcessRegressor
as surrogate function from sklearn
• (Hint: note that some models in sklearn have the option to return the standard deviation of
its predictions using the argument return std=True in the .predict function)
1
3 Requirements
The following requirements hold for this project:
• This is an individual project. You are not allowed to work together with another student on
this. There will be a strict plagiarism check on both the code and the report that you submit.
• The manager has asked you to write a report using the provided template of at most 2
pages excluding references. You are not allowed to deviate from this template (it is also
not allowed to change the margins or font size in the template). The report should contain
the following things and provide an answer to the following questions:
– Contain a clear introduction that tells the reader what the goal of the report (clear
research question) is and and some interesting conclusions
– Brief description of the three techniques: how do the three techniques work? (and what
are their advantages/disadvantages)
– From the AutoML lecture you have learned that Bayesian optimization is often more
efficient than random search and random search more efficient than grid search. Can
you validate/falsify this statement? The points below may help you.
– Complete and clear description of your experimental setup. Your experiements must be
reproducible from just reading the report!
– Rigorous experimental design that is suited for the research question above. You should
reason in a logical way about practical choices like i) how you selected the hyperparame-
ters that you are tuning, ii) how you selected the grid/ranges for the techniques, iii) how
you report your results (informative figures or tables)
– Good analysis and interpretation of the experimental results. Clear and logical conclu-
sions.
• You are NOT allowed to use existing libraries for the implementation of the three hyperparam-
eter optimization techniques (i.e., you are not allowed to use things like RandomizedSearchCV,
GridSearchCV from sklearn)
• The report should be concisely written and void of spelling and grammar mistakes (consider
using the Grammarly Chrome extension)
• To make a submission, upload all your code in a ZIP file to your Google/One Drive account
and make a shareable link (no Github links!). Put this link in your report (in a footnote) for
us to see. Hand in the report.pdf through TurnitIn.
• The project deadline is due the 29th of September 2021, 23:59PM CET
4 Bonus (to get a maximum grade of 10)
The maximum grade without doing this bonus will be a 9. To receive a maximum grade of 10, you can
implement and investigate the Successive Halving technique for hyperparameter optimization.
2