By Ayo Onikoyi

Tuning in Machine Learning

Recent benchmarks show that suboptimal hyperparameter choices can slash a model’s accuracy by 20%. This critical insight inspired a comprehensive review co-authored by Mr. Ikenna Odezuligbo, published in the July 2024 issue of the Journal of Engineering Research and Reports. It delved into the intricacies of hyperparameter optimisation, providing valuable insights for researchers and practitioners alike.

Our correspondent’s findings reveal how Mr. Odezuligbo’s dual expertise, spanning machine learning and biomedical imaging drives his in-depth analysis.

Machine learning models rely on a set of parameters that are learned from data. However, these models also have hyperparameters – settings that are configured by the user before the learning process begins. These hyperparameters, such as the learning rate in neural networks or the depth of a decision tree, significantly influence the model’s ability to learn effectively and generalise to new, unseen data. The process of finding the best combination of hyperparameters is known as hyperparameter tuning or optimisation.

“The selection of appropriate hyperparameters is paramount to the success of any machine learning project,” Mr. Odezuligbo explained. “Poorly chosen hyperparameters can lead to models that either underfit the data, failing to capture the underlying patterns, or overfit the data, performing well on the training set but poorly on new data. Our review aimed to provide a comprehensive overview of the various techniques available for navigating this critical aspect of machine learning.”

The paper critically examined a wide array of hyperparameter tuning techniques, ranging from traditional methods like grid search and random search to more sophisticated approaches such as Bayesian optimisation and evolutionary algorithms. Each method has its strengths and weaknesses in terms of efficiency, effectiveness, and computational cost. Mr. Odezuligbo provided a comparative analysis of these techniques, offering guidance on when and how to apply them effectively.

“Grid search and random search are foundational techniques, but they can be computationally expensive, especially when dealing with a large number of hyperparameters or a wide range of possible values,” Mr. Odezuligbo noted. “More advanced methods like Bayesian optimisation use probabilistic models to intelligently explore the hyperparameter space, often finding better configurations with fewer evaluations.”

The review also highlighted the growing role of automated machine learning (AutoML) platforms, which integrate feature selection, model architecture search, and hyperparameter tuning into unified workflows.

“AutoML is democratising machine learning by making it more accessible to individuals without deep expertise in the field,” Mr. Odezuligbo observed. “These platforms can not only accelerate development but also uncover novel configurations that manual tuning might miss.”

Drawing on his work with fluorescence lifetime imaging microscopy (FLIM), Mr. Odezuligbo illustrates the trade-offs practitioners face. Advanced methods like population-based training can boost accuracy but may require hundreds of GPU hours, an important consideration for teams with limited compute budgets. The review provided concrete figures to help scientists match method choices to available resources.

“Balancing performance gains against computational cost is crucial,” he said. “Our paper offers a detailed breakdown of GPU-hour requirements for each tuning strategy.”

Ultimately, Mr. Odezuligbo’s study does more than catalogue tuning methods; it equips practitioners with a practical guide for navigating the hyperparameters. By combining foundational approaches with cutting-edge techniques and tying each to real world use cases, the paper clarifies not only how to tune but why each strategy matters.