Machine Learning Strategies Part 07: Addressing Bias and Variance

Mohammad Faizan

·

Follow

4 min read

·

Feb 10, 2023

--

Listen

Share

In the previous articles, we saw there are two main components of error, called avoidable bias and variance. In this article, we will see how to address them.

Addressing bias and Variance

Here is the solution to address bias and variance problems:

• If you have a high avoidable bias; then it is recommended to use a bigger model i.e. a neural network with more hidden layers and hidden units.
• If you run into a high variance problem; then it is preferred to use more data to train your model.

Training a bigger neural network may solve your avoidable bias problems; however, increasing the model size makes your training slow and requires high computational power. A more giant network and big data will improve the performance significantly. However, it may not be possible to collect a large dataset for some tasks.

Different neural network architectures will have different bias/variance problems. It is important to search for optimal performance architecture. Academic literature and open-source implementations on GitHub are great sources of inspiration when you are looking for an optimal model architecture that reduces both bias and variance problems.

Increasing model size can increase the chance of over-fitting, increasing the variance while reducing bias. It seems like there is a trade-off between bias and variance. However, you can effectively increase model size without increasing variance by using regularization. For example, if you are using regularization such as L2 regularization or dropout with your deep learning model that performs well on your hold-out-cross-validation set, then increasing the model size won’t hurt performance, it will stay the same or improve. The only drawback of using a bigger model is computational cost.

Bias vs. Variance tradeoff

You might have heard of bias vs. variance tradeoff. You make changes to your learning model to reduce bias and variance, there are some changes that may increase variance and decrease bias and vice versa. For instance, on a speech recognition system, you have high variance and low bias; you might use regularization parameters to reduce variance but your bias will increase. Similarly, if you have a high bias, then you could reduce it by increasing your model size such as more hidden units, more hidden layers, or training longer. However, this will add a tradeoff. How would you handle this?

In the big data and deep learning era, now you have much more flexibility. You can reduce bias without hurting variance, and vice versa. It’s mainly because of the availability of huge data and computational power to train giant deep networks. For example, you can train a bigger network with an optimal regularization parameter to reduce bias without hurting variance and you can reduce variance by using a lot of data.

Techniques for reducing avoidable bias

If you train your machine learning model and you see that your algorithm is suffering from high avoidable bias, you could the following techniques to reduce it.

• Increase the model size: this technique will reduce avoidable bias since it will fit the training set well. You can increase your model size by adding more hidden layers and hidden units. To avoid over-fitting, you can use L2 regularization or dropout.
• Modify input features based on knowledge from error analysis: During error analysis, if you saw that adding some features will reduce your avoidable bias, then you can additional features for performance improvement.
• Reduce or get rid of regularization: You can reduce or eliminate the value of L2, L1, or dropout regularization value to reduce avoidable bias, however, it may increase variance.
• Model architecture search: you will have to search for optimal model architecture that may reduce both bias and variance problems.

Techniques for reducing variance

If your learning algorithm suffers from high variance, you can try the following to address it:

• Adding more training data: This is the most simplest and reliable way to address the variance problem, if you have access to more data and computational power, then this is recommended step for addressing variance.
• Use regularization: This technique reduces variance but increases bias, you can use L1, L2 regularization, or dropout.
• Add Early stopping: stopping learning early based on dev set error. This technique helps reduce variance but it could increase bias, it can be used as a regularization technique.
• Model architecture search: As mentioned above, it can also be used to address variance. You will have to search for optimal model architecture that may reduce both bias and variance problems.

Summary

Bias and variance are two main sources of error in machine learning. Analyzing error and suggesting promising techniques could help speed up development. There are various techniques you can use to address bias and variance. In the next article, I will discuss how you can identify and address your error using the insight from the learning curve.

References

[1].Ng, Andrew. “Machine learning yearning. 2018.” URL: htts://info.deeplearning.ai/machine-learning-yearning-book (2017).

[2]. Java Point. “Bias and Variance in Machine Learning” URL: https://www.javatpoint.com/bias-and-variance-in-machine-learning/February 10, 2023

Thanks for reading the article.

About Me:

I am a Masters student in robotics, and my research interest is computer vision and deep learning. I have 2 years of experience working as a machine learning and python developer. I have been working on object detection, single camera multi object tracking, multi camera multi object tracking, retail analytics, medical imaging, person re-identification, and graph machine learning.

• Connect with me on LinkedIn

Please feel free to comment below if you have any question.

BECOME a WRITER at MLearning.ai