#66 Machine Learning & Data Science Challenge 66
Greetings.
I am a machine learning engineer based in India, possessing a sustained interest in machine learning since my undergraduate studies. I have completed Stanford University's machine learning course (Andrew Ng) via Coursera, and IBM's machine learning and deep learning curriculum. My current focus is on machine learning and data science projects, aiming to leverage my expertise for impactful, real-world problem-solving.
How to initialize biases in deep learning?
It is possible and common to initialize the biases to be zero since the random numbers in the weights provide the asymmetry braking.
For ReLU non-linearities, some people like to use small constant values such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning, therefore obtain, and propagate some gradient.
However, it is unclear if this provides a consistent improvement (in fact some results seem to indicate that this performs worst) and it is more commonly used to use 0 bias initialization.
