<h1 id="heading-how-to-initialize-biases-in-deep-learning">How to initialize biases in deep learning?</h1>
<ul>
<li><p>It is possible and common to initialize the biases to be zero since the random numbers in the weights provide the asymmetry braking.</p>
</li>
<li><p>For ReLU non-linearities, some people like to use small constant values such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning, 
therefore obtain, and propagate some gradient.</p>
</li>
<li><p>However, it is unclear if this provides a consistent improvement (in fact some results seem to indicate that this performs worst) and it is more commonly used to use 0 bias initialization.</p>
</li>
</ul>


# How to initialize biases in deep learning?

- It is possible and common to initialize the biases to be zero since the random numbers in the weights provide the asymmetry braking.

- For ReLU non-linearities, some people like to use small constant values such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning, 
therefore obtain, and propagate some gradient.

- However, it is unclear if this provides a consistent improvement (in fact some results seem to indicate that this performs worst) and it is more commonly used to use 0 bias initialization.


How to initialize biases in deep learning?

#66 Machine Learning & Data Science Challenge 66

A Mountain Hiker, Space, and Tech Enthusiast who became a Data Scientist.
