5 activation functions you should know!
š§µ
Yesterday we touched on activation functions, you should read that thread as well:
twitter.com/levikul09/status/1664915552379482112?s=20
Activation Functions are just like any other mathematical function.
It has three elements/steps:
- Input (X-axis)
- Calculation
- Output (Y-axis)
Different activation functions do different math. Let's discuss 5 š½
1ļøā£ ReLU
ReLU is widely used due to its simplicity and effectiveness.
It returns the input value if it is positive and zero otherwise.
Usually, ReLU is the default activation function.
2ļøā£ Sigmoid
The sigmoid is a smooth S-shaped curve that maps the input to a value between 0 and 1.
Sigmoid can be used for learning complex decision functions since it introduces non-linearity.
It is mainly used for binary classification.
3ļøā£ Tanh
Tanh is similar to the Sigmoid function but maps the input to a value between -1 and 1.
4ļøā£ Leaky ReLU
Leaky ReLU is a variation of the ReLU function.
It introduces a small slope for negative inputs, preventing neurons from becoming completely inactive (zero).
5ļøā£ Softmax
Softmax is primarily used in the output layer for multi-class classification problems.
It transforms the raw outputs of the neural network into a vector of probabilities.
Softmax ensures that the sum of the output probabilities is equal to 1.