5 activation functions you should know!

🧵

Activation Functions are just like any other mathematical function.

It has three elements/steps:

- Input (X-axis)
- Calculation
- Output (Y-axis)

Different activation functions do different math. Let's discuss 5 🔽

1️⃣ ReLU

ReLU is widely used due to its simplicity and effectiveness.

It returns the input value if it is positive and zero otherwise.

Usually, ReLU is the default activation function.

2️⃣  Sigmoid

The sigmoid is a smooth S-shaped curve that maps the input to a value between 0 and 1.

Sigmoid can be used for learning complex decision functions since it introduces non-linearity.

It is mainly used for binary classification.

3️⃣  Tanh

Tanh is similar to the Sigmoid function but maps the input to a value between -1 and 1.

4️⃣ Leaky ReLU

Leaky ReLU is a variation of the ReLU function.

It introduces a small slope for negative inputs, preventing neurons from becoming completely inactive (zero).

5️⃣ Softmax

Softmax is primarily used in the output layer for multi-class classification problems.

It transforms the raw outputs of the neural network into a vector of probabilities.

Softmax ensures that the sum of the output probabilities is equal to 1.


If you liked this thread, you should also join our newsletter, DSBoost.

We share:

• Interviews

• Podcast notes

• Learning resources

• Interesting collections of content

dsboost.dev

5 Activation Functions You Should Know

There are 3 types of means.

But what is the difference between them?

An introduction to the arithmetic, geometric, and harmonic mean.

1️⃣ The well-known Arithmetic Mean

Calculated as the sum of the values divided by the total number of values.

- We use it when the data has the same unit.

- Outliers can heavily influence this mean.

2️⃣  Geometric Mean

Calculated as the N-th root of the product of all values.

(N is the number of values)

- More appropriate for dependent values, percentages, fractions, or widely-ranging data.

- Does not accept negative or zero values

3️⃣  Harmonic Mean

Calculated as the number of values N divided by the sum of the reciprocal of the values.

- Appropriate to calculate the average rate (speed, frequency)

- Does not take rates with a negative or zero value


If you haven't already, join our newsletter DSBoost.

We share:

• Interviews

• Podcast notes

• Learning resources

• Interesting collections of content

dsboost.dev

Understanding the Types of Means

5 Activation Functions to Know

In Statistics we have

- Normal distribution

- Standard normal distribution

But what the f**** is the difference?

Let's figure it out.

When you hear the term normal distribution, you probably think about the perfect bell shape.

But not all Normal distributions are perfectly shaped.

They are usually stretched or squeezed and moved horizontally right or left.

What are the effecting factors?

- Mean determines the center of the curve.

Increasing the mean will move the curve to the right.

- Standard deviation stretches or squeezes the curve.

A small standard deviation results in a narrow curve.

Here are some examples.

A (M = 0, SD = 1) Standard normal distribution

B (M = 0, SD = 0.5) Squeezed, because SD < 1

C (M = 0, SD = 2) Stretched, because SD > 1

D (M = 1, SD = 1) Shifted right, because M > 0

E (M = –1, SD = 1) Shifted left, because M < 0

Now it is time to answer our question.

What is the difference then?

- Normal distribution can take on any value as its mean and standard deviation.

- In the standard normal distribution, the mean and standard deviation are always fixed.

The standard normal distribution is a special normal distribution where:

- The mean is 0

- The standard deviation is 1

It is also called the z-distribution.

Wanna learn more about standard normal distribution?

Follow me at @levikul09 and come back tomorrow!

Thanks 😉

You should also join our newsletter, DSBoost.

We share:

• Interviews

• Podcast notes

• Learning resources

• Interesting collections of content

dsboost.substack. com

Normal vs Standard Normal Distribution

Yesterday we touched on activation functions, you should read that thread as well:

https://twitter.com/levikul09/status/1664915552379482112?s=20

5 Activation Functions You Should Know!

Neural Networks can tackle complex issues with complex calculations.

But today I will make the complex simple for you!

Let's see how Neural Networks work!

Yesterday we looked at the building blocks of Neural Networks.

Today we will build on this, so if you want to go back to the previous post check here:

https://twitter.com/levikul09/status/1664587120458207232?s=20

First of all, why do we use Neural Networks?

Sometimes you will face datasets that are too complicated for a simple model like Linear Regression.

Neural Networks however can identify any relationship between the variables.

Complexity can be tackled by complexity.

If you face a complex dataset, you can add layers and nodes to make your Neural Network more complex and better fit for the data.

Note: More complex Neural Networks are not always equal better results!

How do Neural Networks tackle complex tasks?

By using Activation Functions.

Activation functions introduce non-linearity (complexity) into the Neural Network.

Now let's add activation functions to the building blocks we already discussed.

Each node/neuron applies an activation function to its input, transforming it into an output value.

So nodes basically contain the activation functions.

Activation functions take inputs, do calculations, and then provide output.

There are many activation functions like:

- ReLU
- Sigmoid
- Binary stop function

Activation Functions are just like any other mathematical function.

Neural Networks basically just combine these existing functions to fit any type of data.

Activation functions

Neural Networks can look super complicated.

But they are all made from simple elements.

Let me explain the building blocks of a neural network.

🧵

Neural networks have two main parts:

- Nodes or Neurons
- Connections between the nodes

Here is a simple example below.

Neural Networks are usually organized in Layers. What are those?

We can group the different nodes into layers:

- Input nodes are in the input layers

- Output nodes are in the output layers

Between input and output, we have the hidden layers.

Let's see two examples:

- A simple Neural Network with single input and output nodes and a single hidden layer with 2 nodes.

- A more complex one with 2 input nodes, 2 hidden layers with 6 nodes, and 2 output nodes.

When you work with Neural Networks, you need to decide
how many hidden layers to use and how many nodes the hidden layers should contain.

Generally, more complex NNs (more layers and nodes) can fit complex data better.