The activation functions for Neural Network in PyTorch

An activation function is the function or layer which enables neural network to learn complex(non-linear) relationships by transforming the output of the previous layer. *Without activation functions, neural network can only learn linear relationships.

There are popular activation functions as shown below:

(1) Step Function:

• can convert input values to 0 or 1. *If input < 0, then 0 while if input >= 0, then 1.
• is also called Binary Step Function, Unit Step Function, Binary Threshold Function, Threshold Function, Heaviside Step Function or Heaviside Function.
• is heaviside() in PyTorch.

(2) ReLU(Rectified Linear Unit) Function:

• can convert input values to the output values between 0 and input. *If input <= 0, then 0 while if input > 0, then input.
• is ReLU() in PyTorch.

(3) Leaky ReLU(Leaky Rectified Linear Unit) Function:

• can convert input values to the output values between input * slope and input. *Memos:
  • If input <= 0, then input * slope while if input > 0, then input.
  • slope is 0.01 basically.
• is the improved version of ReLU Function.
• is LeakyReLU() in PyTorch.

(4) ELU(Exponential Linear Unit) Function:

• can convert input values to the output values between a * (e^input - 1) and input. *Memos:
  • If input <= 0, then a * (e^input - 1) while if input > 0, then input.
  • a is 1.0 basically.
• is the improved version of ReLU Function.
• is ELU() in PyTorch.

(5) Sigmoid Function:

• can convert input values to the output values between 0 and 1.
• 's formula is y = 1 / (1 + e^-x).
• is Sigmoid() in PyTorch.

(6) Tanh Function:

• can convert input values to the output values between -1 and 1.
• 's formula is y = (e^x - e^-x) / (e^x + e^-x).
• is also called Hyperbolic Tangent Function.
• is Tanh() in PyTorch.

(7) Softmax Function:

• can convert input values to the output values between 0 and 1 each and whose sum is 1(100%). *If input values are [5, 4, -1], then the output values are [0.730, 0.268, 0.002] which is 0.730(73%) + 0.268(26.8%) + 0.002(0.2%) = 1(100%).
• 's formula is:
• is Softmax() in PyTorch.