Forward Propagation

Forward propagation is the fundamental process in the learning phase of artificial neural networks (ANNs), wherein information is transmitted from the input layer through hidden layers to the final output layer via weighted sums and activation functions. This stage encompasses the processing of given inputs to generate predicted outputs and sets the foundation for optimizing the calculated error through backpropagation.

The forward propagation process is mathematically described for each neuron through the following steps:

Each neuron in the network receives signals from neurons in the preceding layer (input values) and multiplies them by corresponding weights before summing them. Additionally, each neuron has a bias term. Mathematically, the weighted sum applied to the inputs of neuron i is calculated as follows:

$z_i={\sum }_{j=1}^n{w}_i{x}_j+b_i$

Where:

• $x_j$ is the input value from neuron j in the previous layer,
• $w_{i}j$ is the weight associated with that input,
• $b_i$ is the bias term,
• $z_i$ represents the total weighted input to the neuron.

The result of the weighted sum is passed through an activation function to enable the learning of nonlinear relationships. Activation functions enhance the network’s learning capacity and improve performance on complex problems. The general form of the activation function is:

$a_i=f(z_i)$

Common activation functions include:

• Sigmoid Function: Compresses the output into the (0,1) range, enabling probabilistic interpretations.

$f(z)=\frac{1}{1+e^{-}z}$

• Tanh (Hyperbolic Tangent) Function: Normalizes the output to the (-1,1) range.

$f(z)=\frac{e^z-e^{-}z}{e^z+e^{-}z}$

• ReLU (Rectified Linear Unit) Function: Passes only positive values and sets negative inputs to zero.

$f(z)=max(0,z)$

• Softmax Function: Used in multiclass classification problems to determine the probability distribution across all classes.

$f(z^i)=\frac{e^{z}i}{\sum _{i=1}^n{e}^{z}i}$

The outputs from the activation functions are passed as inputs to the next layer. If the neural network contains multiple hidden layers, this process is repeated at each layer to complete forward propagation. The activations of the neurons in the final layer constitute the model’s ultimate prediction.

The values produced by the output layer are accepted as the model’s prediction. The activation function used in the output layer varies depending on the type of problem:

• Regression Problems: A linear activation function f(z) = z or the ReLU function is used.
• Binary Classification: The sigmoid function is used, and the output represents the probability of belonging to the positive class.
• Multiclass Classification: The softmax function is applied to compute the probability distribution across all classes.

After forward propagation, the generated predictions are compared with the actual values to compute the loss function. This error is then used in the backpropagation process to update the network’s weights.

This process is one of the fundamental mechanisms enabling learning in neural networks and constitutes a critical stage that allows artificial neural networks to establish meaningful relationships with data.

^{Example of Forward Propagation (Credit:} Miuul⁾ $z_i={\sum }_{j=1}^n{w}_i{x}_j+b_i$