Forward Propagation

Forward propagation is the fundamental process in artificial neural networks (ANNs) learning that transmits information from the input layer through weighted sums and activation functions to hidden layers and ultimately to the output layer. This stage encompasses the processing of input data provided to the network to generate predicted outputs and sets the stage for optimizing the computed error value using backpropagation backpropagation.

The forward propagation process is mathematically expressed for each neuron through the following steps:

Each neuron in the network receives signals (input values) from neurons in the preceding layer, multiplies them by their corresponding weights, and sums the results. Additionally, each neuron has a bias value. Mathematically, the weighted sum applied to the inputs of the i-th neuron is calculated as: 

$z_i={\sum }_{j=1}^n{w}_i{x}_j+b_i$

• Sigmoid Function: Compresses the output into the (0,1) range, enabling probabilistic interpretations.

$f(z)=\frac{1}{1+e^{-}z}$

$f(z^i)=\frac{e^{z}i}{\sum _{i=1}^n{e}^{z}i}$

• Regression Problems: A linear activation function f(z) = z or the ReLU function is used.
• Binary Classification: The sigmoid function is used, and the output value represents the probability of belonging to the relevant class.
• Multiclass Classification: The softmax function is applied to compute probabilities for each class.

The predictions generated at the end of forward propagation are compared with the real values to compute the loss function. Subsequently, this error is used to update the network’s weights through the backpropagation process.

This process is one of the fundamental mechanisms enabling learning in neural networks and constitutes a critical stage that allows artificial neural networks to establish meaningful relationships with data.

^{Example of Forward Propagation (Credit:} Miuul⁾ $z_i={\sum }_{j=1}^n{w}_i{x}_j+b_i$