Getting ready

The formula for the cost function is the following:

If the cost function looks familiar, it's because it is really just another way of minimizing the squared difference between the actual output and the prediction. The purpose of gradient descent or backpropagation in a neural network is to minimize the cost function until the value is close to 0. At that point, the weights and bias (w1, w2, and b) will no longer be random insignificant values generated by numpy, but actual significant weights contributing to a neural network model.