Site icon Machine Learning For Analytics

Theoretical Approach to Perceptron

Theoretical Approach to Perceptron

Hi ML Enthusiasts! Today, we will take the first step towards learning Artificial Neural Networks. In this post, we will learn about the functioning of neural networks and their analogy with human brain and neurons, i.e., we will step towards learning theoretical approach to perceptron.

For learning how to implement the perceptron algorithm using python, please go through our post on How to implement the perceptron algorithm using Python.

The Basis of Theoretical Approach to Perceptron

The Brain and The Neurons

Human brain is the best example of learning by experience. The concept of generalizing by past experience, adapting to new changes and making decisions based on memories are best seen in the human brain. So, let’s learn about the basic working of brain and neurons.

Thus, one can say that a neuron is a processor/computer. With so many neurons in the brain, the brain is a hub to so many processors/computers.

To increase our understanding of the neural networks, let’s learn about Hebb’s rule first.

Advancing to Theoretical Approach to Perceptron

Hebb’s Rule

Hebb’s rule helps us in understanding that the relation between strength of synaptic connections and firing of these two connecting neurons. Thus, if there are any changes in the strength of synaptic connections, then these changes are proportional to the correlation in the firing of these connecting neurons. Thus, if two neurons fire simultaneously, the connection between them increases in strength. If they never fire simultaneously, the strength of their connection decreases.

McCulloch and Pitts Neurons

McCulloch and Pitts modelled a neurons as:

The diagram

To understand more about it, please have a look at the picture shown below:

Thus, for each neuron j, the transfer function becomes

Also, the output for each output node j is given by the following equation:

where θ is the value given by threshold function.

The Approach

The idea is that firstly, we choose any random values of the weights and then apply inputs xi to the input nodes. Using these random weights, the first prediction is made. The prediction is subtracted from the corresponding target observations to produce errors. These errors are then used to make the weights adaptable using the following equations:

Here, η is called as the learning rate.

The Learning Rate

The learning rate tells the algorithm how much to change the weights by. Very less value of learning rate will make the algorithm run slower than desired. But, it will also make the algorithm more stable and resistant to any inaccuracies. Very large value will make the algorithm never reach the optimum value. Thus, learning rate should be chosen very carefully. Usually, 0.1 ≤ η ≤ 0.4 is the moderate interval for choosing the value of learning rate.

The Bias Input

In order to make the threshold adaptable to any changes, so that neuron firing is made adaptable, we use the bias node. Let’s consider a situation where all the inputs are zero. Now, it doesn’t matter what the weights are since all the inputs are zero. So, to make the neuron fire, we need to adjust the threshold value. If we need to fix the threshold at zero, we include a bias node having input as -1 and we can choose any random weight. Since, the weights are adaptable, the weight to the bias node also gets changed. The only thing impacting the output and weights is the bias node in this case. Bias input node is represented by x0 and the bias weight is represented by w0j.

The Perceptron Algorithm

The perceptron algorithm is divided into three phases:

 Backward Propagation: Update the weights using the following equation:

Please note that for training phase, the time complexity is O(Tmn) because it involves T iterations and for recall phase, the time complexity is O(mn).

So guys, with this we conclude our tutorial. Stay tuned for more interesting blog posts.

Exit mobile version