Artificial Neural Networks

The Neuron

• Branches on top are called Dendrites. These are receivers of signal
• Long tail is called Axon. This is transmitter of signal

 Dendrites of neuron are connected to Axon of other neurons.

The artificial neuron has 

• Input variables which are independent of each other
• Weights for each input variable
• Output value
• This can be continous like price
• Binary like person exists or not
• Categorical(this will have multiple output values)
• Neuron adds (weighted sum) & applies activation function

Activation Function

• Threshold function
• Y is 0 when x<0 
• Y is 1 when x>=0
• Sigmoid function
• 1/(1+e^-x)
• Rectifier
• max(x,0)
• Hyperbolic Tangent
• (1-e^-2x)/1+e^-2x

Cost Function:

• C = 1/2(y hat - y)^2
• Weights are calculated & updated using cost function and updated in system. This is called back propogation

How to minimized cost function

• Brute Force - Take lots of possible ways and find the best fit. However, this will fail when the number of weights/synopsis goes up
• Gradient decent(Batch Gradient decent)
• Start at some place, check the angle and take the correct direction to get to optimal place
• Stochastic Gradient descent
• Weights are adjusted after every row rather than complete set
• This will avoid local minimum problem.

Back Propagation

•  The key thing is that during back propagation - you basically know which part of the error each of your weights in neural network is responsible for.

Training ANN with Stochastic Gradient descent

• Randomly initialize the weights to small numbers(close to 0)
• Input the first observation of your dataset in the input layer, each feature in one input node
• Forward-propagation: from left to right, the neurons are activated in a way that the impact of each neuron's activation is limited by the weights. Propagate the activations until getting predicted result.
• Compare predicted result with actual result. Measure the generated error.
• Back-propagation: from right to left, the error is back propagated. Update the weights according to how much they are responsible for the error. The learning rate decides by how much we update the weights.
• Repeat steps 1 to 5 and update the weights after each observation (reinforcement learning)  OR
  Repeat steps 1 to 5 and update the weights after batch of observation.
• When the whole training set passed through the ANN, that makes an epoch. Redo more epochs.
• 
  

https://www.superdatascience.com/learning

Convolutional network(Input Image -> CNN-> output label)

• Convolutional operation
• ReLU layer
• Pooling
• Flattening
• Full Connection
• Summary
•