Deep Learning using Images and Signals

• When the inter-class distance is too high, the accuracy of the model will be abnormally high. This essentially means that the data is too easy to classify (eg. cherries and mangoes).
• In general, choose data from the recent past only. Data from 10 years ago typically had a higher inter-class distance, designed to cater to the deep learning models of those times (and they weren’t as advanced as today’s models).

• Inhibitory Input: It’s an input which can independently change the decision
• Exhibitory Input: These inputs can only collectively change the decision.

• For example, here’s the AND-NOT function:

  \(x_{1}\)	\(x_{2}\)	\( f(x) = x_{1} \overline{x_{2}} \)
  0	0	0
  0	1	0
  1	0	1
  1	1	0

  When \(x_{2} = 1\), \(f(x)\) is always 0, regardless of what \(x_{1}\) is. So \(x_{2}\) is inhibitory.

• Here’s another example:

  \(x_{1}\)	\(x_{2}\)	\( f(x) \)
  0	0	1
  0	1	0
  1	0	0
  1	1	0

  When \(x_{2} = 1\), \(f(x)\) is always 0, regardless of what \(x_{1}\) is. Similarly, \(x_{1} = 1\), \(f(x)\) is always 0, regardless of what \(x_{2}\) is. In this case, both \(x_{1}\) and \(x_{2}\) are inhibitory.

• \(g(x) = W^{T}X = 0 \). \(W\) is perpendicular to any point X lying on the decision boundary.

• If the angle between W and X is α, then: \[ cos(\alpha) = \frac{W^{T}X}{|W||X|} \]

  α is less than 90 for \(p_{1}\), \(p_{2}\) and \(p_{3}\), and will be greater than 90 for \(n_{1}\), \(n_{2}\) and \(n_{3}\).

• Let \(\alpha_{new}\) be the angle made by the new \(W^{T}\) and X. \[ W_{new}^{T} = W^{T} + \eta X \] where η is called the learning rate.
  • \( cos(\alpha_{new}) = \frac{W_{new}^{T} X}{|W_{new}^{T}|}\)
    • \( cos(\alpha_{new}) \alpha W_{new}^{T} X\)
    • \(\alpha_{new} < \alpha \)

• Given Bipolar data (only -1 and 1)
• \(g(x) = w_{0} + w_{1}x_{1} + w_{2}x_{2} = y_{in}\)
• \(f(x) = 1 if g(x) \ge 0, f(x) = -1 if g(x) < 0 \)
• We try to match a target \(t\) to all of the outputs

• The algorithm is given as:

  for each input:
      compute y_in  # aka. g(x)    aka. aggregation
      compute y_out # aka. f(g(x)) aka. activation aka. y_out
      if t != y:
          delta_w = alpha * t * x
          w = w_old + delta_w
  
  

  Assume all weights to be 1 and bias to be -1 for this example.

\[w_{new} = w_{old} + \Delta w\] where \( \Delta w = \eta t x \)

• The traditional Hebbian rule is unsupervised (no target value). It uses \(y_{out}\) instead of target value \(t\).
• What we’re following is called supervised Hebbian rule.

\[w_{new} = w_{old} + \Delta w\] where \( \Delta w = \eta(t-y)x \)

• Instead of using \(\eta t x \), and hoping t==y at some point, we use the difference between the predicted value and the actual value.
• Bias is also updated using the same formula, just that \(x=1\).

• Assume both weights and the bias to be 0, and the learning rate \(\eta\) to be 0.1.

  Epoch	\(x_{1}\)	\(x_{2}\)	\(t \)	\(y_{in} \)	\(y_{out} = f(x)\)	\((t-y) \)	\(\Delta w_{1} =\)	\(\Delta w_{2} = \)	\(\Delta b \)	\(w_{1} \)	\(w_{2}\)	b
  1	1	1	1	0	1	0	0	0	0	0	0	0

• This rule, is supervised, and error based, as opposed to the correlation based approach in the traditional hebbian rule.

• Let N_i be the number of neurons in layer \(i\), and \(n\) is the number of layers (layer 1, layer 2, layer 3, … layer \(n\)).
• Total = [Number of weights] + [Number of biases]
• Total = [N₁*N₂ + N₂*N₃+ N₃*N₄ + …] + [N₂ + N₃ + N₄ + …]
• Total = [Σ_i=1^n-1N_i*N_i+i] + [Σ_i=2ⁿN_i]

• The error function must be continuous and differentiable, so that you’re not taking sudden and high jumps.

• A general trick is that, the number of neurons in a hidden layer is \[\frac{2}{3}*N_{i} + N_{o}\] where N_i is the number of neurons in the input layer, and N_o is the number of neurons in the output layer.

• In a neural network, you have to initialize weights with random values.
• If you initialize all the weights with zeroes, it can lead to something called the symmetry problem.
  • All of the weights are equal, and hence the aggregation and activation for every neuron gives the same output.
  • All the neurons are learning the same thing, and it’s almost like there’s only one neuron present in each layer.

• There are mainly 3 steps:

• During gradient descent, as errors are propagated backward through the layers, the magnitude of the gradient keeps reducing.
• By the time the errors are propagated to the initial layers, they’re too small.
• Earlier layers are important for the model to understand low level features, so if their weights don’t update, the model can’t understand low level features.
• The easiest fix would be to use ReLU instead of tanh and sigmoid because they suffer the most from vanishing gradient.

• You allocate some data for training.
• You allocate some data for validation, and this is used to see how the model performs after every epoch.
• Initially, the training loss and the validation loss are both very high.
  • Training loss decreases because it’s getting more relevant to the training dataset (it has started to generalize).
  • Validation loss decreases because it has started to generalize.
• After a certain point of time, when the model has overtly learned the data and overfits:
  • Training loss still decreases as it’s still getting more relevant to the training dataset.
  • But validation loss increases because the model is getting relevant only to the training data, and is getting irrelevant to the validation data: it’s deviating from the general pattern of the data.
• You stop training the model when the training loss reduces, but the validation loss starts increasing. This is called early stopping.

• This helps in combating overfitting.

• x + Noise ⇒ \(\bar{x}\)
• \(bar{x}\) is passed to neural network h(x) and outputs \(\hat{x}\)
• x_hat is more similar to x than \(bar{x}\).

• Activations
• Normalize
• Scale and Shift
• Find Output

• Only concerned with the objective function.

• Do something with the objective function, but at the same time, you have some contraints.

• Dense/ Fully Connected Neural Networks: Every neuron in 1 layer is connected to every other neuron in the next layer.
• Up until now, to use images in a neural network, you’d vectorize the white-scale values and pass that 1D vector to the neural network.
• The issue is, you’re losing spacial information. For example:
  • An image of a number absolutely should be spread across the same pixels for every image in the dataset.
  • The number can be in different shapes, but the size and geographical position must remain consistent.
  • If a model is trained on a dataset of images taking the whole space and centered, you can’t expect the model to understand the image of a number smaller in size, and is located on the top-left corner of the image.

• Convolution is a linear mathematical operation \[ S_{(i,j)} = (I*K)_{(i,j)} = \Sigma_{a=0}^{m-1} \Sigma_{b=0}^{n-1} I_{(i-a, j-b)}K_{(a,b)} \]
• The output is called the feature map.

• When S=1, you move the kernel 1 pixel at a time (regardless of direction).
• When S=2, you move the kernel 2 pixels at a time.
• Essentially, striding downsamples.

\[ W_{featureMap} = \frac{W - F + 2P}{S} + 1 \] \[ H_{featureMap} = \frac{H - F + 2P}{S} + 1 \] where

• P is the padding on one side
• W is the width of the image
• H is the height of the image
• F is the side length of the kernel/filter.

• Pooling is a dynamic kernel of sorts. You’re not aggregating, but you’re doing a simpler dynamic operation.
• Pooling is used to downsample (reduce the dimensions) of the feature map.
• You take a lower number of pixels to represent some sort of feature/characteristic, and this helps in faster training, as well as reducing overfitting.
• The number of parameters involved in pooling is 0.

• When a convolutional layer is connected to a dense layer, a flattening layer must be used.
• It converts an n-dimensional layer (the feature map), into a 1-dimensional layer. For example, a 55 x 55 x 96 vector turns into a (55*55*96) x 1 layer. Basically, it becomes a 290400 x 1 layer.

• Proposed to recognize handwritten postal zip codes.

• Instead of relying on manual design by human beings, the model automates the process of finding the best neural network architecture (the layers used, activations, pooling, etc).

• VGG introduced the fact that there’s no need of larger kernels.
• Instead of a larger kernel, you can use a smaller kernel multiple times (sequentially). Use the kernel once, and then on the obtained feature map, use the kernel again.
• VGG-19 means there are 19 layers.
• It’s represented as: \[Conv(m, n, X)\] where
  • m x n is the kernel size
  • X is the number of kernels chosen
• Essentially, each kernel has different weights and each of them form their own feature map. These feature maps make up a big feature map with a certain amount of depth.

• Early layers learn fine details and edges, while later layers learn complex features.
• Essentially, the finer details like edges might be similar across all layers and perhaps color, shape and size of the objects are different.
• This means you’ll unnecessarily keep learning finer details, to get to the part where you actually need to learn (i.e. the more complex features).
• To solve vanishing gradient, you add a skip connection between layers.
• You directly pass the input of a layer to the end, so you have the final layers intact, and all you need to do is to learn upon the changes made to those fine details.
• Mathematically \[y=F(x)+x\] where
  • x is the feature map on one place of the image
  • F(x) is the sequence of convolutions or activations applied
  • y is the new feature map

• This introduced the concept of parallel convolutions.
• Given a W x H x D, you turn it into a W X H X 1 feature map using a 1 x 1 kernel moving across the depth.
• Each layer is a network, called an inception block.

• This is a type of segmentation.
• The entire network is divided into two halves:
  • Encoder
  • Decoder
• The encoder can be made up of smaller encoders too (all these encoders must have the same structure). Same goes with the decoder.
• The number of encoders (= number of decoders) is the maximum number of skip connections you can have.
• As you move across the encoders, the size of the feature map keeps decreasing, but the amount of features contained, keeps increasing.
• The part where the last encoder is connected to the first decoder, is caleld the bottleneck, and it has the smallest feature map, but the most amount of features.

• Up until now:
  • Each input to the network was independent of the previous or future inputs.
  • Inputs were of fixed size.
• RNNs are specialized to work on data where the input depends on previous or future inputs, and are of variable sizes.
• You can have RNNs of CNN, where CNNs are used for recognizing individual images, and RNNs are used to relate the sequence of these images.

• Sentences like “I ate a pizza” and “A pizza was ate by me” use almost the same words and mean the same thing, but an ANN will struggle to figure this out.
• This is because ANNs process things in a fixed order, as one single static chunk. This static chunk would have no information on whether “pizza” came before “ate” or after “ate”, unless it’s explicitly trained on all possible ways you can represent every single sentence of the dataset.
• For this, you’d have to use one-hot encoding for each and every word:

  • Say there are 25000 words in the dataset. Every word in the dataset would be represented as:

    how →	[	0	0	0	0	1	0	…	0	0	]
    		word1	word2	word3	word4	word5	word6		word24999	word25000

    where “how” is the 5th word.

  • Each word in the dataset, is a massive 1x25000 bit vector.

• RNNs process the sentence word-by-word, and for each word, it computes a function called a hidden state.

• A hidden state is a function of the current word and the previous word:

  Hidden_State(t) = f( Current_Word(t), Hidden_State(t-1) )

  It serves as the network’s short term memory.

• ANNs look like:

  x1 → ANN → y1
  x2 → ANN → y2
  x3 → ANN → y3
  

  Whereas, RNNs look like:

  x1 → [RNN Cell] → h1 → y1
        ↑
       h0 (usually zeroes)
  
  x2 → [RNN Cell] → h2 → y2
        ↑
       h1 (from before)
  
  x3 → [RNN Cell] → h3 → y3
        ↑
       h2
  

• One thing to note is that RNNs can’t handle spatial data.

• Each step essentially learns how much it contributed to the future steps’ error.
• Explicit Terms: In this term, you treat all other inputs as constants.
• Implicit Terms: Summing over all indirect paths from that hidden layer to w.q
• tanh is used for the hidden state.

• Batch Size: Number of sequences of words. This doesn’t affect the number of trainable.
• Sequence size: Number of words/vectors in a sequence
• Input Size: Size of the word/vector

• In Batch normalizations, activations for a single feature across all sequences, are normalized.
• In Layer normalizations, activations for all features across one single sequence (basically after every layer) is normalized. This helps because it’s now independent of batch size and sequence size.
• You use these before non-linearity is introduced.

• Vanishing or Exploding Gradients
• RNNs struggle to remember information from many time steps ago.
• It’s bound to forget data that it initially learns, because there is no concept of “importance” given to RNN cells. You’ll never know how important each cell is, because each cell just uses the same weights.
• RNNs can’t parallelize over time-steps and hence they are very slow in training.

• Now, the previous hidden state \(h_{t-1} = s_{t-1} 0 o_{t-1}\).
  • \(s_{t-1}\) is the same as \(h_{t-1}\) in RNN. It’s \(h_{t-1}\) in RNN but multiplied with a vector called the output gate \((o_{t-1})\).
  • Here, you’re selectively writing, which means you’re deciding how much of the previous output, actually becomes the previous hidden state.

• Since \(s_{t-1}\) is the same as \(h_{t-1}\) in RNN, \(s_{t}\) is the new hidden state calculated and that is given as:
  • \(s_{t} = s^{-}_{t} 0 i_{t} + s_{t-1} 0 f_{t}\), and this is the output of the LSTM.
  • \(s^{-}_{t}\) is what was supposed to be \(h_{t}\) in vanilla RNN. i.e. \(h_{t} = tanh( Wx_{t} + Uh_{t-1} + b)\)
  • Here, you’re selectively reading, which means that you’re deciding what part of this becomes input for the next cell, and how much of the previous hidden state is forgotten.

\[o_{t} = tanh(U_{o}x_{t} + W_{o}h_{t-1} + b_{o})\] \[i_{t} = tanh(U_{i}x_{t} + W_{i}h_{t-1} + b_{i})\] \[f_{t} = tanh(U_{f}x_{t} + W_{f}h_{t-1} + b_{f})\]

• Here, U_o, W_o, U_i, W_i, U_f, W_f are all learnable parameters.