Deep Learning CS-661

Monday

Tuesday

Wednesday

Thursday

History of Deep Learning
McCulloch Pitts Neuron, Thresholding Logic, Perceptrons, Perceptron Learning Algorithm and Convergence, Multilayer Perceptrons (MLPs), Representation Power of MLPs.
Sigmoid Neurons, Gradient Descent, Feedforward Neural Networks, Representation Power of Feedforward Neural Networks, Feed forward neural networks, Backpropagation.
Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated GD, Stochastic GD, AdaGrad, RMSProp, Adam
Bias Variance Tradeoff, L2 regularization, Early stopping, Dataset augmentation, Parameter sharing and tying, Injecting noise at input, Ensemble methods, Dropout, Greedy Layerwise Pre-training, Better activation functions, Better weight initialization methods, Batch Normalization.
Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art.
Learning Vectorial Representations Of Words, Recurrent Neural Networks, Backpropagation Through Time (BPTT), Vanishing and Exploding Gradients, Truncated BPTT, Gated Recurrent Units (GRUs), Long Short Term Memory (LSTM) Cells, Solving the vanidhing gradient problem with LSTMs.
Encoder Decoder Models, Attention Mechanism, Attention over images, Hierarchical Attention, Transformers: Multi-headed Self Attention.
Additional Topics: Generative Adversarial Networks (GANs), variational Autoencoders (VAE)

Ian Goodfellow and Yoshua Bengio and Aaron Courville. Deep Learning. An MIT Press book. 2016.
Christopher M. Bishop, Neural Networks for Pattern Recognition.
Neural Networks and Deep Learning [ Online Book ]