Machine Learning (Spring 2020)

Lectures

• (January 22, 2020) Lecture 1 : Introduction and Formal Model of Learning (cf. [R1], Chapter 1). Paper by Leo Breiman on the two cultures can be found here.

• (January 27, 2020) Lecture 2 : PAC Learnability and Agnostic PAC Learnability (cf. [R1], Chapter 2)

• (January 29, 2020) Lecture 3 : Learning via Uniform Convergence (cf. [R1], Chapter 3)

• (February 3, 2020) Lecture 4 : No-Free Lunch Theorem and Bias-Complexity Tradeoff (cf. [R1], Theorem 5.1, Corollary 5.2)

• (February 5, 2020) Lecture 5 : VC-Dimension Theory (cf. [R1], Chapter 6)

• (February 10, 2020) Lecture 6 : The Fundamental Theorem of PAC Learning (cf. [R1], Theorem 6.7, Lemma 6.10, Theorem 6.11)

• (February 12, 2020) Lecture 7 : Convexity and Learning (cf. [R1], Chapter 12)

• (February 17, 2020) Lecture 8 : Regularized Loss Minimization and Stable Rules (cf. [R1], Sections 13.1, 13.2, 13.3)

• (February 24, 2020) Lecture 9 : Overfitting-Stability Tradeoff and Gradient Descent (cf. [R1], Corollary 13.8, Corollary 13.9, Section 14.1)

• (February 26, 2020) Lecture 10 : Gradient Descent (contd.) (cf. [R1], Sectino 14.1, Section 14.2)

• (March 2, 2020) Lecture 11 : Stochastic Gradient Descent (cf. Stochastic Gradient Descent and Its Variants in Machine Learning)

• (March 4, 2020) Lecture 12 : Linear Predictors and Linear Regression (cf. [R1], Section 9.1, Section 9.2)

• (March 9, 2020) Lecture 13 : Logistic Regresstion, Probability Interpretation of Regression, Regression Trees (cf. Andrew Ng's Notes; [R3], Section 9.2)

• (March 11, 2020) Lecture 14 : Classification Trees, Ensemble Learning I : Bagging, Random Forests (cf. [R1], Chapter 18; [R3], Section 9.2, Section 15.1, Section 15.2, Section 15.3)

• (March 23, 2020) Lecture 15 : Boosting, AdaBoost, Validation (cf. [R1], Chapter 10, Chapter 11)

• (March 30, 2020) Lecture 16 : Support Vector Machines (cf. [R1], Chapter 15)

• (April 1, 2020) Lecture 17 : Learning with Kernels (cf. [R1], Chapter 16)

• (April 6, 2020) Lecture 18 : Nearest Neigbor Methods (cf. [R1], Chapter 19. Here is an interesting paper on the application of kNN methods in the estimation of information theoretic quantities.)

• (April 8, 2020) Lecture 19 : Unsupervised Learning : k-means Clustering (cf. [R1], Chapter 22)

• (April 13, 2020) Lecture 20 : k-means Clustering (contd.), Spectral Clustering, Information Bottleneck (cf. [R1], Chapter 22)

• (April 15, 2020) Lecture 21 : Dimentionality Reduction (PCA), Compressed Sensing (cf. [R1], Chapter 23)

• (April 20, 2020) Lecture 22 : Generative Models, Neural Networks (cf. [R1], Chapter 24, Andrew Ng's Notes)

• (April 27, 2020) Lecture 23 : Final Project Presentations

• (April 29, 2020) Lecture 24 : Final Project Presentations

Assignments

• (Uploaded on February 12, 2020. Due by Wednesday, March 4, 2020, 2:00 pm) Assignment 1 (Solutions)

• (Uploaded on March 5, 2020. Due by Monday, March 30, 2020, 4:00 pm) Assignment 2 (Solutions)

• (Uploaded on March 30, 2020. Due by Monday, April 20, 2020, 2:00 pm) Assignment 3 (Solutions)

Final Project

• Term Paper, due by Wednesday, April 29, 2020, 2:00 pm

• Presentations on Monday, April 27, 2020 and Wednesday, April 29, 2020

Reference Books

• [R1] Understanding Machine Learning : From Theory to Algorithms, Shai Shalev-Shwartz and Shai Ben-David

• [R2] Pattern Recognition and Machine Learning, Christopher Bishop

• [R3] Elements of Statistical Learning, Trevor Hastie, Robert Tibshirani, and Jerome Friedman

• [R4] Introduction to Statistical Learning, Gareth James, Daniela Witten, Trevor Hastie, and Robert Tobshirani

• [R5] A Brief Introduction to Machine Learning for Engineers, Osvaldo Simeone

General Information

• Instructor : Himanshu Asnani

• Venue : A-201

• Class Timings : Mondays, 4:00 pm - 5:30 pm and Wednesdays, 2:00 pm - 3:30 pm

• Grading : Class Participation (10%), Homeworks (60%), Final Project (30%)

• Assignments : We will be using Jupyter Notebook for the programming aspect of the course. Programming language will be Python and the installation guide can be located here.

• Tentative Schedule

  • Module I [Theoretical Foundations] PAC Learning, Bias-Complexity Trade-off, VC Theory, Generalization, Gradient Descent, Stochastic Gradient Descent

  • Module II [Supervised Learning] Regression, Classification, Support Vector Machines, Decision Trees, Bagging, Boosting, Kernel Methods, Nearest Neighbor Methods, Neural Networks, Backpropagation

  • Module III [Unsupervised Learning] k-means Clustering, Spectral Clustering, Dimensionality Reduction (PCA), Compressed Sensing, Information Bottleneck, Naive Bayes, Linear Discriminative Analysis, Expectation-Maximization Algorithm