1/17

1/19

Lecture 1: Intro to ML

• What is ML? ML applications
• Machine learning paradigms
• Supervised learning (classification, regression, feature selection)
• Unsupervised learning (density estimation, clustering, dimensionality reduction)
• Data mining concepts and tasks
• Association rules, similarity search, cluster analysis, outlier analysis
• Basic data types
• (Mixed) attribute data, text, time series, sequence, network data
• The problem solving process:
• Business/project understanding, data understanding through EDA, data preparation, modeling, evaluation, deployment

Readings:

• Witten & Frank       Chapter 1.1-1.3
• Provost & Fawcett    Chapter 2

1/19

1/24

Lecture 2: Exploratory Data Analysis

• Getting to know your data
• Data types
• Attribute types
• Data quality issues
• Data visualization
• Histogram, Kernel Density Estimation
• Charts, plots, infographics
• Correlation analysis

Readings:

• Aggarwal         Chapter 2
• Witten & Frank    Chapter 2
• Hastie     Chapter 6.6.1

1/26

1/31

Lecture 3: Data Preparation

• Feature creation
• Data cleaning
• Missing, inaccurate, duplicate values
• Data transformation
• Feature type conversion
• Discretization
• Normalization / Standardization
• Data reduction
• Feature and record selection
• Principal Component Analysis
• Multidimensional scaling
• Manifold learning (Isomap, LLE)

Readings:

• Witten & Frank    Chapter 7.1-7.4
• PCA tutorial
• Isomap website     Isomap paper
• LLE website     LLE paper

2/2

2/7

Lecture 4: Linear Models

• Linear Regression
• Robust Regression
• Sparse Linear Models
• Feature subset selection: revisited
• Shrinkage methods: ridge regression and Lasso
• Principal components regression, Partial least squares

Readings:

• ISLR (James, Witten, Hastie, Tibshirani)      Chapter 3.1, 3.2, 3.3, 3.4
• ISLR (James, Witten, Hastie, Tibshirani)      Chapter 6.1, 6.2.1, 6.2.2, 6.3.1, 6.3.2

• Hastie                Chapter 3.1-3.4, 4.4
• Shalizi               Chapter 2, 11
• Murphy              Chapters 1.4, 7.1-7.5, 13.3-13.5
• Provost & Fawcett     Chapter 4
• Witten & Frank        Chapter 7.5

2/7

2/9

Lecture 5: Logistic Regression and Generalized Models

• Logistic Regression
• Generalized Linear Models (GLMs)
• Generalized Additive Models (GAMs)
• Basis expansions
• Generalizations, shape functions

Readings:

• ISLR (James, Witten, Hastie, Tibshirani)      Chapter 4.1, 4.2, 4.3
• ISLR (James, Witten, Hastie, Tibshirani)      Chapter 7.1, 7.2, 7.3, 7.4, 7.6, 7.7
• https://web.stanford.edu/~hastie/Papers/gam.pdf
• Intelligible Models for Classification and Regression by Yin Lou, Rich Caruana, and Johannes Gehrke.

• Hastie     Chapter 9.1, 9.3, 9.6
• Shalizi     Chapter 12

2/14

Lecture 6: Model Selection

• What is a good model?
• Overfitting
• Decomposition of error
• Bias-Variance tradeoff
• Cross Validation
• Regularization
• Information Criteria (AIC, BIC, MDL)

Readings:

• Hastie                Chapter 7.1 - 7.10
• Provost & Fawcett     Chapter 5

2/16

Lecture 7: Model Evaluation

• Performance measures for Machine Learning
• Creating baseline methods for comparison
• Visualizing model performance

Readings:

• Witten & Frank        Chapter 5
• Provost & Fawcett     Chapter 7, 8, 11
• Shalizi               Chapter 3, 10

2/21

2/23

Lecture 8: Tree-based Methods

• Classification trees
• From trees to rules
• Missing values and pruning
• Regression trees

Readings:

• Hastie               Chapter 9.2
• Witten & Frank        Chapter 4.3-4.4, 6.1-6.2
• Provost & Fawcett     Chapter 3
• Shalizi               Chapter 13
• Murphy              Chapter 16.2

2/28

Lecture 9: Support Vector Machines

• SVM intuition, formulation, and the dual
• Slack variables, Hinge loss
• The Kernel trick
• Kernel SVM
• Kernel Logistic Regression
• Kernel PCA

Readings:

• Witten & Frank        Chapter 6.3
• ISL-with R            Chapter 9
• Murphy               Chapter 14.2, 14.5
• http://www.cs.cornell.edu/courses/cs578/2007fa/slides_sigir03_tutorial.pdf

3/2

3/7

Lecture 10: Instance-based Learning

• Kernel Density Estimation
• k-Nearest Neighbor Classifier
• Kernel Regression
• Locally-Weighted Linear Regression

Readings:

• Hastie            Chapter 6.1-6.3, 6.6.1-6.6.2
• Murphy           Chapter 1.4.1-1.4.3, 14.7
• Shalizi            Chapter 7.1, 7.5

3/7

Lecture 11: Ensemble Learning

• Combining multiple models
• Bagging
• Random Forests

Readings:

• Witten & Frank       Chapter 8
• Hastie              Chapter 10.1, 15, 16
• ISL-with R             Chapter 8.2

Midterm Exam (in class)

Spring Break; No Classes

3/21

Lecture 11 (cont.): Ensemble Learning

• Bagging and Random Forests (Review)
• Boosting

Readings:

• Witten & Frank       Chapter 8
• Hastie              Chapter 10.1, 15, 16
• ISL-with R             Chapter 8.2

3/23

3/28

Lecture 12: Clustering

• Distance functions
• Hierarchical clustering
• k-means clustering
• Kernel k-means clustering
• k-medians clustering
• Mixture models
• The EM algorithm

Readings:

• Witten & Frank        Chapter 6.8
• ISLR (James, Witten, Hastie, Tibshirani)      Chapter 10.3
• Provost & Fawcett     Chapter 6, 12 (part)

3/30

Lecture 13: Association Rules

• Applications
• Frequent itemsets
• Association rule generation
• Interesting patterns

Readings:

• Witten & Frank        Chapter 4.5, 6.3
• Provost & Fawcett     Chapter 12
• Aggarwal             Chapter 4, 5.4

4/4

4/6

Lecture 14: Outlier Analysis

• Definition and types of outliers
• Challenges
• Different types of detection techniques
• Clustering, depth, and distance based techniques
• Density-based techniques: LOF and LOCI
• Ensemble methods: feature bagging, iForest
• High-dimensional approaches

Readings:

• Witten & Frank        Chapter 7.5
• Aggarwal             Chapter 8, 9.4, 9.5

4/11

Lecture 15: Semi-supervised Learning

• Assumptions (smoothness, cluster, manifold)
• Semi-supervised learning
• Self-training
• Generative methods
• Graph-based methods
• Co-training

Readings:

• Witten & Frank        Chapter 6.9
• Introduction to Semi-Supervised Learning
• Graph-based Semi-Supervised Learning Algorithms
• Combining Labeled and Unlabeled Data with Co-Training

HW3 due
HW4 out

4/13

Lecture 16: Unstructured Data: ML for Text

• Representing text
• Named entity extraction
• Novelty and first-story detection
• Topic models
• Applications

Readings:

• Witten & Frank        Chapter 9.5, 9.6
• Provost & Fawcett     Chapter 10
• Aggarwal             Chapter 13

4/18

4/25

Lecture 17: Dependent Data: ML for Time Series

• Time series preparation and similarity
• Trends and Anomalies
• Forecasting with ARMA, ARIMA models
• De-trending and seasonal components
• Change-point detection
• Monitoring the learning process: SPC algorithm
• CUSUM, Minimum MSE
• Multi-variate forecasting with VAR

Readings:

• Aggarwal         Chapter 14
• Shalizi           Chapter 21

4/25

4/27

Lecture 18: Dependent Data: ML for Networks

• Transductive learning
• Learning in networks with and without attributes
• Graph-regularized classification

Readings:

• Collective Classification in Network Data

Project Presentations I    (today's presenters return final report on 5/4)

Project Presentations II   (today's presenters return final report on 5/2)