© Copyright National University of Singapore. All Rights Reserved. Tutorial 3: Cross-Validation & Bootstrap Source: Lab Section of Chapter 5, JWHT13 JWHT13 : James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. © Copyright National University of Singapore. All Rights Reserved. AGENDA (See the Investment Allocation example in lecture note) © Copyright National University of Singapore. All Rights Reserved. Cross-Validation • We illustrate the three CV methods (Validation Set Approach, LOOCV, K-Fold CV) by implementing them on the Auto dataset. • The Auto dataset records 392 instances of (horsepower,mpg) pair. • Our aim is to use the above CV methods to find out the best polynomial regression model (linear or quadratic or cubic or higher-order) that regresses mpg on horsepower, i.e., find polynomial f such that mpg = f(horsepower). Fig. First 5 Rows of Auto Data in “.csv” File (Dataset available on https://www.kaggle.com/ishaanv/ISLR-Auto?select=Auto.csv) © Copyright National University of Singapore. All Rights Reserved. “Auto” Data Processing © Copyright National University of Singapore. All Rights Reserved. The Validation Set Approach Recap: © Copyright National University of Singapore. All Rights Reserved. The Validation Set Approach (Cont’d) horsepower horsepower2 horsepower horsepower2 horsepower3 © Copyright National University of Singapore. All Rights Reserved. K-Fold Cross-Validation (K-Fold CV) Recap: © Copyright National University of Singapore. All Rights Reserved. K-Fold CV (Cont’d) From K-Fold CV, the quadratic polynomial yields the lowest fitting error rate without increasing model complexity too much. © Copyright National University of Singapore. All Rights Reserved. Leave-One-Out Cross-Validation (LOOCV) Recap: © Copyright National University of Singapore. All Rights Reserved. LOOCV (Cont’d) The result of LOOCV coincides with previous results. © Copyright National University of Singapore. All Rights Reserved. AGENDA (See the Investment Allocation example in lecture note) © Copyright National University of Singapore. All Rights Reserved. Bootstrap Fig. First 5 Rows of Portfolio Data in “.csv” File (https://github.com/vincentarelbundock/Rdatasets/blob/master/csv/ISLR/Portfolio.csv) © Copyright National University of Singapore. All Rights Reserved. Bootstrap (Recap) Recap: © Copyright National University of Singapore. All Rights Reserved. Bootstrap (Cont’d) How can we translate the Bootstrap Procedure into code? 1. A function that can randomly select observations from the data, with replacement 2. A function that compute the estimator of interest based on selected observations 3. A function that can repeat the re-sampling procedure and record the value of estimator in each repetition; It should be built upon the above two functions © Copyright National University of Singapore. All Rights Reserved. Bootstrap (Cont’d) Output: © Copyright National University of Singapore. All Rights Reserved. THE END