Please see the homework policy document for detailed instructions and some grading notes. Failure to follow instructions will result in point reductions.

“Nobody actually creates perfect code the first time around, except me. But there’s only one of me.”

— Linus Torvalds

For this homework, you may only use the following packages:

# general
library(MASS)
library(caret)
library(tidyverse)
library(knitr)
library(kableExtra)
library(mlbench)

# specific
library(randomForest)
library(gbm)
library(klaR)
library(ellipse)

If you feel additional general packages would be useful for future homework, please pass these along to the instructor.

You should use the caret package and training pipeline to complete this homework. Any time you use the train() function, first run set.seed(1337).

Exercise 1 (Tuning KNN Regression with caret)

[6 points] For this exercise we will train KNN regression models for the Boston data from the MASS package. Use medv as the response and all other variables as predictors. Use the test-train split given below. When tuning models and reporting cross-validated error, use 5-fold cross-validation.

data(Boston, package = "MASS")
set.seed(1)
bstn_idx = createDataPartition(Boston$medv, p = 0.75, list = FALSE)
bstn_trn = Boston[bstn_idx, ]
bstn_tst = Boston[-bstn_idx, ]

Consider \(k \in \{1, 5, 10, 15, 20, 25, 30, 35\}\) and two pre-processing setups:

Provide plots of cross-validated error versus tuning parameters for both KNN pre-processing setups. Use the same value on the \(y\) axis for both plots. (You can be lazy and let caret create these plots. Since it will use lattice plotting, putting them side-by-side, or on the same plot would be difficult.)

Exercise 2 (More Regression with caret)

[7 points] For this exercise we will train more regression models for the Boston data from the MASS package. Use medv as the response and all other variables as predictors. Use the test-train split given previously. When tuning models and reporting cross-validated error, use 5-fold cross-validation.

Traing a total of three new models:

gbm_grid = expand.grid(interaction.depth = c(1, 2, 3),
                       n.trees = (1:20) * 100,
                       shrinkage = c(0.1, 0.3),
                       n.minobsinnode = 20)

Provide plots of error versus tuning parameters for the the boosted tree model. Also provide a table that summarizes the cross-validated and test RMSE for each of the three (tuned) models as well as the two models tuned in the previous exercise.

Exercise 3 (Clasification with caret)

[7 points] For this exercise we will train a number of classifiers using the training data generated below. The categorical response variable is classes and the remaining variables should be used as predictors. When tuning models and reporting cross-validated error, use 10-fold cross-validation. We will not use a test set for this exercise.

set.seed(42)
# simulate data using mlbench
sim_trn = mlbench.2dnormals(n = 500, cl = 7, r = 10, sd = 3)
# create tidy data
sim_trn = data.frame(
  classes = sim_trn$classes,
  sim_trn$x
)
featurePlot(x = sim_trn[, -1], 
            y = sim_trn$classes, 
            plot = "pairs",
            auto.key = list(columns = 2),
            par.settings = list(superpose.symbol = list(pch = 1:9))
)

Fit a total of five models:

Provide a plot of acuracy versus tuning parameters for the RDA model. Also provide a table that summarizes the cross-validated accuracy and their standard deviations for each of the five (tuned) models.

Exercise 4 (Concept Checks)

[1 point each] Answer the following questions based on your results from the three exercises.

Regression

(a) What value of \(k\) is chosen for KNN without predictor scaling?

(b) What value of \(k\) is chosen for KNN with predictor scaling?

(c) What are the values of the tuning parameters chosen for the boosted tree regression model?

(d) Which regression model achieves the lowest cross-validated error?

(e) Which method achieves the lowest test error?

Classification

(f) What are the values of the tuning parameters chosen for the RDA model?

(g) Based on the scatterplot, which method, LDA or QDA, do you think is more appropriate? Explain.

(h) Based on the scatterplot, which method, QDA or Naive Bayes, do you think is more appropriate? Explain.

(i) Which model achieves the best cross-validated accuracy?

(j) Do you believe the model in (i) is the model that should be chosen? Explain.