Chapter 31 Regularized Discriminant Analysis

We now use the Sonar dataset from the mlbench package to explore a new regularization method, regularized discriminant analysis (RDA), which combines the LDA and QDA. This is similar to how elastic net combines the ridge and lasso.

31.1 Sonar Data

# this is a temporary workaround for an issue with glmnet, Matrix, and R version 3.3.3
# see here: http://stackoverflow.com/questions/43282720/r-error-in-validobject-object-when-running-as-script-but-not-in-console
library(methods)
library(mlbench)
library(caret)
library(glmnet)
library(klaR)
data(Sonar)
#View(Sonar)
table(Sonar$Class) / nrow(Sonar)
## 
##         M         R 
## 0.5336538 0.4663462
ncol(Sonar) - 1
## [1] 60

31.2 RDA

Regularized discriminant analysis uses the same general setup as LDA and QDA but estimates the covariance in a new way, which combines the covariance of QDA \((\hat{\Sigma}_k)\) with the covariance of LDA \((\hat{\Sigma})\) using a tuning parameter \(\lambda\).

\[ \hat{\Sigma}_k(\lambda) = (1-\lambda)\hat{\Sigma}_k + \lambda \hat{\Sigma} \]

Using the rda() function from the klaR package, which caret utilizes, makes an additional modification to the covariance matrix, which also has a tuning parameter \(\gamma\).

\[ \hat{\Sigma}_k(\lambda,\gamma) = (1 -\gamma) \hat{\Sigma}_k(\lambda) + \gamma \frac{1}{p} \text{tr}(\hat{\Sigma}_k(\lambda)) I \]

Both \(\gamma\) and \(\lambda\) can be thought of as mixing parameters, as they both take values between 0 and 1. For the four extremes of \(\gamma\) and \(\lambda\), the covariance structure reduces to special cases:

  • \((\gamma=0, \lambda=0)\): QDA - individual covariance for each group.
  • \((\gamma=0, \lambda=1)\): LDA - a common covariance matrix.
  • \((\gamma=1, \lambda=0)\): Conditional independent variables - similar to Naive Bayes, but variable variances within group (main diagonal elements) are all equal.
  • \((\gamma=1, \lambda=1)\): Classification using euclidean distance - as in previous case, but variances are the same for all groups. Objects are assigned to group with nearest mean.

31.5 Comparison to Elastic Net

set.seed(1337)
fit_elnet_grid = train(Class ~ ., data = Sonar, method = "glmnet",
                       trControl = cv_5_grid, tuneLength = 10)
set.seed(1337)
fit_elnet_int_grid = train(Class ~ . ^ 2, data = Sonar, method = "glmnet",
                           trControl = cv_5_grid, tuneLength = 10)

31.6 Results

get_best_result = function(caret_fit) {
  best_result = caret_fit$results[as.numeric(rownames(caret_fit$bestTune)), ]
  rownames(best_result) = NULL
  best_result
}
knitr::kable(rbind(
  get_best_result(fit_rda_grid),
  get_best_result(fit_rda_rand)))
gamma lambda Accuracy Kappa AccuracySD KappaSD
0.5000000 0.0000000 0.8217189 0.6390489 0.0455856 0.0926920
0.1460436 0.3391397 0.8362369 0.6705529 0.0631932 0.1255389
knitr::kable(rbind(
  get_best_result(fit_elnet_grid),
  get_best_result(fit_elnet_int_grid)))
alpha lambda Accuracy Kappa AccuracySD KappaSD
0.4 0.0065641 0.8034843 0.6041866 0.0645470 0.1297952
0.1 0.0243225 0.8418118 0.6809599 0.0539204 0.1088486

31.8 RMarkdown

The RMarkdown file for this chapter can be found here. The file was created using R version 3.5.1 and the following packages:

  • Base Packages, Attached
## [1] "stats"     "graphics"  "grDevices" "utils"     "datasets"  "methods"  
## [7] "base"
  • Additional Packages, Attached
## [1] "klaR"    "MASS"    "glmnet"  "foreach" "Matrix"  "caret"   "ggplot2"
## [8] "lattice" "mlbench"
  • Additional Packages, Not Attached
##  [1] "Rcpp"         "lubridate"    "class"        "assertthat"  
##  [5] "rprojroot"    "digest"       "ipred"        "mime"        
##  [9] "R6"           "plyr"         "backports"    "stats4"      
## [13] "e1071"        "evaluate"     "highr"        "pillar"      
## [17] "rlang"        "lazyeval"     "rstudioapi"   "data.table"  
## [21] "miniUI"       "rpart"        "combinat"     "rmarkdown"   
## [25] "labeling"     "splines"      "gower"        "stringr"     
## [29] "questionr"    "munsell"      "shiny"        "compiler"    
## [33] "httpuv"       "xfun"         "pkgconfig"    "htmltools"   
## [37] "nnet"         "tidyselect"   "tibble"       "prodlim"     
## [41] "bookdown"     "codetools"    "later"        "crayon"      
## [45] "dplyr"        "withr"        "recipes"      "ModelMetrics"
## [49] "grid"         "xtable"       "nlme"         "gtable"      
## [53] "magrittr"     "scales"       "stringi"      "reshape2"    
## [57] "promises"     "bindrcpp"     "timeDate"     "generics"    
## [61] "lava"         "iterators"    "tools"        "glue"        
## [65] "purrr"        "survival"     "yaml"         "colorspace"  
## [69] "knitr"        "bindr"