# Chapter 31 Regularized Discriminant Analysis

We now use the Sonar dataset from the mlbench package to explore a new regularization method, regularized discriminant analysis (RDA), which combines the LDA and QDA. This is similar to how elastic net combines the ridge and lasso.

## 31.1 Sonar Data

# this is a temporary workaround for an issue with glmnet, Matrix, and R version 3.3.3
# see here: http://stackoverflow.com/questions/43282720/r-error-in-validobject-object-when-running-as-script-but-not-in-console
library(methods)
library(mlbench)
library(caret)
library(glmnet)
library(klaR)
data(Sonar)
#View(Sonar)
table(Sonar$Class) / nrow(Sonar) ## ## M R ## 0.5336538 0.4663462 ncol(Sonar) - 1 ##  60 ## 31.2 RDA Regularized discriminant analysis uses the same general setup as LDA and QDA but estimates the covariance in a new way, which combines the covariance of QDA $$(\hat{\Sigma}_k)$$ with the covariance of LDA $$(\hat{\Sigma})$$ using a tuning parameter $$\lambda$$. $\hat{\Sigma}_k(\lambda) = (1-\lambda)\hat{\Sigma}_k + \lambda \hat{\Sigma}$ Using the rda() function from the klaR package, which caret utilizes, makes an additional modification to the covariance matrix, which also has a tuning parameter $$\gamma$$. $\hat{\Sigma}_k(\lambda,\gamma) = (1 -\gamma) \hat{\Sigma}_k(\lambda) + \gamma \frac{1}{p} \text{tr}(\hat{\Sigma}_k(\lambda)) I$ Both $$\gamma$$ and $$\lambda$$ can be thought of as mixing parameters, as they both take values between 0 and 1. For the four extremes of $$\gamma$$ and $$\lambda$$, the covariance structure reduces to special cases: • $$(\gamma=0, \lambda=0)$$: QDA - individual covariance for each group. • $$(\gamma=0, \lambda=1)$$: LDA - a common covariance matrix. • $$(\gamma=1, \lambda=0)$$: Conditional independent variables - similar to Naive Bayes, but variable variances within group (main diagonal elements) are all equal. • $$(\gamma=1, \lambda=1)$$: Classification using euclidean distance - as in previous case, but variances are the same for all groups. Objects are assigned to group with nearest mean. ## 31.5 Comparison to Elastic Net set.seed(1337) fit_elnet_grid = train(Class ~ ., data = Sonar, method = "glmnet", trControl = cv_5_grid, tuneLength = 10) set.seed(1337) fit_elnet_int_grid = train(Class ~ . ^ 2, data = Sonar, method = "glmnet", trControl = cv_5_grid, tuneLength = 10) ## 31.6 Results get_best_result = function(caret_fit) { best_result = caret_fit$results[as.numeric(rownames(caret_fit\$bestTune)), ]
rownames(best_result) = NULL
best_result
}
knitr::kable(rbind(
get_best_result(fit_rda_grid),
get_best_result(fit_rda_rand)))
gamma lambda Accuracy Kappa AccuracySD KappaSD
0.5000000 0.0000000 0.8217189 0.6390489 0.0455856 0.0926920
0.1460436 0.3391397 0.8362369 0.6705529 0.0631932 0.1255389
knitr::kable(rbind(
get_best_result(fit_elnet_grid),
get_best_result(fit_elnet_int_grid)))
alpha lambda Accuracy Kappa AccuracySD KappaSD
0.4 0.0065641 0.8034843 0.6041866 0.0645470 0.1297952
0.1 0.0243225 0.8418118 0.6809599 0.0539204 0.1088486

## 31.8 RMarkdown

The RMarkdown file for this chapter can be found here. The file was created using R version 3.5.2 and the following packages:

• Base Packages, Attached
##  "stats"     "graphics"  "grDevices" "utils"     "datasets"  "methods"
##  "base"
##  "klaR"    "MASS"    "glmnet"  "foreach" "Matrix"  "caret"   "ggplot2"
##  "lattice" "mlbench"
##   "Rcpp"         "lubridate"    "class"        "assertthat"
##   "digest"       "ipred"        "mime"         "R6"
##   "plyr"         "stats4"       "e1071"        "evaluate"
##  "highr"        "pillar"       "rlang"        "lazyeval"
##  "rstudioapi"   "data.table"   "miniUI"       "rpart"
##  "combinat"     "rmarkdown"    "labeling"     "splines"
##  "gower"        "stringr"      "questionr"    "munsell"
##  "shiny"        "compiler"     "httpuv"       "xfun"
##  "pkgconfig"    "htmltools"    "nnet"         "tidyselect"
##  "tibble"       "prodlim"      "bookdown"     "codetools"
##  "crayon"       "dplyr"        "withr"        "later"
##  "recipes"      "ModelMetrics" "grid"         "nlme"
##  "xtable"       "gtable"       "magrittr"     "scales"
##  "stringi"      "reshape2"     "promises"     "timeDate"
##  "generics"     "lava"         "iterators"    "tools"
##  "glue"         "purrr"        "survival"     "yaml"
##  "colorspace"   "knitr"