This project consists of a large-scale simulation study to compare the performance of the two-sample \(t\)-test, a permutation test, and a nonparametric test.
We wish to compare the performance of three tests that can be applied to two independent random samples: the two-sample \(t\)-test, the permutation test discussed in class, and the nonparametric Mann-Whitney U Test (see Sections 15.5 - 15.6 in your textbook). The necessary code to carry out the test is generally available in R
, or was provided in class. The main goal of this project is to explore the relative performance of the methods under a variety of conditions, in order to gain an understanding of when one of the tests might be preferred over the other. This project is similar in spirit to the work you performed in Labs.
Provide a thorough examination of the relative performance of the three tests described above. In particular, you should consider the type I error and the power of the tests under various true population distributions. You will want to include population distributions which satisfy the assumptions of some or all of the methods, as well as population distributions that violate the assumptions of some or all of the methods. You should also examine the effect of sample size and of effect size. The following give some guidelines for conditions to consider: