Overview

This project consists of a large-scale simulation study to compare the performance of the two-sample \(t\)-test, a permutation test, and a nonparametric test.

Details

We wish to compare the performance of three tests that can be applied to two independent random samples: the two-sample \(t\)-test, the permutation test discussed in class, and the nonparametric Mann-Whitney U Test (see Sections 15.5 - 15.6 in your textbook). The necessary code to carry out the test is generally available in R, or was provided in class. The main goal of this project is to explore the relative performance of the methods under a variety of conditions, in order to gain an understanding of when one of the tests might be preferred over the other. This project is similar in spirit to the work you performed in Labs.

Objectives

Provide a thorough examination of the relative performance of the three tests described above. In particular, you should consider the type I error and the power of the tests under various true population distributions. You will want to include population distributions which satisfy the assumptions of some or all of the methods, as well as population distributions that violate the assumptions of some or all of the methods. You should also examine the effect of sample size and of effect size. The following give some guidelines for conditions to consider:

  1. Consider two populations that are normally distributed, and consider drawing samples of sizes \(n_1\) and \(n_2\) from these populations. Consider (at least) three choices of sample sizes: small, medium, and large. Consider several choices for the means of the two populations. To assess the level, you’ll need a simulation condition in which both means are equal. Consider also differing effect sizes (i.e., differences between the two means).
  2. Repeat part (1), but select a distribution other than the normal distribution. Pick a distribution that is symmetric and unimodal.
  3. Repeat part (1), but select a distribution other than the normal distribution or the distribution you picked in part (2). Pick a distribution that is not symmetric.
  4. Consider any other simulation conditions that you wish, perhaps motivated by your results in parts (1) - (3)
  5. Present your results in graphical and tabular format, and describe all of your observations. Compare the performance of the three tests across the range of conditions you examined.