Goal: After completing this lab, you should be able to…

In this lab we will use, but not focus on…

Some additional notes:

Exercise 0A - \(t\)-Test in R

Recall the Deflategate data from previous homework, in particular the measurements for the Patriots.

pats = c(11.50, 10.85, 11.15, 10.70, 11.10, 11.60, 11.85, 11.10, 10.95, 10.50, 10.90)

Suppose that we wanted to test

\[ H_0\colon \mu_P = 12.5 \quad \text{vs} \quad H_A\colon \mu_P < 12.5 \]

To do so, we use the t.test() function in R. We use three of the function’s arguments:

t.test(x = pats, mu = 12.5, alternative = "less")
## 
##  One Sample t-test
## 
## data:  pats
## t = -11.465, df = 10, p-value = 2.241e-07
## alternative hypothesis: true mean is less than 12.5
## 95 percent confidence interval:
##      -Inf 11.32898
## sample estimates:
## mean of x 
##  11.10909

The output gives, among other things, the value of the test statistic as well as the p-value of the test.

The mu and alternative arguments have default values of 0 and two.sided respectively. So, suppose that we wished to test

\[ H_0\colon \mu = 0 \quad \text{vs} \quad H_A\colon \mu \neq 0 \]

for the following data.

some_data = c(-0.22, -0.51, 0.12, 0.14, -0.19)

Then, we would simply use.

t.test(some_data)
## 
##  One Sample t-test
## 
## data:  some_data
## t = -1.0934, df = 4, p-value = 0.3356
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.4671803  0.2031803
## sample estimates:
## mean of x 
##    -0.132

Exercise 0B - Sign Test in R

In class we only used the sign test for paired data, but it can also be used as a test about the median of a population.

Again, recall the Deflategate data from previous homework, this time the measurements for the Colts. (One datapoint was modified for illustration purposes.)

colts = c(12.70, 12.75, 12.10, 12.55)

Suppose we wished to test

\[ H_0\colon m_C = 12.5 \quad \text{vs} \quad H_A\colon m_C < 12.5 \]

where \(m_C\) is the median of the PSI of the Colts’ footballs.

Instead of looking at the sign of the difference of paired data, we compare each datapoint to the hypothesized median, in particular noting how many observations are greater than the hypothesized value. (You could alternatively use the number less than, but that would have a reversing affect on the alternative hypothesis.)

(num_inflated = sum(colts > 12.5))
## [1] 3

To carry out the sign test in R we use the binom.test() function where:

binom.test(x = num_inflated, n = length(colts), alternative = "less")
## 
##  Exact binomial test
## 
## data:  num_inflated and length(colts)
## number of successes = 3, number of trials = 4, p-value = 0.9375
## alternative hypothesis: true probability of success is less than 0.5
## 95 percent confidence interval:
##  0.0000000 0.9872585
## sample estimates:
## probability of success 
##                   0.75

Here the output gives a number of things, including most importantly the p-value.

Again, most of the time we would use a two-sided test, which like t.test() is the default for binom.test(). So, suppose that we wished to test

\[ H_0\colon m = 0 \quad \text{vs} \quad H_A\colon m \neq 0 \]

for the following data.

We now longer need to supply an alternative to binom.test() and this time we compare each observation to 0. (Also, with a two sided test, how the test statistic relates to the alternative is less important.)

some_data = c(-0.22, -0.51, 0.12, -0.14, -0.19)
num_pos = sum(some_data > 0)
binom.test(x = num_pos, n = length(some_data))
## 
##  Exact binomial test
## 
## data:  num_pos and length(some_data)
## number of successes = 1, number of trials = 5, p-value = 0.375
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.005050763 0.716417936
## sample estimates:
## probability of success 
##                    0.2

Exercise 0C - The Cauchy Distribution

Whenever you would like to break a statistical procedure, use the Cauchy distribution! It has a couple interesting properties:

x_vals = seq(from = -5, to = 5, length = 10000)
plot(x_vals, dnorm(x_vals), type = "l", lwd = 2, lty = 1, 
     ylim = c(0, 0.45), xlab = "x", ylab = "density", col = "dodgerblue")
lines(x_vals, dcauchy(x_vals), lwd = 2, lty = 3, col = "darkorange")
legend("topleft", legend = c("Normal", "Cauchy"), 
       col = c("dodgerblue", "darkorange"), lwd = 2, lty = c(1, 3))
grid()

Like the \(t\) distribution, the Cauchy distribution has “fatter” tails than a normal distribution. Even “fatter” than a \(t\) distribution, expect when degrees of freedom is 1, in which case the \(t\) and Cauchy are the same!

dnorm(x = 0, mean = 0, sd = 1)
## [1] 0.3989423
dcauchy(x = 0, location = 0, scale = 1)
## [1] 0.3183099
pnorm(q = 2, mean = 0, sd = 1, lower.tail = FALSE)
## [1] 0.02275013
pcauchy(q = 2, location = 0, scale = 1, lower.tail = FALSE)
## [1] 0.1475836

Exercise 0D - Lab Setup

A few things to note for this lab.

The usual \(t\)-test assumes that we are sampling from a normal distribution, which is a symmetric distribution. There for, \(\mu = m\), that is, the mean is equal to the median. That means that the \(t\)-test about

\[ H_0\colon \mu = 0 \quad \text{vs} \quad H_A\colon \mu \neq 0 \]

is the same as

\[ H_0\colon m = 0 \quad \text{vs} \quad H_A\colon m \neq 0 \]

in this case.

The sign test makes no assumption about the population that we are sampling from and tests

\[ H_0\colon m = 0 \quad \text{vs} \quad H_A\colon m \neq 0 \]

Since both tests can be formulated as a test about the median (the sign test cannot necessarily be formatted as a test about the mean, and the Cauchy distribution has no mean) we will view both tests as tests about the median for ease of comparison.

Two functions may be of use to you throughout this lab, calc_power() and gen_p_value(), both given below.

calc_power = function(p_values, alpha = 0.05) {
  mean(p_values < alpha)
}
gen_p_value = function(dist = "norm", test = "t", m = 0, sample_size = 25) {

  # create sample data depending on specified distribution
  if (dist == "norm") {
    sim_data = rnorm(n = sample_size, mean = m)
  }
  if (dist == "cauchy") {
    sim_data = rcauchy(n = sample_size, location = m)
  }

  #perform test depending on requested test
  if (test == "t") {
    p_value = t.test(x = sim_data)$p.value
  }
  if (test == "sign") {
    num_pos = sum(sim_data > 0)
    p_value = binom.test(x = num_pos, n = sample_size)$p.value
  }

  # return p-value
  p_value

}

Throughout this lab we will use samples of size \(n = 25\) and a significance level of \(\alpha = 0.05\) for all tests. Both the \(t\)-test and the sign test will refer to a two-sided test of the median, \(m\), in particular,

\[ H_0\colon m = 0 \quad \text{vs} \quad H_A\colon m \neq 0 \]

Exercise 1 - Test Validity, Sampling From Normal

Suppose data is sampled from a Normal distribution. (In particular standard normal, that is, a mean of 0 and a standard deviation of 1.) Consider testing

\[ H_0\colon m = 0 \quad \text{vs} \quad H_A\colon m \neq 0 \]

using a \(t\)-test and a sign test. (Note in this case that the null hypothesis is true.)

Use 10000 simulated datasets of size \(n = 25\) to determine the validity of each test.

set.seed(42)
# perform and store simulated p-values for t-test here
# perform and store simulated p-values for sign test here,
# for both, use replicate() with gen_p_value()
par(mfrow = c(1, 2))
# create histogram for t-test p-values here
# create histogram for sign test p-values here
# power calculation for t-test p-values here
# power calculation for sign test p-values here

Exercise 2 - Test Validity, Sampling From Cauchy

Suppose data is sampled from a Cauchy distribution. (In particular a Cauchy with a median (location) of 0 and a scale of 1.) Consider testing

\[ H_0\colon m = 0 \quad \text{vs} \quad H_A\colon m \neq 0 \]

using a \(t\)-test and a sign test. (Note in this case that the null hypothesis is true.)

Use 10000 simulated datasets of size \(n = 25\) to determine the validity of each test.

set.seed(42)
# perform and store simulated p-values for t-test here
# perform and store simulated p-values for sign test here
# for both, use replicate() with gen_p_value()
par(mfrow = c(1, 2))
# create histogram for t-test p-values here
# create histogram for sign test p-values here
# power calculation for t-test p-values here
# power calculation for sign test p-values here

Exercise 3 - Comparing Power, Sampling From Normal

Compare the power of the \(t\)-test and the sign test of significance level \(\alpha = 0.05\) when sampling from a Normal distribution. To do so, use \(2500\) simulated samples of size \(n = 25\) from Normal distributions with

That is, you will calculate power six times (once for each true median \(m\)) for the \(t\)-test, then you will repeat this process for the sign test.

Some notes:

set.seed(42)
# perform simulations for t-test here
# store results of using replicate() with gen_p_value() six times (once for each m)
set.seed(42)
# perform simulations for sign test here
# store results of using replicate() with gen_p_value() six times (once for each m)
# calculate power of t-test for each m here
# store results in a vector, perhaps named power_norm_ttst
# order the results according to the m value (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
# calculate power of sign test for each m here
# store results in a vector, perhaps named power_norm_sign
# order the results according to the m value (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
m = c(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
# plot both power "curves" here

Exercise 4 - Comparing Power, Sampling From Cauchy

Compare the power of the \(t\)-test and the sign test of significance level \(\alpha = 0.05\) when sampling from a Cauchy distribution. To do so, use \(2500\) simulated samples of size \(n = 25\) from Cauchy distributions with

That is, you will calculate power six times (once for each true median \(m\)) for the \(t\)-test, then you will repeat this process for the sign test.

Some notes:

set.seed(42)
# perform simulations for t-test here
# store results of using replicate() with gen_p_value() six times (once for each m)
set.seed(42)
# perform simulations for sign test here
# store results of using replicate() with gen_p_value() six times (once for each m)
# calculate power of t-test for each m here
# store results in a vector, perhaps named power_cchy_ttst
# order the results according to the m value (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
# calculate power of sign test for each m here
# store results in a vector, perhaps named power_cchy_sign
# order the results according to the m value (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
m = c(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
# plot both power "curves" here