Consider independent random variables \(X_1\), \(X_2\), and \(X_3\) with
(a) Calculate \(\text{E}[5 X_1 + 2]\).
Solution:
\[ \text{E}[5 X_1 + 2] = 5 \cdot \text{E}[X_1] + 2 = \boxed{7} \]
(b) Calculate \(\text{E}[4 X_1 + 2 X_2 - 6 X_3]\).
Solution:
\[ \begin{aligned} \text{E}[4 X_1 + 2 X_2 - 6 X_3] &= 4 \cdot \text{E}[X_1] + 2 \cdot \text{E}[X_2] - 6 \cdot \text{E}[X_3] \\ &= 4 \cdot (1) + 2 \cdot (2) - 6 \cdot (3) = \boxed{-10} \end{aligned} \]
(c) Calculate \(\text{Var}[5 X_1 + 2]\).
Solution:
\[ \text{Var}[5 X_1 + 2] = 5^2 \cdot \text{Var}[X_1] = 25 \cdot (4) = \boxed{100} \]
(d) Calculate \(\text{Var}[4 X_1 + 2 X_2 - 6 X_3]\).
Solution:
\[ \begin{aligned} \text{Var}[4 X_1 + 2 X_2 - 6 X_3] &= 4^2 \cdot \text{Var}[X_1] + 2^2 \cdot \text{Var}[X_2] + (-6)^2 \cdot \text{Var}[X_3] \\ &= 4^2 \cdot (4) + 2^2 \cdot (3^2) + (-6)^2 \cdot (5^2) = \boxed{1000} \end{aligned} \]
Consider random variables \(H\) and \(Q\) with
(a) Calculate \(\text{E}[5H^2 - 10]\).
Solution:
\[ \begin{aligned} \text{Var}[H] &= \text{E}\left[H^2\right] - \left(\text{E}[H]\right)^2 \\ 16 &= \text{E}\left[H^2\right] - 3^2 \text{E}\left[H^2\right] = 25 \end{aligned} \]
\[ \text{E}\left[5H^2 - 10\right] = 5 \cdot \text{E}\left[H^2\right] - 10 = 5 \cdot 25 - 10 = \boxed{115} \]
(b) Calculate \(\text{E}[Q]\).
Solution:
\[ \text{E}\left[Q^2\right] = 16 \]
\[ \text{Var}[Q] = \left(\text{Var}[Q]\right)^2 = 16 \]
\[ \begin{aligned} \text{Var}[Q] &= \text{E}\left[Q^2\right] - \left(\text{E}[Q]\right)^2 \\ 16 &= 16 - \text{E}[Q]^2 \end{aligned} \]
\[ \text{E}[Q] = 0 \]
Consider a random variable \(S\) with probability density function
\[ f(s) = \frac{1}{9000}(2s + 10), \ \ 40 \leq s \leq 100. \]
(a) Calculate \(\text{E}[S]\).
Solution:
\[ \text{E}[S] = \int_{40}^{100} s \cdot \frac{2s + 10}{9000}\ ds = \boxed{74} \]
(b) Calculate \(\text{SD}[S]\).
Solution:
\[ \text{E}[S^2] = \int_{40}^{100} s^2 \cdot \frac{2s + 10}{9000}\ ds = 5760 \]
\[ \text{Var}[S] = \text{E}[S^2] - \left(\text{E}[S] \right) ^ 2 = 5760 - 74^2 = 284 \]
\[ \text{SD}[S] = \sqrt{\text{Var}[S]} = \sqrt{284} \approx \boxed{16.8523} \]
Consider independent random variables \(X\) and \(Y\) with
(a) Calculate \(P[X > 5]\).
Solution:
\[ P[X > 5] = P\left[\frac{X -\mu_X}{\sigma_X} > \frac{5 - 2}{3}\right] = P[Z > 1] = 1 - P[Z < 1] = 1 - 0.8413 = \boxed{0.1587} \]
pnorm(q = 5, mean = 2, sd = sqrt(9), lower.tail = FALSE)## [1] 0.15865531 - pnorm(1)## [1] 0.1586553(b) Calculate \(P[X + 2Y > 5]\).
Solution:
\[ \text{E}[X + 2Y] = 1 \cdot \text{E}[X] + 2 \cdot \text{E}[Y] = 12 \]
\[ \text{Var}[X + 2Y] = 1^2 \cdot \text{Var}[X] + 2^2 \cdot \text{Var}[Y] = 25 \]
\[ X + 2Y \sim N(\mu_{X+2Y} = 12, \sigma^2_{X+2Y} = 25) \]
\[ P[X + 2Y] = P\left[\frac{X + 2Y -\mu_{X + 2Y}}{\sigma_{X + 2Y}} > \frac{5 - 12}{5}\right] = P[Z > -1.4] = 1 - P[Z < -1.4] = 1 - 0.0808 = \boxed{0.9192} \]
pnorm(q = 5, mean = 12, sd = sqrt(25), lower.tail = FALSE)## [1] 0.91924331 - pnorm(-1.4)## [1] 0.9192433Consider random variables \(Y_1\), \(Y_2\), and \(Y_3\) with
(a) Calculate \(\text{Var}[3Y_1 - 2Y_2]\).
Solution:
\[ \text{Var}[3Y_1 - 2Y_2] = 3^2 \cdot \text{Var}[Y_1] + (-2)^2 \cdot \text{Var}[Y_2] + 2(3)(-2) \cdot \text{Cov}[Y_1, Y_2] = \boxed{48} \]
(b) Calculate \(\text{Var}[3Y_1 - 4Y_2 + 2Y_3]\).
Solution:
\[ \begin{aligned} \text{Var}[3Y_1 - 4Y_2 + 2Y_3] &= (3)^2 \cdot \text{Var}[Y_1] + (-4)^2 \cdot \text{Var}[Y_2] + (2)^2 \cdot \text{Var}[Y_3] \\ &+ (2)(3)(-4) \cdot \text{Cov}[Y_1, Y_2] + (2)(3)(2) \cdot \text{Cov}[Y_1, Y_3] + (2)(-4)(2) \cdot \text{Cov}[Y_2, Y_3] \\ & = \boxed{128} \end{aligned} \]
Consider using \(\hat{\xi}\) to estimate \(\xi\).
(a) If \(\text{Bias}[\hat{\xi}] = 5\) and \(\text{Var}[\hat{\xi}] = 4\), calculate \(\text{MSE}[\hat{\xi}]\)
Solution:
\[ \text{MSE}[\hat{\xi}] = \left( \text{bias}[\hat{\xi}] \right)^2 + \text{var}[\hat{\xi}] = 5 ^ 2 + 4 = \boxed{29} \]
(b) If \(\hat{\xi}\) is unbiased, \(\xi = 6\), and \(\text{MSE}[\hat{\xi}] = 30\), calculate \(\text{E}\left[\hat{\xi}^2\right]\)
Solution:
First we use the definition of \(\text{MSE}\) to recover the variance of \(\hat{\xi}\).
\[ \begin{aligned} \text{MSE}[\hat{\xi}] &= \left( \text{bias}[\hat{\xi}] \right)^2 + \text{var}[\hat{\xi}] \\ 30 &= 0 + \text{var}[\hat{\xi}] \\ \text{var}[\hat{\xi}] &= 30 \\ \end{aligned} \]
Then using the fact that the estimator is unbiased, we have
\[ \text{E}\left[\hat{\xi}^2\right] = \text{var}\left[\hat{\xi}\right] + \left(\text{E}\left[\hat{\xi}\right]\right)^2 = 30 + 6^2 = \boxed{66} \]
Using the identity
\[ (\hat{\theta}-\theta) = \left(\hat{\theta}-\text{E}[\hat{\theta}]\right) + \left(\text{E}[\hat{\theta}] - \theta\right) = \left(\hat{\theta} - \text{E}[\hat{\theta}]\right) + \text{Bias}[\hat{\theta}] \]
show that
\[ \text{MSE}[\hat{\theta}] = \text{E}\left[(\hat{\theta} - \theta)^2\right] = \text{Var}[\hat{\theta}] + \left(\text{Bias}[\hat{\theta}]\right)^2 \]
Solution:
We compute:
\[ \begin{aligned} \text{MSE}[\hat{\theta}] & = \text{E}\left[(\hat{\theta}-\theta)^2\right] \\ & = \text{E}\left[ \left(\hat{\theta} - \text{E}[\hat{\theta}] + \text{E}[\hat{\theta}] - \theta\right)^2\right] ~~(\text{ using the given identity}) \\ & = \text{E}\left[ \left(\hat{\theta} - \text{E}[\hat{\theta}]\right)^2 + 2 \cdot \left(\hat{\theta} - \text{E}[\hat{\theta}]\right) \left(\text{E}[\hat{\theta}] - \theta\right) + \left(\text{E}[\hat{\theta}] - \theta\right)^2\right] ~~\text{ (algebra)}\\ & = \text{E}\left[\left(\hat{\theta} - \text{E}[\hat{\theta}]\right)^2\right] +2 \cdot \left(\text{E}[\hat{\theta}] - \theta\right)\left(\text{E}[\hat{\theta} - \text{E}[\hat{\theta}]\right) + \left(\text{E}[\hat{\theta}] - \theta\right)^2 ~~\text{ (linearity of expectation)} \end{aligned} \]
where \(\left(\text{E}[\hat{\theta}] - \theta\right)\) in the last line is a constant.
Note that:
\(\text{E}\left[\left(\hat{\theta} - \text{E}[\hat{\theta}]\right)^2\right] = \text{Var}(\hat{\theta})\), by the definition of variance
\(\left(\text{E}[\hat{\theta}] - \theta\right)^2 = \text{Bias}[\hat{\theta}]^2\), by definition of bias
Substituting these into the last expression above, we have
\[ \text{MSE}[\hat{\theta}] = \text{Var}[\hat{\theta}] + \left(\text{Bias}[\hat{\theta}]\right)^2 \]
Let \(X_1, X_2, \ldots, X_n\) denote a random sample from a population with mean \(\mu\) and variance \(\sigma^2\).
Consider three estimators of \(\mu\):
\[ \hat{\mu}_1 = \frac{X_1 + X_2 + X_3}{3}, ~~~\hat{\mu}_2 = \frac{X_1}{4} + \frac{X_2 + \cdots + X_{n - 1}}{2(n - 2)} + \frac{X_n}{4}, ~~~\hat{\mu}_3 = \bar{X}, \]
Calculate the mean squared error for each estimator. (It will be useful to first calculate their bias and variances.)
Solution:
We first calculate the expected value of each estimator.
\[ \begin{aligned} \text{E}\left[\hat{\mu}_1\right] &= \text{E}\left[ \frac{X_1 + X_2 + X_3}{3}\right] = \frac{1}{3}(\mu + \mu + \mu ) = \mu \\ \text{E}\left[\hat{\mu}_2\right] &= \text{E}\left[ \frac{X_1}{4} + \frac{X_2 + \cdots + X_{n - 1}}{2(n - 2)} + \frac{X_n}{4} \right] = \frac{\mu}{4} + \frac{(n-2)\mu}{2(n-2)} + \frac{\mu}{4} = \frac{\mu}{4} + \frac{\mu}{2} + \frac{\mu}{4} = \mu \\ \text{E}\left[\hat{\mu}_3\right] &= \text{E}\left[\bar{X}\right] = \mu \end{aligned} \]
Thus we see that each estimator is unbiased.
Next we calculate the variance of each estimator.
\[ \begin{aligned} \text{Var}\left[\hat{\mu}_1\right] &= \text{Var}\left[ \frac{X_1 + X_2 + X_3}{3}\right] = \frac{1}{9}\left(\sigma^2 + \sigma^2 + \sigma^2\right) = \frac{\sigma^2}{3} \\ \text{Var}\left[\hat{\mu}_2\right] &= \text{Var}\left[ \frac{X_1}{4} + \frac{X_2 + \cdots + X_{n - 1}}{2(n - 2)} + \frac{X_n}{4} \right] = \frac{\sigma^2}{16} + \frac{(n-2)\sigma^2}{4(n-2)^2} + \frac{\sigma^2}{16} = \frac{n\sigma^2}{8(n - 2)} \\ \text{Var}\left[\hat{\mu}_3\right] &= \text{Var}\left[\bar{X}\right] = \frac{\sigma^2}{n} \end{aligned} \]
Since each estimator is unbiased, and for any estimator
\[ \text{MSE}\left[\hat{\theta}\right] = \text{E}\left[(\hat{\theta} - \theta) ^ 2\right] = \left( \text{bias}\left[\hat{\theta}\right] \right)^2 + \text{var}\left[\hat{\theta}\right] \]
we see that the resulting mean squared errors are simply the variances.
\[ \begin{aligned} \text{MSE}\left[\hat{\mu}_1\right] &= \boxed{\frac{\sigma^2}{3}} \\ \text{MSE}\left[\hat{\mu}_2\right] &= \boxed{\frac{n\sigma^2}{8(n - 2)}} \\ \text{MSE}\left[\hat{\mu}_3\right] &= \boxed{\frac{\sigma^2}{n}} \end{aligned} \]
Let \(X_1, X_2, \ldots, X_n\) denote a random sample from a distribution with density
\[ f(x) = \frac{1}{\theta}e^{-x/\theta}, ~~x > 0, \theta \geq 0 \]
Consider five estimators of \(\theta\):
\[ \hat{\theta}_1 = X_1, ~~~\hat{\theta}_2 = ~~~\frac{X_1 + X_2}{2}, ~~~\hat{\theta}_3 = ~~~\frac{X_1 + 2X_2}{3}, ~~~\hat{\theta}_4 = \bar{X}, ~~~\hat{\theta}_5 = 5 \]
Calculate the mean squared error for each estimator. (It will be useful to first calculate their bias and variances.)
Solution:
First, notice that this is an exponential distribution, thus we have
\[ \text{E}[X_i] = \theta, ~~~\text{Var}[X_i] = \theta^2 \]
Following the work from the previous exercise, we will find
\[ \begin{aligned} \text{MSE}\left[\hat{\theta}_1\right] &= \boxed{\theta^2} \\ \text{MSE}\left[\hat{\theta}_2\right] &= \boxed{\frac{\theta^2}{2}} \\ \text{MSE}\left[\hat{\theta}_3\right] &= \boxed{\frac{5\theta^2}{9}} \\ \text{MSE}\left[\hat{\theta}_4\right] &= \boxed{\frac{\theta^2}{n}} \\ \text{MSE}\left[\hat{\theta}_5\right] &= \boxed{\left(5 - \theta\right)^2} \\ \end{aligned} \]
Note that the first four estimators are unbiased, and thus their mean squared error is simply their variance. The fifth estimator has no variance, and thus its mean squared error is its bias squared.
Suppose that \(\text{E}\left[\hat{\theta}_1\right] = \text{E}\left[\hat{\theta}_2\right] = \theta\), \(\text{Var}\left[\hat{\theta}_1\right] = \sigma_1^2\), \(\text{Var}\left[\hat{\theta}_2\right] = \sigma_2^2\), and \(\text{Cov}\left[\hat{\theta}_1, \hat{\theta}_2\right] = \sigma_{12}\). Consider the unbiased estimator
\[ \hat{\theta}_3 = a\hat{\theta}_1 + (1-a)\hat{\theta}_2. \]
If \(\hat{\theta}_1\) and \(\hat{\theta}_2\) are independent, what value should be chosen for the constant \(a\) in order to minimize the variance and thus mean squared error of \(\hat{\theta}_3\) as an estimator of \(\theta\)?
Solution:
We first verify that \(\hat{\theta}_3\) is unbiased.
\[ \text{E}\left[\hat{\theta}_3\right] = \text{E}\left[a\hat{\theta}_1+(1-a)\hat{\theta}_2\right] = a\text{E}\left[\hat{\theta}_1\right] +(1-a)\text{E}\left[\hat{\theta}_2\right] = a\theta + (1-a) \theta = \theta \]
If \(\hat{\theta}_1\) and \(\hat{\theta}_2\) are independent, then
\[ \text{Var}\left[\hat{\theta}_3\right] = \text{Var}\left[a\hat{\theta}_1+(1-a)\hat{\theta}_2\right] = a^2\text{Var}\left[\hat{\theta}_1\right] +(1-a)^2\text{Var}\left[\hat{\theta}_2\right] = a^2 \sigma^2_1 + (1-a)^2 \sigma^2_2 \]
Thus we want to find \(a\) to minimize the function
\[f(a) = a^2 \sigma^2_1 + (1-a)^2 \sigma^2_2\]
Taking the derivative, we have
\[ f'(a) = 2a \sigma^2_1 - 2(1-a) \sigma^2_2 \]
Setting this equal to 0 and solving for \(a\) gives
\[ a = \frac{\sigma^2_2}{\sigma^2_1 + \sigma^2_2} \]
Finally, we check that this gives a minimum by evaluating the second derivative.
\[ f''(a) = 2\sigma^2_1+2\sigma^2_2 \]
Since since this is in fact always positive
\[ \boxed{a = \frac{\sigma^2_2}{\sigma^2_1 + \sigma^2_2}} \]
gives a minimum.
Let \(Y\) have a binomial distribution with parameters \(n\) and \(p\). Consider two estimators for \(p\):
\[ \hat{p}_1 = \frac{Y}{n} \]
and
\[ \hat{p}_2 = \frac{Y + 1}{n + 2} \]
For what values of \(p\) does \(\hat{p}_2\) achieve a lower mean square error than \(\hat{p}_1\)?
Solution:
Recall that for a binomial random variable we have
\[ \begin{aligned} \text{E}\left[Y\right] &= np \\ \text{Var}\left[Y\right] &= np(1 - p) \\ \end{aligned} \]
We first calculate the bias, variance, and mean squared error of \(\hat{p}_1\).
\[ \begin{aligned} \text{Bias}\left[\hat{p}_1\right] &= \text{E}\left[\hat{p}_1\right] - p = p - p = 0 \\ \text{Var}\left[\hat{p}_1\right] &= \frac{1}{n^2}\text{Var}\left[Y\right] = \frac{np(1-p)}{n^2} = \frac{p(1-p)}{n} \\ \text{MSE}\left[\hat{p}_1\right] &= \frac{p(1-p)}{n} \\ \end{aligned} \]
Then we first calculate the bias, variance, and mean squared error of \(\hat{p}_2\).
\[ \begin{aligned} \text{E}\left[\hat{p}_2\right] &= \text{E}\left[\frac{Y + 1}{n + 2}\right] = \frac{np + 1}{n + 2} \\ \text{Bias}\left[\hat{p}_2\right] &= \text{E}\left[\hat{p}_2\right] - p = \frac{np + 1}{n + 2} - p\\ \text{Var}\left[\hat{p}_2\right] &= \text{Var}\left[\frac{Y + 1}{n + 2}\right] = \frac{1}{(n + 2)^2}\text{Var}\left[Y\right] = \frac{np(1-p)}{(n+2)^2}\\ \text{MSE}\left[\hat{p}_2\right] &= \left(\text{Bias}\left[\hat{p}_2\right]\right)^2 + \text{Var}\left[\hat{p}_2\right] = \left(\frac{np + 1}{n + 2} - p\right)^2 + \frac{np(1-p)}{(n+2)^2}\\ \end{aligned} \]
Finally, we solve for values of \(\mu\) where \(\text{MSE}\left[\hat{p}_2\right] < \text{MSE}\left[\hat{p}_1\right].\) We have,
\[ \begin{aligned} \text{MSE}\left[\hat{p}_2\right] &< \text{MSE}\left[\hat{p}_1\right]\\[1em] \left(\frac{np + 1}{n + 2} - p\right)^2 + \frac{np(1-p)}{(n+2)^2} &< \frac{p(1-p)}{n} \end{aligned} \]
Now we decide to be lazy and use WolframAlpha.
Thus, \(\text{MSE}\left[\hat{p}_2\right] < \text{MSE}\left[\hat{p}_1\right]\) when
\[ \boxed{\frac{1}{2}\left( 1 - \sqrt{\frac{n + 1}{2n + 1}} \right) < p < \frac{1}{2}\left( 1 + \sqrt{\frac{n + 1}{2n + 1}} \right)} \]
Note that this interval is symmetric about 0.5, which makes sense because \(\hat{p}_2\) is biased towards 0.5
Let \(X_1, X_2, \ldots, X_n\) denote a random sample from a population with mean \(\mu\) and variance \(\sigma^2\).
Create an unbiased estimator for \(\mu^2\). Hint: Start with \(\bar{X}^2\).
Solution:
First note that from previous results we know that
\[ \text{E}\left[\bar{X}\right] = \mu, \quad \text{Var}\left[\bar{X}\right] = \frac{\sigma^2}{n} \]
Also, recall that re-arranging the variance formula based on the first and second moments, for any random variable we have
\[ \text{E}\left[Y^2\right] = \text{Var}[Y] + \left(\text{E}[Y]\right) ^ 2 \]
Using these, we find that
\[ \text{E}\left[\bar{X}^2\right] = \text{Var}\left[\bar{X}\right] + \left(\text{E}\left[\bar{X}\right]\right) ^ 2 = \frac{\sigma^2}{n} + \mu^2 \]
This expression has the \(\mu^2\) that we’re after, but also has an extra term that we would like to disappear.
Recall that
\[ \text{E}\left[S^2\right] = \sigma^2 \]
Let’s now consider the estimator
\[ \hat{\xi} = \bar{X}^2 - \frac{1}{n}S^2 \]
It’s expected value is
\[ \text{E}\left[\hat{\xi}\right] = \text{E}\left[\bar{X}^2 - \frac{1}{n}S^2\right] = \text{E}\left[\bar{X}^2\right] - \frac{1}{n} \text{E}\left[S^2\right] = \frac{\sigma^2}{n} + \mu^2 - \frac{1}{n}\sigma^2 = \mu^2 \]
Thus \(\boxed{\hat{\xi} = \bar{X}^2 - \frac{1}{n}S^2}\) is an unbiased estimator for \(\mu^2\).
Let \(X_1, X_2, X_3, \ldots, X_n\) be iid random variables form \(\text{U}(\theta, \theta + 2)\). (That is, a uniform distribution with a minimum of \(\theta\) and a maximum of \(\theta + 2\).)
Consider the estimator
\[ \hat{\theta} = \frac{1}{n}\sum_{i = 1}^{n}X_i = \bar{X} \]
(a) Calculate the bias of \(\hat{\theta}\) when estimating \(\theta\).
Solution:
\[ \text{E}\left[\hat{\theta}\right] = \text{E}\left[\bar{X}\right] = \text{E}[X] = \frac{\theta + (\theta + 2)}{2} = \theta + 1 \]
\[ \text{bias}\left[\hat{\theta}\right] = \text{E}\left[\hat{\theta}\right] - \theta = (\theta + 1) - \theta = \boxed{1} \]
(b) Calculate the variance of \(\hat{\theta}\).
Solution:
\[ \text{Var}\left[\hat{\theta}\right] = \text{Var}\left[\bar{X}\right] = \frac{\text{Var}[X]}{n} = \frac{[(\theta + 2) - \theta]^2}{12n} = \boxed{\frac{1}{3n}} \]
(c) Calculate the mean squared error of \(\hat{\theta}\) when estimating \(\theta\).
Solution:
\[ \text{MSE}\left[\hat{\theta}\right] = \left( \text{bias}\left[\hat{\theta}\right] \right)^2 + \text{var}\left[\hat{\theta}\right] = 1^2 + \frac{1}{3n} = \boxed{\frac{3n + 1}{3n}} \]