Exercise 1

Let \(X_1, X_2, \ldots, X_n\) be iid \(N(\theta,1)\) and consider \(\bar{X}_n = \frac{1}{n} \sum_{i=1}^n X_i\). Show that \(\bar{X}_n\) is a consistent estimator of \(\theta\).

Solution:

First, note that \(\bar{X}_n = \frac{1}{n} \sum_{i=1}^n X_i\) is an unbiased estimator as we have seen many times before.

\[ \text{E}\left[\bar{X}_n\right] = \text{E}\left[\sum_{i=1}^n X_i\right] = \frac{1}{n}\cdot n\theta = \theta \]

Also, as we have seen before, the variance of \(\bar{X}_n\) is given by

\[ \text{Var}\left[\bar{X}_n\right] = \frac{\sigma^2}{n} = \frac{1}{n} \]

Since we have an unbiased estimator, we simply need for the variance to vanish as \(n\) goes to infinity. We see that

\[ \lim_{n \to \infty} \text{Var}\left[\bar{X}_n\right] = \lim_{n \to \infty} \frac{1}{n} = 0 \]

Thus \(\bar{X}_n\) is a consistent estimator for \(\theta\).


Exercise 2

Suppose that \(X_1, X_2, \ldots, X_n\) are an iid sample from the distribution

\[ f(x; \theta) = \frac{1}{2}(1+\theta x), \quad -1 < x < 1, -1 < \theta < 1. \]

Show that \(3 \bar{X}_n\) is a consistent estimator of \(\theta\).

Solution:

First, recall that we have previously calculated,

\[ \text{E}\left[X_i\right] = \frac{\theta}{3} \]

and

\[ \text{Var}\left[X_i\right] = \frac{3 - \theta^2}{9} \]

Thus, \(3 \bar{X}_n\) is an unbiased estimator of \(\theta\) since

\[ \text{E}\left[3 \bar{X}_n\right] = 3 \cdot \text{E}\left[X_i\right] = 3 \cdot \frac{\theta}{3} = \theta \]

Then, we calculate the variance of the proposed estimator.

\[ \text{Var}\left[3 \bar{X}_n\right] = 9 \cdot \frac{\text{Var}[X_i]}{n} = 9 \cdot \frac{3 - \theta^2}{9n} = \frac{3 - \theta^2}{n} \]

Then, since \(3 \bar{X}_n\) is an unbiased estimator of \(\theta\) and

\[ \lim_{n \to \infty} \text{Var}\left[3 \bar{X}_n\right] = \lim_{n \to \infty} \frac{3 - \theta^2}{n} = 0 \]

we conclude that \(3 \bar{X}_n\) is a consistent estimator of \(\theta\).


Exercise 3

Let \(Y_1, Y_2, \ldots, Y_n\) be a random sample such that

Suggest a consistent estimator for \(\mu^2\).

Solution:

First note that,

\[ \text{E}\left[\bar{Y}_n\right] = \mu \]

and

\[ \lim_{n \to \infty} \text{Var}\left[\bar{Y}_n\right] = \lim_{n \to \infty} \frac{\sigma^2}{n} = 0 \]

Thus, \(\bar{Y}_n\) is a consistent estimator of \(\mu\).

Therefore, \(\bar{Y}_n^2\) is a consistent estimator of \(\mu^2\).


Exercise 4

Let \(X_1, X_2, \ldots, X_n\) be iid \(N(\mu_X, \sigma^2_X\)). Also, let \(Y_1, Y_2, \ldots, Y_n\) be iid \(N(\mu_Y, \sigma^2_Y\)).

Suggest a consistent estimator for \(\mu_X - \mu_Y\).

Solution:

First note that \(\bar{X}\) is a consistent estimator for \(\mu_X\) and \(\bar{Y}\) is a consistent estimator for \(\mu_Y\).

Because of this, we can say that \(-\bar{Y}\) is a consistent estimator for \(-\mu_Y\). Then finally we can conclude that \(\bar{X} - \bar{Y}\) is a consistent estimator for \(\mu_X - \mu_Y\).

Or, more directly note that

\[ \text{E}\left[\bar{X} - \bar{Y}\right] = \text{E}\left[\bar{X}\right] - \text{E}\left[\bar{Y}\right] = \mu_X - \mu_Y \]

thus \(\bar{X} - \bar{Y}\) is an unbiased estimator for \(\mu_X - \mu_Y\).

Also note that,

\[ \text{Var}\left[\bar{X} - \bar{Y}\right] = \frac{\text{Var}\left[\bar{X}\right]}{n} + \frac{\text{Var}\left[\bar{Y}\right]}{n} = \frac{\sigma^2_X}{n} + \frac{\sigma^2_Y}{n} \]

Then, since

\[ \lim_{n \to \infty} \text{Var}\left[\bar{X} - \bar{Y}\right] = \lim_{n \to \infty} \left( \frac{\sigma^2_X}{n} + \frac{\sigma^2_Y}{n} \right) = 0 \]

we can conclude that \(\bar{X} - \bar{Y}\) is a consistent estimator for \(\mu_X - \mu_Y\).


Exercise 5

Let \(X_1, X_2, \ldots, X_n\) be iid \(N(\mu_X, \sigma^2\)). Also, let \(Y_1, Y_2, \ldots, Y_n\) be iid \(N(\mu_Y, \sigma^2\)). Note that both distributions have the same variance.

Show that

\[ \frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2 + \sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{2n - 2} \]

is a consistent estimator for \(\sigma^2\).

Hint: Note that

\[ \frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2}{\sigma^2} \sim \chi^2_{n - 1} \]

\[ \frac{\sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{\sigma^2} \sim \chi^2_{n - 1} \]

Also, recall that, if \(W \sim \chi^2_k\), then \(\text{E}[W] = k\) and \(\text{Var}[W] = 2k\).

Solution:

First, using the hint, we can show that the estimator is unbiased.

\[ \begin{aligned} \text{E}\left[\frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2 + \sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{2n - 2}\right] &= \text{E}\left[\frac{\sigma^2 \cdot \frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2}{\sigma^2} + \sigma^2 \cdot\frac{\sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{\sigma^2}}{2n - 2} \right] \\[1em] &= \frac{\sigma^2}{2n - 2} \cdot \text{E}\left[\frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2}{\sigma^2}\right] + \frac{\sigma^2}{2n - 2} \cdot \text{E}\left[\frac{\sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{\sigma^2}\right] \\[1em] &= \frac{\sigma^2}{2n - 2} \cdot (n - 1) + \frac{\sigma^2}{2n - 2} \cdot (n - 1) \\[1em] &= \frac{n\sigma^2 - \sigma^2 + n\sigma^2 - \sigma^2}{2n - 2} = \frac{2n - 2}{2n - 2} \cdot \sigma^2 = \sigma^2 \end{aligned} \]

Next, we calculate the variance.

\[ \begin{aligned} \text{Var}\left[\frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2 + \sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{2n - 2}\right] &= \text{Var}\left[\frac{\sigma^2 \cdot \frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2}{\sigma^2} + \sigma^2 \cdot\frac{\sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{\sigma^2}}{2n - 2} \right] \\[1em] &= \left(\frac{\sigma^2}{2n - 2}\right) ^ 2 \cdot \text{Var}\left[\frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2}{\sigma^2}\right] + \left(\frac{\sigma^2}{2n - 2}\right) ^ 2 \cdot \text{Var}\left[\frac{\sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{\sigma^2}\right] \\[1em] &= \left(\frac{\sigma^2}{2n - 2}\right) ^ 2 \cdot 2(n - 1) + \left(\frac{\sigma^2}{2n - 2}\right) ^ 2 \cdot 2(n - 1) \\[1em] &= \frac{2\left(\sigma^2\right)^2}{2n - 2} \end{aligned} \]

Then finally we see that

\[ \lim_{n \to \infty} \text{Var}\left[\frac{\sum_{i = 1}^{n}\left(X_i - \bar{X}\right)^2 + \sum_{i = 1}^{n}\left(Y_i - \bar{Y}\right)^2}{2n - 2}\right] = \lim_{n \to \infty} \frac{2\left(\sigma^2\right)^2}{2n - 2} = 0 \]

Thus the proposed estimator is consistent.


Exercise 6

Let \(Y_1, Y_2, \ldots, Y_n\) be iid observations from a Poisson distribution with parameter \(\lambda\). Show that \(U = \sum_{i=1}^n Y_i\) is sufficient for \(\lambda\).

Solution:

First, recall that the pmf for a Poisson random variable is given by

\[ f(y \mid \lambda) = \frac{\lambda^y e^{-\lambda}}{y!}, \quad y = 0, 1, 2, \ldots, \lambda > 0 \]

Then we have

\[ \begin{aligned} f(y_1, y_2, \ldots, y_n \mid \lambda) &= \prod_{i = 1}^{n} \frac{\lambda^{y_i} e^{-\lambda}}{y_i!} \\[1em] &= \frac{\lambda^{\sum_{i = 1}^{n} y_i}e^{-n\lambda}}{\displaystyle\prod_{i = 1}^{n} y_i!} \\[1em] &= \lambda^ue^{-n\lambda}\frac{1}{\prod_{i = 1}^{n} y_i!} \\ \end{aligned} \]

Here we have

\[ \begin{aligned} U & = \sum_{i = 1}^{n} Y_i\\[1em] g(u, \lambda) &= \lambda^ue^{-n\lambda}\\[1em] h(y_1, y_2, \ldots, y_n) &= \frac{1}{\prod_{i = 1}^{n} y_i!} \\ \end{aligned} \]

Thus, by the Factorization Theorem,

\[ U = \sum_{i = 1}^{n} Y_i \]

is a sufficient statistic for \(\lambda\).


Exercise 7

Let \(X_1, X_2, \ldots, X_n\) be iid observations from a distribution with density

\[ f(x \mid \theta) = \frac{\theta}{(1+x)^{\theta + 1}}, \quad 0< \theta < \infty, 0< x <\infty \]

Find a sufficient statistic for \(\theta\).

Solution:

First note that

\[ \begin{aligned} f(x_1, x_2, \ldots, x_n \mid \theta) &= \prod_{i = 1}^{n}\frac{\theta}{(1+x_i)^{\theta + 1}} \\[1em] &= \frac{\theta^n}{\left[\displaystyle\prod_{i = 1}^{n}(1 + x_i)\right]^{\theta + 1}} \\[1em] &= \frac{\theta^n}{u ^ {\theta + 1}} \\ \end{aligned} \]

Here we have

\[ \begin{aligned} U & = \prod_{i = 1}^{n}(1 + X_i)\\ g(u, \theta) &= \frac{\theta^n}{u ^ {\theta + 1}}\\ h(x_1, x_2, \ldots, x_n) &= 1 \\ \end{aligned} \]

Thus, by the Factorization Theorem,

\[ U = \prod_{i = 1}^{n}(1 + X_i) \]

is a sufficient statistic for \(\theta\).


Exercise 8

Let \(X_1, X_2, \ldots, X_n\) be iid observations from a normal distribution with a unknown mean, \(\mu\), and known variance \(\sigma^2 = 9\).

Show that \(\sum_{i = 1}^{n}X_i\) is a sufficient statistic for \(\mu\), then use this statistic to create an estimator that is both unbiased and sufficient for estimating \(\mu\).

Solution:

First recall that the pdf for a normal distribution is given by

\[ f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left[-\frac{1}{2}\left(\frac{x -\mu}{\sigma}\right)^2\right], \quad x \in \mathbb{R}, \mu \in \mathbb{R}, \sigma^2 > 0 \]

In this case, with \(\sigma^2 = 9\), we have

\[ f(x \mid \mu) = \frac{1}{3\sqrt{2\pi}}\exp\left[-\frac{1}{18}\left(x -\mu\right)^2\right] \]

Now note that

\[ \begin{aligned} f(x_1, x_2, \ldots, x_n \mid \theta) &= \prod_{i = 1}^{n} \frac{1}{3\sqrt{2\pi}}\exp\left[-\frac{1}{18}\left(x_i -\mu\right)^2\right] \\[1em] &= \left(\frac{1}{3\sqrt{2\pi}}\right)^n \prod_{i = 1}^{n} \exp\left[-\frac{1}{18}\left(x_i^2 - 2\mu x_i+\mu^2\right)\right] \\[1em] &= \left(\frac{1}{3\sqrt{2\pi}}\right)^n \exp\left[{-\frac{1}{18}}\sum_{i = 1}^{n}x_i^2\right] \exp\left[\frac{2\mu}{18}\sum_{i = 1}^{n}x_i\right] \exp\left[-\frac{1}{18}n\mu^2\right]\\[1em] &= \left(\frac{1}{3\sqrt{2\pi}}\right)^n \exp\left[{-\frac{1}{18}}\sum_{i = 1}^{n}x_i^2\right] \exp\left[\frac{2\mu}{18}u\right] \exp\left[-\frac{1}{18}n\mu^2\right]\\[1em] \end{aligned} \]

Here we have

\[ \begin{aligned} U & = \sum_{i = 1}^{n}X_i\\[1em] g(u, \mu) &= \exp\left[\frac{2\mu}{18}u\right] \exp\left[-\frac{1}{18}n\mu^2\right]\\[1em] h(x_1, x_2, \ldots, x_n) &= \left(\frac{1}{3\sqrt{2\pi}}\right)^n \exp\left[{-\frac{1}{18}}\sum_{i = 1}^{n}x_i^2\right] \\ \end{aligned} \]

Thus, by the Factorization Theorem,

\[ U = \sum_{i = 1}^{n}X_i \]

is a sufficient statistic for \(\mu\).

Then, because \(f(u) = \frac{1}{n}u\) is a one-to-one function,

\[ U^\star = \frac{1}{n}\sum_{i = 1}^{n}X_i \]

is also a sufficient statistic for \(\mu\), which also happens to be unbiased, which we have seen many times before.


Exercise 9

Let \(Y_1, Y_2, \ldots, Y_n\) be iid observations from a distribution with density

\[ f(y \mid \beta) = \frac{y}{\beta}\cdot\exp\left(\frac{-y^2}{2\beta}\right), \quad y \geq 0, \beta > 0 \]

Find a sufficient statistic for \(\beta\).

Solution:

First note that

\[ \begin{aligned} f(y_1, y_2, \ldots, y_n \mid \theta) &= \prod_{i = 1}^{n} \frac{y_i}{\beta}\cdot\exp\left(\frac{-y_i^2}{2\beta}\right)\\[1em] &= \beta^{-n} \left(\prod_{i = 1}^{n} y_i\right) \exp \left[-\frac{\sum_{i = 1}^{n}y_i^2}{2\beta}\right] \\[1em] &= \beta^{-n} \left(\prod_{i = 1}^{n} y_i\right) \exp \left[-\frac{u}{2\beta}\right] \\ \end{aligned} \]

Here we have

\[ \begin{aligned} U & = \sum_{i = 1}^{n}Y_i^2\\[1em] g(u, \beta) &= \beta^{-n} \exp \left[-\frac{u}{2\beta}\right]\\[1em] h(y_1, y_2, \ldots, y_n) &= \prod_{i = 1}^{n} y_i \\ \end{aligned} \]

Thus, by the Factorization Theorem,

\[ U = \sum_{i = 1}^{n}Y_i^2 \]

is a sufficient statistic for \(\beta\).


Exercise 10

Let \(X_1, X_2, \ldots, X_n\) be iid observations from a distribution with density

\[ f(x \mid \alpha, \beta) = \frac{\alpha}{\beta}\left(\frac{x}{\beta}\right)^{\alpha - 1}e^{-(x/\beta)^\alpha}, \quad x \geq 0, \alpha > 0, \beta > 0 \]

Let \(\alpha\) be a known constant and \(\beta\) be unknown. Find a sufficient statistic for \(\beta\).

Solution:

First note that

\[ \begin{aligned} f(x_1, x_2, \ldots, x_n \mid \theta) &= \prod_{i = 1}^{n} \frac{\alpha}{\beta}\left(\frac{x}{\beta}\right)^{\alpha - 1}e^{-(x/\beta)^\alpha} \\[1em] &= \alpha^{n}\beta^{-n} \frac{\prod_{i = 1}^{n} x_i ^ {\alpha - 1}}{\prod_{i = 1}^{n} \beta^{\alpha - 1}} \exp\left[ - \frac{\sum_{i = 1}^{n}x_i^\alpha}{\beta^\alpha} \right] \\[1em] &= \alpha^{n}\beta^{-n} \left(\prod_{i = 1}^{n} x_i\right) ^ {\alpha - 1} \beta^{n - n\alpha} \exp\left[ - \frac{\sum_{i = 1}^{n}x_i^\alpha}{\beta^\alpha} \right] \\[1em] & = \alpha^{n}\beta^{-n\alpha} \left(\prod_{i = 1}^{n} x_i\right) ^ {\alpha - 1} \exp\left[ - \frac{\sum_{i = 1}^{n}x_i^\alpha}{\beta^\alpha} \right] \\[1em] & = \alpha^{n}\beta^{-n\alpha} \left(\prod_{i = 1}^{n} x_i\right) ^ {\alpha - 1} \exp\left[ - \frac{u}{\beta^\alpha} \right] \end{aligned} \]

Here we have

\[ \begin{aligned} U & = \sum_{i = 1}^{n}X_i^\alpha\\[1em] g(u, \beta) &= \beta^{-n\alpha} \exp\left[ - \frac{u}{\beta^\alpha} \right]\\[1em] h(x_1, x_2, \ldots, x_n) &= \alpha^{n}\left(\prod_{i = 1}^{n} x_i\right) ^ {\alpha - 1} \\ \end{aligned} \]

Thus, by the Factorization Theorem,

\[ U = \sum_{i = 1}^{n}X_i^\alpha\\[1em] \]

is a sufficient statistic for \(\beta\).