Let \(X_{1}, X_{2}, \ldots X_{n}\) be a random sample of size \(n\) from a distribution with probability density function

\[ f(x, \theta) = \frac{1}{\theta}e^{-x/\theta}, \quad x > 0, \ \theta > 0 \]

Note that, the moments of this distribution are given by

\[ E[X^k] = \int_{0}^{\infty} \frac{x^k}{\theta}e^{-x/\theta} = k! \cdot \theta^k. \]

This will be a useful fact for Exercises 2 and 3.

**(a)** Obtain the maximum likelihood *estimator* of \(\theta\), \(\hat{\theta}\). (This should be a function of the unobserved \(x_i\) and the sample size \(n\).) Calculate the *estimate* when

\[ x_{1} = 0.50, \ x_{2} = 1.50, \ x_{3} = 4.00, \ x_{4} = 3.00. \]

(This should be a single number, for this dataset.)

**Solution:**

We first obtain the likelihood by **multiplying** the probability density function for each \(X_i\). We then **simplify** this expression.

\[ L(\theta) = \prod_{i = 1}^{n} f(x_i; \theta) = \prod_{i = 1}^{n} \frac{1}{\theta}e^{-x_i/\theta} = \theta^{-n}\exp\left({\frac{-\sum_{i = i}^{n} x_i}{\theta}}\right) \]

Instead of directly maximizing the likelihood, we instead maximize the **log**-likelihood.

\[ \log L(\theta) = -n \log \theta - \frac{\sum_{i = i}^{n} x_i}{\theta} \]

To maximize this function, we take a **derivative** with respect to \(\theta\).

\[ \frac{d}{d\theta} \log L(\theta) = \frac{-n}{\theta} + \frac{\sum_{i = i}^{n} x_i}{\theta^2} \]

We set this derivative equal to **zero**, then **solve** for \(\theta\).

\[ \frac{-n}{\theta} + \frac{\sum_{i = i}^{n} x_i}{\theta^2} = 0 \]

Solving gives our *estimator*, which we denote with a **hat**.

\[ \hat{\theta} = \frac{\sum_{i = i}^{n} x_i}{n} = \bar{x} \]

Using the given data, we obtain an *estimate*.

\[ \hat{\theta} = \frac{0.50 + 1.50 + 4.00 + 3}{4} = \boxed{2.25} \]

**(b)** Calculate the bias of the maximum likelihood *estimator* of \(\theta\), \(\hat{\theta}\). (This will be a number.)

**Solution:**

Note that we have an exponential distribution.

\[ E[X_i] = \theta \]

\[ \text{Var}[X_i] = \theta^2 \]

\[\begin{align*} \text{Bias}(\hat{\theta}) &= E[\hat{\theta}] - \theta \\[1.5ex] &= E\left[\frac{\sum_{i = i}^{n} X_i}{n}\right] - \theta \\ &= \frac{1}{n} \sum_{i = i}^{n} E[X_i] - \theta \\ &= \frac{1}{n} n\theta - \theta \\ &= \theta - \theta = \boxed{0} \end{align*}\]**(c)** Find the mean squared error of the maximum likelihood *estimator* of \(\theta\), \(\hat{\theta}\). (This will be an expression based on the parameter \(\theta\) and the sample size \(n\). Be aware of your answer to the previous part, as well as the distribution given.)

**Solution:**

**(d)** Provide an *estimate* for \(P[X > 4]\) when

\[ x_{1} = 0.50, \ x_{2} = 1.50, \ x_{3} = 4.00, \ x_{4} = 3.00. \]

**Solution:**

\[ P[X > 4] = e^{-4 / \theta} \]

\[ \hat{P}[X > 4] = e^{-4 / \hat{\theta}} = e^{-4 / 2.25} = \boxed{0.1690} \]

Let \(X_{1}, X_{2}, \ldots X_{n}\) be a random sample of size \(n\) from a distribution with probability density function

\[ f(x, \alpha) = \alpha^{-2}xe^{-x/\alpha}, \quad x > 0, \ \alpha > 0 \]

**(a)** Obtain the maximum likelihood *estimator* of \(\alpha\), \(\hat{\alpha}\). Calculate the *estimate* when

\[ x_{1} = 0.25, \ x_{2} = 0.75, \ x_{3} = 1.50, \ x_{4} = 2.5, \ x_{5} = 2.0. \]

**Solution:**

We first obtain the likelihood by **multiplying** the probability density function for each \(X_i\). We then **simplify** this expression.

\[ L(\alpha) = \prod_{i = 1}^{n} f(x_i; \alpha) = \prod_{i = 1}^{n} \alpha^{-2} x_i e^{-x_i/\alpha} = \alpha^{-2n} \left(\prod_{i = 1}^{n} x_i \right) \exp\left({\frac{-\sum_{i = i}^{n} x_i}{\alpha}}\right) \]

Instead of directly maximizing the likelihood, we instead maximize the **log**-likelihood.

\[ \log L(\alpha) = -2n \log \alpha + \sum_{i = i}^{n} \log x_i - \frac{\sum_{i = i}^{n} x_i}{\alpha} \]

To maximize this function, we take a **derivative** with respect to \(\alpha\).

\[ \frac{d}{d\alpha} \log L(\alpha) = \frac{-2n}{\alpha} + \frac{\sum_{i = i}^{n} x_i}{\alpha^2} \]

We set this derivative equal to **zero**, then **solve** for \(\alpha\).

\[ \frac{-2n}{\alpha} + \frac{\sum_{i = i}^{n} x_i}{\alpha^2} = 0 \]

Solving gives our *estimator*, which we denote with a **hat**.

\[ \hat{\alpha} = \frac{\sum_{i = i}^{n} x_i}{2n} = \frac{\bar{x}}{2} \]

Using the given data, we obtain an *estimate*.

\[ \hat{\alpha} = \frac{0.25 + 0.75 + 1.50 + 2.50 + 2.0}{2 \cdot 5} = \boxed{0.70} \]

**(b)** Obtain the method of moments *estimator* of \(\alpha\), \(\tilde{\alpha}\). Calculate the *estimate* when

\[ x_{1} = 0.25, \ x_{2} = 0.75, \ x_{3} = 1.50, \ x_{4} = 2.5, \ x_{5} = 2.0. \]

**Solution:**

We first obtain the first **population moment**. Notice the integration is done by identifying the form of the integral is that of the second moment of an exponential distribution.

\[ E[X] = \int_{0}^{\infty} x \cdot \alpha^{-2}xe^{-x/\alpha} dx = \frac{1}{\alpha}\int_{0}^{\infty} \frac{x^2}{\alpha} e^{-x/\alpha} dx = \frac{1}{\alpha}(2\alpha^2) = 2\alpha \]

We then set the first population moment, which is a function of \(\alpha\), equal to the first **sample moment**.

\[ 2\alpha = \frac{\sum_{i = i}^{n} x_i}{n} \]

Solving for \(\alpha\), we obtain the method of moments *estimator*.

\[ \tilde{\alpha} = \frac{\sum_{i = i}^{n} x_i}{2n} = \frac{\bar{x}}{2} \]

Using the given data, we obtain an *estimate*.

\[ \tilde{\alpha} = \frac{0.25 + 0.75 + 1.50 + 2.50 + 2.0}{2 \cdot 5} = \boxed{0.70} \]

Note that, in this case, the MLE and MoM estimators are the same.

Let \(X_{1}, X_{2}, \ldots X_{n}\) be a random sample of size \(n\) from a distribution with probability density function

\[ f(x, \beta) = \frac{1}{2 \beta^3} x^2 e^{-x/\beta}, \quad x > 0, \ \beta > 0 \]

**(a)** Obtain the maximum likelihood *estimator* of \(\beta\), \(\hat{\beta}\). Calculate the *estimate* when

\[ x_{1} = 2.00, \ x_{2} = 4.00, \ x_{3} = 7.50, \ x_{4} = 3.00. \]

**Solution:**

We first obtain the likelihood by **multiplying** the probability density function for each \(X_i\). We then **simplify** this expression.

\[ L(\beta) = \prod_{i = 1}^{n} f(x_i; \beta) = \prod_{i = 1}^{n} \frac{1}{2 \beta^3} x^2 e^{-x/\beta} = 2^{-n} \beta^{-3n} \left(\prod_{i = 1}^{n} x_i \right) \exp\left({\frac{-\sum_{i = i}^{n} x_i}{\beta}}\right) \]

Instead of directly maximizing the likelihood, we instead maximize the **log**-likelihood.

\[ \log L(\beta) = -n \log 2 - 3n \log \beta + \sum_{i = i}^{n} \log x_i - \frac{\sum_{i = i}^{n} x_i}{\beta} \]

To maximize this function, we take a **derivative** with respect to \(\beta\).

\[ \frac{d}{d\beta} \log L(\beta) = \frac{-3n}{\beta} + \frac{\sum_{i = i}^{n} x_i}{\beta^2} \]

We set this derivative equal to **zero**, then **solve** for \(\beta\).

\[ \frac{-3n}{\beta} + \frac{\sum_{i = i}^{n} x_i}{\beta^2} = 0 \]

Solving gives our *estimator*, which we denote with a **hat**.

\[ \hat{\beta} = \frac{\sum_{i = i}^{n} x_i}{3n} = \frac{\bar{x}}{3} \]

Using the given data, we obtain an *estimate*.

\[ \hat{\beta} = \frac{2.00 + 4.00 + 7.50 + 3.00}{3 \cdot 4} = \boxed{1.375} \]

**(b)** Obtain the method of moments *estimator* of \(\beta\), \(\tilde{\beta}\). Calculate the *estimate* when

\[ x_{1} = 2.00, \ x_{2} = 4.00, \ x_{3} = 7.50, \ x_{4} = 3.00. \]

**Solution:**

We first obtain the first **population moment**. Notice the integration is done by identifying the form of the integral is that of the third moment of an exponential distribution.

\[ E[X] = \int_{0}^{\infty} x \cdot \frac{1}{2 \beta^3} x^2 e^{-x/\beta} dx = \frac{1}{2\beta^2}\int_{0}^{\infty} \frac{x^3}{\beta} e^{-x/\beta} dx = \frac{1}{2\beta^2}(6\beta^3) = 3\beta \]

We then set the first population moment, which is a function of \(\beta\), equal to the first **sample moment**.

\[ 3\beta = \frac{\sum_{i = i}^{n} x_i}{n} \]

Solving for \(\beta\), we obtain the method of moments *estimator*.

\[ \tilde{\beta} = \frac{\sum_{i = i}^{n} x_i}{3n} = \frac{\bar{x}}{3} \]

Using the given data, we obtain an *estimate*.

\[ \tilde{\beta} = \frac{2.00 + 4.00 + 7.50 + 3.00}{3 \cdot 4} = \boxed{1.375} \]

Note again, the MLE and MoM estimators are the same.

\[ f(x, \lambda) = \lambda x^{\lambda - 1}, \quad 0 < x < 1, \lambda > 0 \]

**(a)** Obtain the maximum likelihood *estimator* of \(\lambda\), \(\hat{\lambda}\). Calculate the *estimate* when

\[ x_{1} = 0.10, \ x_{2} = 0.20, \ x_{3} = 0.30, \ x_{4} = 0.40. \]

**Solution:**

**multiplying** the probability density function for each \(X_i\). We then **simplify** this expression.

\[ L(\lambda) = \prod_{i = 1}^{n} f(x_i; \lambda) = \prod_{i = 1}^{n} \lambda x_i^{\lambda - 1} = \lambda^n \left(\prod_{i = 1}^{n} x_i \right)^{\lambda - 1} \]

Instead of directly maximizing the likelihood, we instead maximize the **log**-likelihood.

\[ \log L(\lambda) = n \log \lambda + (\lambda - 1) \sum_{i = i}^{n} \log x_i \]

To maximize this function, we take a **derivative** with respect to \(\lambda\).

\[ \frac{d}{d\lambda} \log L(\lambda) = \frac{n}{\lambda} + \sum_{i = i}^{n} \log x_i \]

We set this derivative equal to **zero**, then **solve** for \(\beta\).

\[ \frac{n}{\lambda} + \sum_{i = i}^{n} \log x_i = 0 \]

Solving gives our *estimator*, which we denote with a **hat**.

\[ \hat{\lambda} = -\frac{n}{\sum_{i = i}^{n} \log x_i} \]

Using the given data, we obtain an *estimate*.

\[ \hat{\lambda} = -\frac{n}{\sum_{i = i}^{n} \log x_i} = -\frac{4}{\log(0.1 \cdot 0.2 \cdot 0.3 \cdot 0.4)} = \boxed{0.6631} \]

Note that this is actually a reparameterization of an example seen in class where \(\lambda = \frac{1}{\theta}\). Had you realized this, you could have simply found the answer via invariance.

**(b)** Obtain the method of moments *estimator* of \(\lambda\), \(\tilde{\lambda}\). Calculate the *estimate* when

\[ x_{1} = 0.10, \ x_{2} = 0.20, \ x_{3} = 0.30, \ x_{4} = 0.40. \]

**Solution:**

We first obtain the first **population moment**.

\[ E[X] = \int_{0}^{1} x \cdot \lambda x^{\lambda - 1} dx = \frac{\lambda}{\lambda + 1} \]

We then set the first population moment, which is a function of \(\beta\), equal to the first **sample moment**.

\[ \frac{\lambda}{\lambda + 1} = \frac{\sum_{i = i}^{n} x_i}{n} = \bar{x} \]

Solving for \(\lambda\), we obtain the method of moments *estimator*.

\[ \tilde{\lambda} = \frac{\bar{x}}{1 - \bar{x}} \]

Using the given data, we obtain an *estimate*.

\[ \bar{x} = \frac{0.1 + 0.2 + 0.3 + 0.4}{4} = 0.25 \]

\[ \tilde{\lambda} = \frac{0.25}{1 - 0.25} = \boxed{\frac{1}{3}} \]

Note that the MLE and MoM *estimators* are different.