Postscript version of this file

STAT 801 Lecture 13

Goals of Today's Lecture:

• Develop large sample theory for the mle.

• Define and interpret Fisher information.

• Extend the ideas to estimating equations.

Today's notes

Large Sample Theory of the MLE

Application of the law of large numbers to the likelihood function:

The log likelihood ratio for $\theta$ to $\theta_0$ is

$$\ell(\theta) -\ell(\theta_0) = \sum L_i$$

where $L_i = \log[f(X_i,\theta)/f(X_i,\theta_0)]$. We proved that $\mu(\theta) \equiv E_{\theta_0}(L_i) < 0$. Then

$$\frac{\ell(\theta) -\ell(\theta_0)}{n} \to \mu(\theta)$$

so

$$P_{\theta_0}(\ell(\theta)< \ell(\theta_0)) \to 1 \, .$$

Definition A sequence $\hat\theta_n$ of estimators of $\theta$ is consistent if $\hat\theta_n$ converges weakly (or strongly) to $\theta$.

Proto theorem: In regular problems the mle $\hat\theta$ is consistent.

Now let us study the shape of the log likelihood near the true value of $\hat\theta$ under the assumption that $\hat\theta$ is a root of the likelihood equations close to $\theta_0$. We use Taylor expansion to write, for a 1 dimensional parameter $\theta$,
$$\begin{align*}U(\hat\theta) = & 0 \\ = & U(\theta_0) + U^\prime(\theta_0)(\h... ... \\ & + U^{\prime\prime}(\tilde\theta) (\hat\theta-\theta_0)^2/2 \end{align*}$$
for some $\tilde\theta$ between $\theta_0$ and $\hat\theta$. (This form of the remainder in Taylor's theorem is not valid for multivariate $\theta$.) The derivatives of U are each sums of n terms and so should be both proportional to n in size. The second derivative is multiplied by the square of the small number $\hat\theta-\theta_0$ so should be negligible compared to the first derivative term. If we ignore the second derivative term we get

$$-U^\prime(\theta_0)(\hat\theta-\theta_0) \approx U(\theta_0)$$

Now let's look at the terms U and $U^\prime$.

In the normal case

$$U(\theta_0) = \sum (X_i-\mu_0)$$

has a normal distribution with mean 0 and variance n (SD $\sqrt{n}$). The derivative is simply

$$U^\prime(\mu) = -n$$

and the next derivative $U^{\prime\prime}$ is 0. We will analyze the general case by noticing that both U and $U^\prime$ are sums of iid random variables. Let

$$U_i = \frac{\partial \log f}{\partial\theta} (X_i,\theta_0)$$

and

$$V_i = -\frac{\partial^2 \log f}{\partial\theta^2} (X_i,\theta)$$

In general, $U(\theta_0)=\sum U_i$ has mean 0 and approximately a normal distribution. Here is how we check that:
$$\begin{align*}E_{\theta_0}(U(\theta_0)) & = n E_{\theta_0} (U_1) \\ & = n \int ... ...heta=\theta_0} \\ &= n\frac{\partial}{\partial\theta} 1 \\ & = 0\ \end{align*}$$

Notice that I have interchanged the order of differentiation and integration at one point. This step is usually justified by applying the dominated convergence theorem to the definition of the derivative. The same tactic can be applied by differentiating the identity which we just proved

$$\int\frac{\partial\log f}{\partial\theta}(x,\theta) f(x,\theta) dx =0$$

Taking the derivative of both sides with respect to $\theta$ and pulling the derivative under the integral sign again gives

$$\int \frac{\partial}{\partial\theta} \left[ \frac{\partial\log f}{\partial\theta}(x,\theta) f(x,\theta) \right] dx =0$$

Do the derivative and get
$$\begin{multline*}-\int\frac{\partial^2\log(f)}{\partial\theta^2} f(x,\theta) dx ... ...artial\log f}{\partial\theta}(x,\theta) \right]^2 f(x,\theta) dx \end{multline*}$$

Definition: The Fisher Information is

$$I(\theta)=-E_{\theta}(U^\prime(\theta))=nE_{\theta_0}(V_1)$$

We refer to ${\cal I}(\theta_0) = E_{\theta_0}(V_1)$ as the information in 1 observation.

The idea is that I is a measure of how curved the log likelihood tends to be at the true value of $\theta$. Big curvature means precise estimates. Our identity above is

$$I(\theta) = {\rm Var}_\theta(U(\theta))=n{\cal I}(\theta)$$

Now we return to our Taylor expansion approximation

$$-U^\prime(\theta_0)(\hat\theta-\theta_0) \approx U(\theta_0)$$

and study the two appearances of U.

We have shown that $U=\sum U_i$ is a sum of iid mean 0 random variables. The central limit theorem thus proves that

$$n^{-1/2} U(\theta) \Rightarrow N(0,\sigma^2)$$

where $\sigma^2 = {\rm Var}(U_i)=E(V_i)={\cal I}(\theta)$.

Next observe that

$$-U^\prime(\theta) = \sum V_i$$

where again

$$V_i = -\frac{\partial U_i}{\partial\theta}$$

The law of large numbers can be applied to show

$$-U^\prime(\theta_0)/n \to E_{\theta_0}[ V_1] = {\cal I}(\theta_0)$$

Now manipulate our Taylor expansion as follows

$$n^{1/2} (\hat\theta - \theta_0) \approx \left[\frac{\sum V_i}{n}\right]^{-1} \frac{\sum U_i}{\sqrt{n}}$$

Apply Slutsky's Theorem to conclude that the right hand side of this converges in distribution to $N(0,\sigma^2/{\cal I}(\theta)^2)$ which simplifies, because of the identities, to $N(0,1/{\cal I}(\theta))$.

Summary

In regular families:

• Under strong regularity conditions Jensen's inequality can be used to demonstrate that $\hat\theta$ which maximizes $\ell$ globally is consistent and that this $\hat\theta$ is a root of the likelihood equations.

• It is generally easier to study $\ell$ only close to $\theta_0$. For instance define A to be the event that $\ell$ is concave on the set of $\theta$ such that $\vert\theta-\theta_0\vert <\delta$ and the likelihood equations have a unique root in that set. Under weaker conditions than the previous case we can prove that there is a $\delta > 0$ such that
  
  $$P(A) \to 1$$
  
  
  In that case we can prove that the root $\hat\theta$ of the likelihood equations mentioned in the definition of A is consistent.

• Sometimes get still weaker conclusion. Let B be the event that $\ell(\theta)$ is concave for $n^{1/2}\vert\theta-\theta_0\vert < L$ and there is a unique root of $\ell$ over this range. Again this root is consistent but there might be other consistent roots of the likelihood equations.

• Under any of these scenarios there is a consistent root of the likelihood equations which is definitely the closest to the true value $\theta_0$. This root $\hat\theta$ has the property
  
  $$\sqrt{n}(\hat\theta-\theta_0) \Rightarrow N(0,1/{\cal I}(\theta)) \, .$$
  
  

We usually simply say that the mle is consistent and asymptotically normal with an asymptotic variance which is the inverse of the Fisher information. This assertion is actually valid for vector valued $\theta$ where now I is a matrix with ijth entry

$$I_{ij} = - E\left(\frac{\partial^2\ell}{\partial\theta_i\partial\theta_j}\right)$$

Estimating Equations

Same ideas arise whenever estimates derived by solving some equation. Example: large sample theory for Generalized Linear Models.

Suppose that for $i=1,\ldots, n$ we have observations of the numbers of cancer cases Y_i in some group of people characterized by values x_i of some covariates. You are supposed to think of x_i as containing variables like age, or a dummy for sex or average income or $\ldots$A parametric regression model for the Y_i might postulate that Y_i has a Poisson distribution with mean $\mu_i$ where the mean $\mu_i$ depends somehow on the covariate values. Typically we might assume that $g(\mu_i) = \beta_0+ x_i \beta_1$ where g is a so-called link function, often for this case $g(\mu) = \log(\mu)$ and $x_i\beta$ is a matrix product with x_i written as a row vector and $\mu$ a column vector. This is supposed to function as a ``linear regression model with Poisson errors''. I will do as a special case $\log(\mu_i) = \beta x_i$ where x_i is a scalar.

The log likelihood is simply

$$\ell(\beta) = \sum(Y_i \log(\mu_i) - \mu_i)$$

ignoring irrelevant factorials. The score function is, since $\log(\mu_i) = \beta x_i$,

$$U(\beta) = \sum (Y_i x_i - x_i \mu_i) = \sum x_i(Y_i-\mu_i)$$

(Notice again that the score has mean 0 when you plug in the true parameter value.) The key observation, however, is that it is not necessary to believe that Y_i has a Poisson distribution to make solving the equation U=0 sensible. Suppose only that $\log(E(Y_i)) = x_i\beta$. Then we have assumed that

$$E_\beta(U(\beta)) = 0$$

This was the key condition in proving that there was a root of the likelihood equations which was consistent and here it is what is needed, roughly, to prove that the equation $U(\beta)=0$ has a consistent root $\hat\beta$. Ignoring higher order terms in a Taylor expansion will give

$$V(\beta)(\hat\beta-\beta) \approx U(\beta)$$

where $V=-U^\prime$. In the mle case we had identities relating the expectation of V to the variance of U. In general here we have

$${\rm Var}(U) = \sum x_i^2 {\rm Var}(Y_i)$$

If Y_i is Poisson with mean $\mu_i$ (and so ${\rm Var}(Y_i)=\mu_i$) this is

$${\rm Var}(U) = \sum x_i^2\mu_i$$

Moreover we have

$$V_i = x_i^2 \mu_i$$

and so

$$V(\beta) = \sum x_i^2 \mu_i$$

The central limit theorem (the Lyapunov kind) will show that $U(\beta)$ has an approximate normal distribution with variance $\sigma_U^2 = \sum x_i^2 {\rm Var}(Y_i)$ and so

$$\hat\beta-\beta \approx N(0,\sigma_U^2/(\sum x_i^2\mu_i)^2)$$

If ${\rm Var}(Y_i)=\mu_i$, as it is for the Poisson case, the asymptotic variance simplifies to $1/\sum x_i^2\mu_i$.

Other estimating equations are possible. People suggest alternatives very often. If w_i is any set of deterministic weights (even possibly depending on $\mu_i$then we could define

$$U(\beta) = \sum w_i (Y_i-\mu_i)$$

and still conclude that U=0 probably has a consistent root which has an asymptotic normal distribution. Idea widely used: see, e.g., Zeger and Liang's idea of Generalized Estimating Equations (GEE) which the econometricians call Generalized Method of Moments.

Problems with maximum likelihood

1.

In problems with many parameters the approximations don't work very well and maximum likelihood estimators can be far from the right answer. See your homework for the Neyman Scott example where the mle is not consistent.

2.

When there are multiple roots of the likelihood equation you must choose the right root. To do so you might start with a different consistent estimator and then apply some iterative scheme like Newton Raphson to the likelihood equations to find the mle. It turns out not many steps of NR are generally required if the starting point is a reasonable estimate.

Finding (good) preliminary Point Estimates

Method of Moments

Basic strategy: set sample moments equal to population moments and solve for the parameters.

Definition: The $r^{\mbox{th}}$ sample moment (about the origin) is

$$\frac{1}{n}\sum_{i=1}^n X_i^r$$

The $r^{\mbox{th}}$ population moment is

$${\rm E}(X^r)$$

(Central moments are

$$\frac{1}{n}\sum_{i=1}^n (X_i-\bar X)^r$$

and

$${\rm E}\left[(X-\mu)^r\right] \, .$$

If we have p parameters we can estimate the parameters $\theta_1,\ldots,\theta_p$ by solving the system of pequations:

$$\mu_1 = \bar{X}$$

$$\mu_2^\prime = \overline{X^2}$$

and so on to

$$\mu_p^\prime = \overline{X^p}$$

You need to remember that the population moments $\mu_k^\prime$ will be formulas involving the parameters.

Gamma Example

The Gamma( $\alpha,\beta$) density is

$$f(x;\alpha,\beta) = \frac{1}{\beta\Gamma(\alpha)}\left(\frac{... ...ta}\right)^{\alpha-1} \exp\left[-\frac{x}{\beta}\right] 1(x>0)$$

and has

$$\mu_1 = \alpha\beta$$

and

$$\mu_2^\prime = \alpha\beta^2$$

This gives the equations
$$\begin{align*}\alpha\beta & = \overline{X} \\ \alpha\beta^2 & = \overline{X^2} \end{align*}$$
Divide the second equation by the first to find the method of moments estimate of $\beta$ is

$$\tilde\beta = \overline{X^2}/\overline{X}$$

Then from the first equation get

$$\tilde\alpha = \overline{X}/\tilde\beta= (\overline{X})^2/\overline{X^2}$$

The method of moments equations are much easier to solve than the likelihood equations which involve the function

$$\psi(\alpha) = \frac{d}{d\alpha} \log(\Gamma(\alpha))$$

called the digamma function.