Postscript version of these notes

STAT 804

Lecture 11

Likelihood Theory

First we review likelihood theory for conditional and full maiximum likelihood estimation.

Suppose the data is $X=(Y,Z)$ and write the density of $X$ as

$$f(x\vert\theta) = f(y\vert z,\theta)f(z\vert\theta)$$

Differentiate the identity

$$1 = \int f(y\vert z,\theta) dy$$

with respect to $\theta_j$ (the $j$th component of $\theta$) and pull the derivative under the integral sign to get

$$= \int \frac{\partial f(y\vert z,\theta_j)}{\partial\theta} dy$$ 0

$$= \int \frac{\partial \log f(y\vert z,\theta_j)}{\partial\theta} f(y\vert z,\theta) dy$$

$$= _\theta(U_{Y\vert Z;j}(\theta)\vert Z)$$

where $U_{Y\vert Z;j}(\theta)$ is the $j$th component of $U_{Y\vert Z}(\theta)$, the derivative of the log conditional likelihood; $U_{Y\vert Z}$ is called a conditional score. Since

   E$$_\theta(U_{Y\vert Z;j}(\theta)\vert Z) = 0$$

we may take expected values to see that

   E$$_\theta(U_{Y\vert Z;j}(\theta)) = 0$$

It is also true that the other two scores $U_X(\theta)$ and $U_Z(\theta)$ have mean 0 (when $\theta$ is the true value of $\theta$). Differentiate the identity a further time with respect to $\theta_k$ to get

$$0 = \int \frac{\partial^2 \log f(y\vert z,\theta)}{\partial\theta... ...rac{\partial \log f(y\vert z,\theta_k)}{\partial\theta} f(y\vert z,\theta) dy$$

We define the conditional Fisher information matrix $I_{Y\vert Z}(\theta)$ to have $jk$th entry

   E$$\left[- \frac{\partial^2 \ell}{\partial\theta_j\partial\theta_k}\vert Z\right]$$

and get

$$I_{Y\vert Z}(\theta\vert Z) =$$   Var$$_\theta(U_{Y\vert Z}(\theta)\vert Z)$$

The corresponding identities based on $f_X$ and $f_Z$ are

$$I_X(\theta) =$$   Var$$_\theta(U_X(\theta))$$

and

$$I_Z(\theta) =$$   Var$$_\theta(U_Z(\theta))$$

Now let's look at the model $X_t = \rho X_{t-1}+\epsilon_t$. Putting $Y=(X_1,\ldots,X_{T-1})$ and $Z=X_0$ we find

$$U_{Y\vert Z}(\rho,\sigma) = \begin{array}{l} \frac{\sum_1^{T... ..._t-\rho X_{t-1})^2}{\sigma^3} -\frac{T-1}{\sigma} \end{array}$$

Differentiating again gives the matrix of second derivatives

$$\left[ \begin{array}{cc} -\frac{\sum_1^{T-1}X_{t-1}^2}{\sigm... ...X_{t-1})^2}{\sigma^4} +\frac{T-1}{\sigma^2} \end{array}\right]$$

Taking conditional expectations given $X_0$ gives

$$I_{Y\vert Z}(\rho,\sigma) = \left[ \begin{array}{cc} \frac{\... ...gma^2} & 0 \\ 0 & \frac{2(T-1)}{\sigma^2} \end{array}\right]$$

To compute $W_k \equiv$   E$[X_k^2\vert X_0]$ write $X_k = \rho X_{k-1}+\epsilon_k$ and get

$$W_k =\rho^2 W_{k-1} + \sigma^2$$

with $W_0=X_0^2$. You can check carefully that in fact $W_k$ converges to some $W_\infty$ as $k\to\infty$. This $W_\infty$ satisfies $W_\infty = \rho^2W_\infty + \sigma^2$ which gives

$$W_\infty = \frac{\sigma^2}{1-\rho^2}$$

It follows that

$$\frac{1}{T} I_{Y\vert Z}(\rho,\sigma) \to \left[ \begin{ar... ...{1}{1-\rho^2} & 0 \\ 0 & \frac{2}{\sigma^2}\end{array}\right]$$

Notice that although the conditional Fisher information might have been expected to depend on $X_0$ it does not, at least for long series.