Spring 2001 John Rust
Economics 551b 37 Hillhouse, Rm. 27

PROBLEM SET 1

Regression Basics

QUESTION 1 This question is designed to help you get a better appreciation for what the ``Projection Theorem'' is and how it relates to doing regression in practical situations.

1.

Let P(y|X) denote the projection of the vector y onto the subspace X. Define formally what P(y|X) is.

2.

State the Projection Theorem and give a geometrical interpretation of the ``orthogonality'' between the projection residual $\epsilon=y-P(y\vert X)$ and the subspace X onto which y is being projected.

3.

If $y \in R^N$ and X is the subspace spanned by the $N \times K$ matrix of regressors (which I also denote by X) in a regression, show that

$$P(y\vert X) = X \hat\beta = X (X'X)^{-1} X'y.$$ (1)

4.

Verify by a direct calculation that $\epsilon=y-P(y\vert X)$ is orthogonal to X for the closed form expression for P(y|X) given above.

5.

Prove that the projection operator satisfies:

P(P(y|X)|X) = P(y|X). (2)

Interpret this condition geometrically, and relate it to the property of idempotence of the projection matrix X(X'X)^-1 X'.

6.

Prove the Law of Iterated Projections i.e. if X and Z are subspaces of a Hilbert space H and X is a subspace of Z, then for any $y \in H$ we have:

P(y|X) = P(P(y|Z)|X). (3)

7.

Use the Law of Iterated Projections to show that if you first regress y on the variables in the matrices X and Z:

$$y = X\beta_1 + Z\beta_2 + \epsilon$$ (4)

and then you use the fitted $\hat y=X\hat\beta_1+Z\hat\beta_2$ as the dependent variable in the regression

$$\hat y = X\gamma + u$$ (5)

you will get the same numerical estimate of the estimated regression coefficient $\hat\gamma$ as if you regressed y on X

$$y = X \gamma + u$$ (6)

However is it generally the case that $\hat\gamma = \hat\beta_1$? If so, provide a proof, if not, provide a counterexample where $\hat\gamma \ne \hat\beta_1$?

8.

Can you state conditions under which you can guarantee that $\hat\beta_1 = \hat\gamma$ in the two regression in part 4 above?

9.

Let H be $L_2(\Omega,{\cal F},\mu)$ be the classical L₂ space, i.e. the space of all random variables defined on the probability space $(\Omega,{\cal F},\mu)$ that have finite variance, with inner product given by

$$\langle \tilde X,\tilde Y\rangle = \int X(\omega)Y(\omega)\mu(d\omega).$$ (7)

Let X denote the subspace spanned by the K random variables $(\tilde X_1,\ldots,\tilde X_K)$. Find a formula for the projection of the random variable $\tilde y$ on X, $P(\tilde y\vert X)$, and provide an interpretation of what it means.

10.

Let X be the space of all measurable functions of the random variables $(\tilde X_1,\ldots,\tilde X_K)$ that have finite variance. Show that this is a subspace of $L_2(\Omega,{\cal F},\mu)$. Given any $\tilde y \in L_2(\Omega,{\cal F},\mu)$, what is $P(\tilde y\vert X)$?

11.

If instead of $L_2(\Omega,{\cal F},\mu)$ we consider the space H=R^N, and if X is the space of all measurable functions of K vectors $(X_1,\ldots,X_K)$ in R^N, for any $y \in R^N$ what is P(y|X)?

QUESTION 2 Consider the ``textbook'' regression model:

$$y = X\beta^* + \epsilon$$

where X is regarded as a fixed (non-random) $N \times K$ matrix and the error vector $\epsilon$ is a random vector with a $N(\boldmath {0},\sigma^2 \boldmath {I})$ distribution, where $\boldmath {0}$ is an $N \times 1$ vector of 0's and $\boldmath {I}$ is an $N \times N$identity matrix, and $\sigma^2 > 0$ is a constant.

1.

Show that OLS is a linear estimator of $\beta^*$.

2.

Show that an arbitrary linear estimator of $\beta^*$ must have the form M y for some matrix M. What are the dimensions of M?

3.

What constraints must be placed on M to result in an unbiased estimator of $\beta^*$?

4.

What is the matrix M for the OLS estimator? Show that for this choice of M the unbiasedness constraint that you derived above is satisfied.

5.

Show that the variance-covariance matrix for a linear estimator of $\beta^*$ is given by $\sigma^2 M'M$. Does this formula depend on M satisfying the restriction for unbiasedness, or will it hold even for unbiased estimators of $\beta^*$?

6.

Derive the covariance matrix for $\hat\beta$, the OLS estimator.

7.

Prove the Gauss Markov Theorem, i.e. show that the OLS estimator is the best, linear, unbiased estimator of $\beta^*$. Hint: for an alternative estimator of the form $\tilde \beta=My$ for some matrix M, write M as

M= X(X'X)^-1 X' + C (8)

for some matrix C. Figure out what restrictions C needs to satisfy so that $\tilde \beta$ is an unbiased estimator, and then use this to compute the covariance matrix for $\tilde \beta$ and show that this exceeds the covariance matrix for $\hat\beta$ by a positive semi-definite matrix.