Simple Linear Regression Regression Assumptions

A very basic review of the assumptions of simple linear regression. Future posts will expand on the effects we get when we relax these assumptions, as well as assumptions we use in other econometric models.

Simple Linear Regression Setup

• Population Regression Line.

$E(y\vert x) = \mu_{y \vert x} = \beta_0 + \beta_1 x$

• Individual response.

$y_i = \beta_0 + \beta_1 x_i + e_i $

where parameters have the same interpretation and the error $e_i$ represents the differences between the true $y_i$ and the conditional mean $\mu_{y \vert x}$.

• $e_i \sim \mathcal{N}(0,\sigma_e^2)$, $\rightarrow$: Homoskedasticty and normality.

• $e_i \sim iid $ (Of concern when working with time series): Nonautocorrelation.

Main assumptions on variables

• $x_i$ and $y_i$ are observed and nonrandom after observation.

• $K$ regressors $(x_i)$ are independet (Absence of perfect multicollinearity).

•  Errors and indep variables are independent: $E(x' \epsilon=0) $.

Consequences of the assumption on parameter estimates

• Sampling distribution of $\beta_0$ 

1.  $E(\hat{\beta}_0)=\beta_0$
2. $var(\hat{\beta_0}) = \sigma_e^2 \left( \frac{1}{N} + \frac{\bar{x}^2}{\sum_{i=1}^N ( x_i-\bar{x} )^2} \right )  = \sigma_e^2 \left( \frac{1}{N} + \frac{\bar{x}^2}{(N-1) s_x^2 } \right )$

• Sampling distribution of $\beta_1$ 

1. $E(\hat{\beta}_1)=\beta_1$
2. $var(\hat{\beta_1}) = \frac{\sigma_e^2}{\sum_{i=1}^N ( x_i-\bar{x} )^2} =  \frac{\sigma_e^2}{(N-1) s_x^2 }$ where $s_x^2 = \frac{ \sum_{i=1}^N (x_i-\bar{x})^2}{(N-1)} $
3. The sampling distribution of $\beta_0$ is normally distributed.

• Estimate of $\sigma_e^2$: 

          $s_e^2=\frac{\sum_i (y_i-\hat{y_i})^2}{N-2}=\frac{SSE}{N-2}=MSE$

Characteristics of the parameter estimates

• Unbiased Estimators: The mean of the sampling distribution is equal to the population parameter being estimated. 
•  Consistent Estimators: As $N$ increases, the probability that the estimator will be close to the true parameter increases: As $N\rightarrow \infty$ then $\hat{\beta}=\beta$.
• Minimum variance estimators: The variance of $\hat{\beta}$ is smaller than the variance of any other linear unbiased estimator of $\beta_1$.

These are the most basic assumptions of the simple linear regression. The consequences require these assumptions to be satisfied. Empirical analyses that use simple linear regression are making these assumptions implicitly. There are many cases in which it should be obvious that they will not be satisfied. These posts analyze these cases and their consequences.