Ordinary Least Square Estimation

1/30

EC114 Introduction to Quantitative Economics 12. Ordinary Least Squares Estimation

Marcus Chambers Department of Economics University of Essex

24/26 January 2012

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

2/30

Outline

1

Ordinary Least Squares (OLS) Estimation

2

Goodness-of-fit

3

Computing OLS Estimates

Reference: R. L. Thomas, Using Statistics in Economics, McGraw-Hill, 2005, sections 9.3 and 9.4.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

3/30

Recall that the population regression line is given by E(Y) = α + βX and the sample regression line is given by Yˆ = a + bX where a and b can be regarded as estimates of α and β. Another way to think of these relationships is in terms of Y itself: Y = α + βX + , Y = a + bX + e, where  is the disturbance and e is the residual.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

4/30

It is clear that, if we vary the sample regression line in some way, we will obtain a different set of residuals. In other words, if we vary the estimation method for the sample regression line, we will obtain a different set of residuals. EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

5/30

The best known method of fitting a straight line to a scatter diagram is Ordinary Least Squares (OLS). The sample regression line is determined by the intercept a and the slope b. A good criterion in the choice of a and b is to make the residuals ‘small’ somehow. Small residuals imply that the differences between the actual Y and the fitted Yˆ are small. The OLS method of estimation chooses a and b in order to minimize the sum of the squares of the residuals: n X

e2i = e21 + e22 + . . . + e2n .

i=1

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

6/30

We know that ei = Yi − a − bXi so that the sum of squared residuals can be written X X S= e2i = (Yi − a − bXi )2 , i

i

which is a function of a and b alone because Yi and Xi are the given data points. The objective is to minimise S with respect to a and b. To do this we need to partially differentiate S with respect to a and b and set these derivatives equal to zero: ∂S ∂a ∂S ∂b

X (Yi − a − bXi ) = 0, X = −2 Xi (Yi − a − bXi ) = 0. = −2

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

7/30

As these derivatives are set equal to zero we can divide both sides by −2 and re-arrange the terms to give: X X Yi = na + b Xi , X X X Xi Yi = a Xi + b Xi2 , noting that

P

a = na.

These are known as the normal equations (but are not related to the normal distribution). Note that, because Yi − a − bXi = ei , we can also write the first-order conditions in the form: ∂S ∂a ∂S ∂b EC114 Introduction to Quantitative Economics

= −2

X

ei = 0,

= −2

X

Xi ei = 0.

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

8/30

We therefore have two equations in two unknowns which we can solve for a and b. The extension question on Problem Set 12 deals with this solution. A compact representation of the solution is: P xi yi ¯ b = P 2 , a = Y¯ − bX, xi ¯ yi = Yi − Y, ¯ and X ¯ and Y¯ are the sample where xi = Xi − X, means of X and Y respectively. The above expressions for a and b are the OLS estimators of α and β.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

9/30

We can compute a and b from various sample sums, making use of the following: P 2 X X X ( Xi ) 2 2 2 ¯ xi = (Xi − X) = Xi − n X 2 2 ¯ = Xi − nX , P P X X X Xi Yi ¯ i − Y) ¯ = xi yi = (Xi − X)(Y Xi Yi − n X ¯ ¯ = Xi Yi − nX Y. In view of this another common expression for b is: P P P Xi Yi Xi Yi − Pn b= . P 2 ( Xi )2 Xi − n EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

10/30

The data on money stock (Y) and GDP (X) in Table 9.1 of Thomas yield: X X X Xi = 132.004, Xi2 = 1247.66, Xi Yi = 220.956, X

Yi = 23.718,

X

Yi2 = 45.154, n = 30.

Based on these quantities we obtain: P X X ( Xi )2 2 2 xi = Xi − n 132.0042 = 1247.66 − = 666.86, 30P P X X Xi Yi xi yi = Xi Yi − n 132.004 × 23.718 = 220.956 − = 116.60. 30 EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

11/30

The slope coefficient is therefore P xi yi 116.60 b= P 2 = = 0.1748. 666.86 xi We also find that ¯ = 132.004 = 4.40, Y¯ = 23.718 = 0.7906 X 30 30 and so the intercept is ¯ = 0.7906 − (0.1748 × 4.40) = 0.0212. a = Y¯ − bX The sample regression line is therefore Yˆ = 0.021 + 0.175X.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

12/30

Note that the residuals (the vertical distance between the data point and the line) are larger for the countries with larger GDP (X).

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

13/30

Note, too, that the sample regression line passes through ¯ Y. ¯ the point X, The reason can be seen by re-arranging the equation for a, which gives ¯ Y¯ = a + bX. This is known as the point of sample means. In our example this point is (4.40,0.791). Note that the intercept a > 0, although its value is small. We had expected a relationship of the form Y = bX, suggesting a = 0. Although a is small we will want to know whether it is significantly different from zero – we shall consider testing the hypothesis that a = 0 at a later point.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

14/30

The value b = 0.175 means that the demand for money per head will increase by $175 whenever GDP per head increases by $1000. But a more interesting quantity is the income (GDP) elasticity of the demand for money. We can use the previous results to compute an estimate of it – the required elasticity is given by the formula η=

dY X . dX Y

However the elasticity varies along our sample regression line because the values of X and Y vary along the line. It is, however, common to evaluate η at the sample means of X and Y, while dY/dX can be estimated by b.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Ordinary Least Squares (OLS) Estimation

15/30

In our case the elasticity evaluated at the sample means is η = 0.175

4.40 = 0.973. 0.791

Thus we obtain a GDP elasticity close to unity. A 1% rise in GDP per head leads to a 0.97% rise in demand for money per head. It would be of interest to test the hypothesis that η = 1 and we will examine how to do this later on in the term.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

16/30

So far we have fitted the sample regression line Yˆ = 0.021 + 0.175X to our scatter of points in the money-income example. The values of the intercept (0.021) and slope (0.175) were obtained by the method of ordinary least squares (OLS) which chooses these values so as to minimise the sum of squared residuals: X X e2i = (Yi − a − bXi )2 . i

i

But we might want to ask the question: how well does our sample regression line fit the data? Let’s begin by taking a look at the graph: EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

17/30

We can observe that the sample regression line passes ‘fairly close’ to each point in the scatter, although with greater dispersion for larger values of X (GDP per head). We need, however, to be more precise about this; in other words we need some sort of numerical measure of goodness of fit. EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

18/30

We will use the coefficient of determination, R2 , which is equal to the square of the correlation coefficient R, where P ¯ ¯ (X − X)(Y − Y) pP R = pP ¯ 2 ¯ 2 (X − X) (Y − Y) We know that −1 ≤ R ≤ 1, and so it follows that 0 ≤ R2 ≤ 1. In our example of the demand for money we found that R = 0.8787. Hence the coefficient of determination must therefore be R2 = (0.8787)2 = 0.772. In regression analysis it is possible to give a precise interpretation to the value 0.772 obtained for R2 .

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

19/30

Suppose we ask the question: What proportion of the variation in the demand for money in our 30 countries can be attributed to the variation in GDP? If our sample regression line is able to explain a high proportion of the variation in the demand for money then it must provide a good fit to the data. Consider the next Figure, which refers to a single sample point, namely France, which is observation i = 8. We have Y8 = 2.3912; Yˆ8 = 1.6776; and the overall sample mean is Y¯ = 0.7906.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

20/30

The diagram shows the fitted line, the sample mean line, the residual e8 = Y8 − Yˆ8 , as well as the deviations Y8 − Y¯ ¯ and Yˆ8 − Y.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

21/30

The variations in demand for money are measured relative to the mean. The following relationship holds: total = variation due variation to X Y8 − Y¯

=

Yˆ8 − Y¯

+

residual variation

+

e8

1.6006 = 0.8870 + 0.7136 Such a relationship holds for all points in the sample, so that we can write ¯ = (Yˆi − Y) ¯ + ei , i = 1, . . . , n. (Yi − Y)

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

22/30

Note that these variations can be positive or negative, and that they only apply to a single point in the sample. However, we require an overall measure for the entire sample, and when we talk about variation we usually have in mind a positive measure. A measure of variation of Y taken over the entire sample is the total sum of squares (SST): n X ¯ 2. (Yi − Y) i=1

This is the total variation in Y that we attempt to explain by our regression line, and is always non-negative. We have seen this sort of quantity before – dividing by n − 1 gives the sample variance. EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

23/30

A sample-wide measure of the variation in Y due to X is given by the explained sum of squares (SSE): n X ¯ 2. (Yˆi − Y) i=1

This quantity is also non-negative. Finally, a measure of the total residual variation is the residual sum of squares (SSR): n X

e2i ,

i=1

which is also non-negative.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

24/30

The following relationship holds: total sum of = explained sum squares of squares Pn

i=1 (Yi

¯ 2 − Y)

=

Pn

ˆ − Y) ¯ 2

i=1 (Yi

+

+

residual sum of squares Pn

2 i=1 ei

SST = SSE + SSR The extension question on Problem Set 12 deals with this identity. These quantities are used to define the coefficient of determination, R2 , as follows: R2 =

variation in Y due to X SSE = . total variation in Y SST

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

25/30

Alternative (but equivalent) expressions for R2 include R2 = 1 −

SSR SST

obtained by making the substitution SSE=SST−SSR. Another expression is P b2 xi2 R = P 2 yi 2

¯ and yi = Yi − Y. ¯ where xi = Xi − X The derivation requires showing that P 2 of this last expression P 2 SSE= b xi and noting that SST= y2i (see the extension question on Problem Set 12).

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

26/30

In our demand for money example we have already shown that R2 = 0.772 by squaring the correlation coefficient R = 0.8787. However, we know that X X b = 0.17485, xi2 = 666.86, y2i = 26.403, and hence an alternative derivation is P b2 xi2 0.174852 × 666.86 2 = 0.772. R = P 2 = 26.403 yi This implies that just over 77% of the variation in the demand for money can be attributed to the variation in GDP. Remember: 0 ≤ R2 ≤ 1. EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

27/30

In (a) and (c) R2 = 1 because all points lie on a single sample line – in (a) R = +1 and in (c) R = −1. In (b) R = 0 due to the lack of association between the two variables and hence R2 = 0.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Goodness-of-fit

28/30

The correlation coefficient, R, is a measure of the strength of association between two variables, and says nothing about the direction of causation (if any exists). The coefficient of determination, R2 , however, is based on the regression model Y = α + βX +  in which the causation is assumed to go from X to Y. However, we should be careful to refer to R2 as the percentage of the variation in Y attributed to X rather than explained by X, because any such relationship could be spurious.

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Computing OLS Estimates

29/30

In practice we use computer software for OLS calculations. As an example, the Stata output for the money demand example is of the form: . regress m g Source | SS df MS -------------+-----------------------------Model | 20.3862321 1 20.3862321 Residual | 6.01600434 28 .214857298 -------------+-----------------------------Total | 26.4022364 29 .910421946

Number of obs F( 1, 28) Prob > F R-squared Adj R-squared Root MSE

= = = = = =

30 94.88 0.0000 0.7721 0.7640 .46353

-----------------------------------------------------------------------------m | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------g | .1748489 .0179502 9.74 0.000 .1380795 .2116182 _cons | .0212579 .1157594 0.18 0.856 -.2158645 .2583803 ------------------------------------------------------------------------------

Quite a lot of information is provided by default, but note that the estimates a and b are given at the start of the final two rows (under the heading ‘Coef.’). EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation

Summary

30/30

Summary

Ordinary Least Squares (OLS) Estimation Goodness-of-fit

Next week: Non-Linear Models

EC114 Introduction to Quantitative Economics

12. Ordinary Least Squares Estimation