Predictive Analytics Using Regression

PREDICTIVE ANALYTICS USING REGRESSION Sumeet Gupta Associate Professor Indian Institute of Management Raipur

Outline •  Basic Concepts •  Applications of Predictive Modeling •  Linear Regression in One Variable using OLS •  Multiple Linear Regression •  Assumptions in Regression •  Explanatory Vs Predictive Modeling •  Performance Evaluation of Predictive Models •  Practical Exercises •  Case: Nils Baker •  Case: Pedigree Vs Grit

BASIC CONCEPTS

4

Predictive Modeling: Applications •  Predictive customer activity on credit cards from their

demographic and historical activity patterns •  Predicting the time to failure or equipment based on utilization and environment conditions •  Predicting expenditures on vacation travel based on historical frequent flyer data •  Predicting staffing requirements at help desks based on historical data and product and sales information •  Predicting sales from cross selling of products from historical information •  Predicting the impact of discounts on sales in retail outlets

5

Basic Concept: Relationships Examples of relationships: •  Sales and earnings •  Cost and number produced •  Microsoft and the stock market •  Effort and results

•  Scatterplot •  A picture to explore the relationship in bivariate data •  Correlation r •  Measures strength of the relationship (from –1 to 1) •  Regression •  Predicting one variable from the other

6

Basic Concept: Correlation •  r = 1 •  A perfect straight line tilting up to the right •  r = 0 •  No overall tilt •  No relationship? •  r = – 1 •  A perfect straight line tilting down to the right

Y

Y

X Y

X Y

X Y

X Y

X

X

7

Basic Concepts: Simple Linear Model •  Linear Model for the Population •  The foundation for statistical inference in regression •  Observed Y is a straight line, plus randomness

{

Y = α + βX + ε Randomness of individuals Population relationship, on average Y ε

X

8

Basic Concepts: Simple Linear Model •  Time Spent vs. Internet Pages Viewed •  Two measures of the abilities of 25 Internet sites •  At the top right are eBay, Yahoo!, and MSN

•  Correlation is r = 0.964 •  Linear relationship •  Straight line

with scatter •  Increasing relationship •  Tilts up and to the right

Minutes per person

•  Very strong positive association (since r is close to 1)

90 eBay Yahoo!

60 MSN

30 0

0

100 200 Pages per person

9

Basic Concepts: Simple Linear Model •  Dollars vs. Deals •  For mergers and acquisitions by investment bankers •  244 deals worth $756 billion by Goldman Sachs

•  Correlation is r = 0.419 •  Positive association •  Straight line

with scatter •  Increasing relationship •  Tilts up and to the right

$1,000 Dollars (billions)

•  Linear relationship

$500

$0

0

100

200 300 Deals

400

10

Basic Concepts: Simple Linear Model •  Interest Rate vs. Loan Fee •  For mortgages •  If the interest rate is lower, does the bank make it up with a higher loan

fee? •  Correlation is r = – 0.890 •  Linear relationship •  Straight line

with scatter •  Decreasing relationship •  Tilts down and to the right

Interest rate

•  Strong negative association

6.0% 5.5% 5.0% 0%

1%

2% 3% Loan fee

4%

11

Basic Concepts: Simple Linear Model •  Today’s vs. Yesterday’s Percent Change •  Is there momentum? •  If the market was up yesterday, is it more likely to be up today? Or is

each day’s performance independent? 3%

•  Correlation is r = 0.11 •  No relationship? •  Tilt is neither

up nor down

2%

Today's change

•  A weak relationship?

1% 0% -1% -2% -3% -3% -2% -1% 0% 1% 2% 3% Yesterday's change

12

Basic Concepts: Simple Linear Model •  Call Price vs. Strike Price •  For stock options •  “Call Price” is the price of the option contract to buy stock at the

“Strike Price” •  The right to buy at a lower strike price has more value •  A nonlinear relationship

A curved relationship •  Correlation r = – 0.895 •  A negative relationship:

Higher strike price goes with lower call price

$100

Call Price

•  Not a straight line:

$75 $50 $25 $0 $450

$500

$550 Strike Price

$600

$650

13

Basic Concepts: Simple Linear Model •  Output Yield vs. Temperature •  For an industrial process •  With a “best” optimal temperature setting

•  A nonlinear relationship •  Not a straight line:

•  Correlation r = – 0.0155 •  r suggests no relationship

•  But relationship is strong •  It tilts neither

up nor down

Yield of process

A curved relationship 160 150 140 130 120 500

600 700 800 Temperature

900

14

Basic Concepts: Simple Linear Model •  Circuit Miles vs. Investment (lower left) •  For telecommunications firms •  A relationship with unequal variability •  More vertical variation at the right than at the left •  Variability is stabilized by taking logarithms (lower right)

r = 0.957 Log of miles

Circuit miles (millions)

•  Correlation r = 0.820

2,000 1,000 0 0

1,000 2,000 Investment ($millions)

20

15

15 20 Log of investment

15

Basic Concepts: Simple Linear Model •  Price vs. Coupon Payment •  For trading in the bond market •  Bonds paying a higher coupon generally cost more

•  Two clusters are visible •  Ordinary bonds (value is from coupon) •  Inflation-indexed bonds (payout rises with inflation) •  for all bonds

•  Correlation r = 0.994 •  Ordinary bonds only

Bid price

•  Correlation r = 0.950

$150 $100 0%

5% 10% Coupon rate

16

Basic Concepts: Simple Linear Model •  Cost vs. Number Produced •  For a production facility •  It usually costs more to produce more

•  An outlier is visible •  A disaster (a fire at the factory)

Cost

•  High cost, but few produced

r = – 0.623

10,000

0

0

20 40 60 Number produced

Cost

5,000

Outlier removed: More details, r = 0.869

4,000 3,000 20 30 40 50 Number produced

17

Basic Concepts: OLS Modeling •  Salary vs. Years Experience •  For n = 6 employees •  Linear (straight line) relationship •  Increasing relationship •  higher salary generally goes with higher experience

Experience 15 10 20 5 15 5

Salary 30 35 55 22 40 27

Salary ($thousand)

•  Correlation r = 0.8667

60 50 40 30 20 0

10

20 Experience

18

Basic Concepts: OLS Modeling •  Summarizes bivariate data: Predicts Y from X •  with smallest errors (in vertical direction, for Y axis) •  Intercept is 15.32 salary (at 0 years of experience) •  Slope is 1.673 salary (for each additional year of experience, on average)

Salary (Y)

60 50 40 30 20 10 0

10

20

Experience (X)

19

Basic Concepts: OLS Modeling •  Predicted Value comes from Least-Squares Line •  For example, Mary (with 20 years of experience) has predicted salary 15.32+1.673(20) = 48.8 •  So does anyone with 20 years of experience

•  Residual is actual Y minus predicted Y •  Mary’s residual is 55 – 48.8 = 6.2 •  She earns about $6,200 more than the predicted salary for a person

with 20 years of experience •  A person who earns less than predicted will have a negative residual

20

Basic Concepts: OLS Modeling Mary’s residual is 6.2 60

Mary earns 55 thousand

50 Mary’s predicted value is 48.8 Salary

40 30 20 10 0

10

Experience

20

21

Basic Concepts: OLS Modeling •  Standard Error of Estimate

Se = SY

(1 − r ) nn −− 12 2

•  Approximate size of prediction errors (residuals) Actual Y minus predicted Y: Y–[a+bX] •  Example (Salary vs. Experience)

Se = 11.686 (1 − 0.86672 )

6 −1 = 6.52 6−2

Predicted salaries are about 6.52 (i.e., $6,520) away from actual salaries

22

Basic Concepts: OLS Modeling •  Interpretation: similar to standard deviation •  Can move Least-Squares Line up and down by Se •  About 68% of the data are within one “standard error of estimate” of the least-squares line •  (For a bivariate normal distribution)

Salary

60 50 40 30 20 0

10 Experience20

23

Multiple Linear Regression •  Linear Model for the Population Y = (α + β1 X1 + β2 X2 + … + βk Xk) + ε = (Population relationship) + Randomness •  Where ε has a normal distribution with mean 0 and constant

standard deviation σ, and this randomness is independent from one case to another •  An assumption needed for statistical inference

24

Multiple Linear Regression: Results •  Intercept: a •  Predicted value for Y when every X is 0 •  Regression Coefficients: b1, b2, …bk •  The effect of each X on Y, holding all other X variables constant •  Prediction Equation or Regression Equation (Predicted Y) = a+b1 X1+b2 X2+…+bk Xk •  The predicted Y, given the values for all X variables •  Prediction Errors or Residuals (Actual Y) – (Predicted Y)

25

Multiple Linear Regression: Results •  t Tests for Individual Regression Coefficients •  Significant or not significant, for each X variable •  Tests whether a particular X variable has an effect on Y, holding the other X variables constant •  Should be performed only if the F test is significant •  Standard Errors of the Regression Coefficients

Sb1 , Sb2 ,!, Sbk

(with n – k – 1 degrees of freedom)

•  Indicates the estimated sampling standard deviation of each

regression coefficient •  Used in the usual way to find confidence intervals and hypothesis tests for individual regression coefficients

26

Multiple Linear Regression: Results •  Predicted Page Costs for Audubon = a + b1 X1 + b2 X2 + b3 X3 = $4,043 + 3.79(Audience) – 124(Percent Male) + 0.903(Median Income) = $4,043 + 3.79(1,645) – 124(51.1) + 0.903(38,787)

= $38,966 •  Actual Page Costs are $25,315 •  Residual is $25,315 – 38,966 = –$13,651 •  Audubon has Page Costs $13,651 lower than you would expect for a magazine with its characteristics (Audience, Percent Male, and Median Income)

27

Standard Error •  Standard Error of Estimate Se •  Indicates the approximate size of the prediction errors •  About how far are the Y values from their predictions? •  For the magazine data •  Se = S = $21,578 •  Actual Page Costs are about $21,578 from their predictions for this

group of magazines (using regression) •  Compare to SY = $45,446: Actual Page Costs are about $45,446 from their average (not using regression) •  Using the regression equation to predict Page Costs (instead of simply using Y ) the typical error is reduced from $45,446 to $21,578

28

Coeff. of Determination The strength of association is measured by the square of the multiple correlation coefficient, R2, which is also called the coefficient of multiple determination.

R2 =

SS reg SS y

R2 is adjusted for the number of independent variables and the sample size by using the following formula: Adjusted

R2

= R2

k(1 - R 2) n-k-1

29

Coeff. of Determination •  Coefficient of Determination R2 •  Indicates the percentage of the variation in Y that is explained by (or attributed to) all of the X variables •  How well do the X variables explain Y? •  For the magazine data •  R2 = 0.787 = 78.7% •  The X variables (Audience, Percent Male, and Median Income) taken

together explain 78.7% of the variance of Page Costs •  This leaves 100% – 78.7% = 21.3% of the variation in Page Costs unexplained

30

The F test •  Is the regression significant? •  Do the X variables, taken together, explain a significant amount of the variation in Y? •  The null hypothesis claims that, in the population, the X variables do not help explain Y; all coefficients are 0 H0: β1 = β2 = … = βk = 0 •  The research hypothesis claims that, in the population, at least

one of the X variables does help explain Y H1: At least one of β1, β2, …, βk ≠ 0

31

The F test H0 : R2pop = 0 This is equivalent to the following null hypothesis:

H0: β1 = β2 = β 3 = . . . = βk = 0 The overall test can be conducted by using an F statistic: F=

SS reg /k SS res /(n - k - 1)

=

R 2 /k (1 - R 2 )/(n- k - 1)

which has an F distribution with k and (n - k -1) degrees of freedom.

32

Performing the F test •  Three equivalent methods for performing F test; they

always give the same result •  Use the p-value •  If p < 0.05, then the test is significant •  Same interpretation as p-values in Chapter 10

•  Use the R2 value •  If R2 is larger than the value in the R2 table, then the result is significant •  Do the X variables explain more than just randomness?

•  Use the F statistic •  If the F statistic is larger than the value in the F table, then the result is

significant

33

Example: F test •  For the magazine data, The X variables (Audience, Percent Male, and Median Income) explain a very highly significant

percentage of the variation in Page Costs •  The p-value, listed as 0.000, is less than 0.0005, and is therefore

very highly significant (since it is less than 0.001) •  The R2 value, 78.7%, is greater than 27.1% (from the R2 table at level 0.1% with n = 55 and k = 3), and is therefore very highly significant •  The F statistic, 62.84, is greater than the value (between 7.054 and 6.171) from the F table at level 0.1%, and is therefore very highly significant

34

t Tests •  A t test for each regression coefficient •  To be used only if the F test is significant •  If F is not significant, you should not look at the t tests

•  Does the jth X variable have a significant effect on Y, holding the

other X variables constant? •  Hypotheses are H0: βj = 0, H1: βj ≠ 0 •  Test using the confidence interval

b j ± tSb j

•  use the t table with n – k – 1 degrees of freedom

•  Or use the t statistic

tstatistic = b j / Sb j

•  compare to the t table value with n – k – 1 degrees of freedom

35

Example: t Tests •  Testing b1, the coefficient for Audience b1 = 3.79, t = 13.5, p = 0.000 •  Audience has a very highly significant effect on Page Costs, after

adjusting for Percent Male and Median Income

•  Testing b2, the coefficient for Percent Male b2 = – 124, t = – 0.90, p = 0.374 •  Percent Male does not have a significant effect on Page Costs, after

adjusting for Audience and Median Income

•  Testing b3, the coefficient for Median Income b3 = 0.903, t = 2.44, p = 0.018 •  Median Income has a significant effect on Page Costs, after adjusting

for Audience and Percent Male

36

Assumptions in Regression •  Assumptions underlying the statistical techniques

should be tested twice •  First for the separate variables •  Second for the multivariate model variate, which acts

collectively for the variables in the analysis and thus must meet the same assumption as individual variables. Differs for different multivariate technique

Assumptions in Regression •  Linearity •  The independent variable has a linear relationship with the dependent variable •  Normality •  The residuals or the dependent variable follow a normal distribution •  Multicollinearity •  When some X variables are too similar to one another •  Homoskedasticity •  The variability in Y values for a given set of predictors is the same regardless of the values of the predictors •  Independence among cases (Absence of correlated errors) •  The cases are independent of each other

38

Assumptions in Regression Normality •  The residuals or the dependent variable follow a normal

distribution •  If the variation from normality is significant then all statistical tests are invalid •  Graphical Analysis •  Histogram and Normal probability plot •  Peaked and Skewed distribution result in non-normality

•  Statistical Analysis

•  If Z value exceeds critical value, then the distribution is non-

normal •  Kolmogorov – Smirnov Test; Shapiro-Wilk’s Test

39

Assumptions in Regression Normality

40

Assumptions in Regression Homoskedasticity •  Assumption related primarily to dependence

relationships between variables •  Assumption that the dependent variable(s) exhibit equal levels of variance across the range of predictor variable(s). •  The variance of the dependent variable should not be concentrated in only a limited range of the independent values •  Source •  Type of variable •  Skewed distribution

41

Assumptions in Regression Homoskedasticity •  Graphical Analysis •  Analysis of residuals in case of Regression •  Statistical Analysis •  Variances within groups formed by non-metric variables •  Levene Test •  Box’s M Test •  Remedy •  Data Transformation

42

Assumptions in Regression Homoskedasticity •  Graphical Analysis

43

Assumptions in Regression Linearity •  Assumption for all multivariate techniques based on

correlational measures such as •  multiple regression, •  logistics regression, •  factor analysis, and •  structural equation modeling

•  Correlation represents only the linear association

between variables •  Identification

•  Scatterplots or examination of residuals using regression

•  Remedy •  Data Transformations

44

Assumptions in Regression Linearity

45

Assumptions in Regression Absence of Correlated Errors •  Prediction errors should not be correlated with each

other •  Identification •  Most possible cause is the data collection process, such as

two separate groups in the data collection process •  Remedy •  Including the omitted causal factor into the multivariate analysis

46

Assumptions in Regression Multicollinearity •  Multicollinearity arises when intercorrelations among the predictors

are very high. •  Multicollinearity can result in several problems, including: •  The partial regression coefficients may not be estimated precisely. The standard errors are likely to be high. •  The magnitudes as well as the signs of the partial regression coefficients may change from sample to sample. •  It becomes difficult to assess the relative importance of the independent variables in explaining the variation in the dependent variable. •  Predictor variables may be incorrectly included or removed in stepwise regression.

47

Assumptions in Regression Multicollinearity •  The ability of an independent variable to improve the prediction of the

dependent variable is related not only to its correlation to the dependent variable, but also to the correlation(s) of the additional independent variable to the independent variable(s) already in the regression equation •  Collinearity is the association, measured as the correlation,

between tow independent variables •  Multicollinearity refers to the correlation among three or more independent variables •  Impact •  Reduces any single IVs predictive power by the extent to which it is

associated with the other independent variables

48

Assumptions in Regression Multicollinearity •  Measuring Multicollinearity •  Tolerance •  Amount of variability of the selected independent variable not explained

by the other independent variables •  Tolerance Values should be high •  Cut-off is 0.1 but greater than 0.5 gives better results •  VIF •  Inverse of Tolerance •  Should be low (typically below 2.0 and usually below 10)

49

Assumptions in Regression Multicollinearity •  Remedy for Multicollinearity •  A simple procedure for adjusting for multicollinearity consists of using only one of the variables in a highly correlated set of variables. •  Omit highly correlated independent variables and identify other independent variables to help the prediction •  Alternatively, the set of independent variables can be transformed into a new set of predictors that are mutually independent by using techniques such as principal components analysis. •  More specialized techniques, such as ridge regression and latent root regression, can also be used.

50

Assumptions in Regression Data Transformations •  To correct violations of the statistical assumptions

underlying the multivariate techniques •  To improve the relationship between variables •  Transformation to achieve Normality and Homoscedasticity •  Flat Distribution – Inverse transformation •  Negatively Skewed Distribution – Square Root

Transformation •  Positively Skewed Distribution – Logarithmic Transformation •  If the residuals in regression are cone shaped then •  Cone opens to right – Inverse transformation •  Cone opens to left – Square root transformation

51

Assumptions in Regression Data Transformations •  Transformation to achieve

Linearity

52

Assumptions in Regression Data Transformations

53

Assumptions in Regression •  General guidelines for transformation •  For a noticeable effect of transformation the ratio of a variable’s mean to the standard deviation should be less than 4.0 •  When the transformation can be performed on either of the two variables, select the one with smallest ratio of mean/sd. •  Transformation should be applied to independent variables except in case of heteroscedasticity •  Heteroscedasticity can only be remedied by transformation of the dependent variable in a dependent relationship •  If the heteroscedastic relationship is also non-linear the dependent variable and perhaps the independent variables must be transformed •  Transformations may change the interpretation of the variables

54

Issues in Regression •  Variable Selection •  How to choose from a long list of X variables? •  Too many: waste the information in the data •  Too few: risk ignoring useful predictive information

•  Model Misspecification •  Perhaps the multiple regression linear model is wrong •  Unequal variability? Nonlinearity? Interaction?

EXPLANATORY VS PREDICTIVE MODELING

Explanatory Vs Predictive Modeling •  Explanatory models fits the data closely, whereas a good

predictive model predicts new cases accurately •  Explanatory models uses entire dataset for estimating the best-fit model and to maximize explanatory variance (R2). Predictive models estimate the model on training set and assess it on the new, unobserved data •  Performance measures for explanatory models measures how close the data fit the models, whereas in predictive models performance is measured by predictive accuracy

Performance Evaluation •  Prediction Error for observation ‘i’= Actual y value –

predicted y value •  Popular numerical measures of predictive accuracy •  MAE or MAD (Mean absolute error / deviation)

•  Average Error

•  MAPE (Mean Absolute Percentage Error)

Performance Evaluation •  RMSE (Root mean squared error)

‘ •  Total SSE (total sum of squared errors)

CASE

Case: Pedigree Vs Grit •  Why does a low R2 does not make the regression useless? •  Describe a situation in which a useless regression has a high R2. •  Check the validity of the linear regression model assumptions. •  Estimate the excess returns of Bob’s and Putney’s funds. Between them, who •  •  •  •  • 

is expected to obtain higher returns at their current funds and by how much? If hired by the firm, who is expected to obtain higher returns and by how much? Can you prove at the 5% level of significance that Bob would get higher expected returns if he had attended Princeton instead of Ohio State? Can you prove at the 10% level of significance that Bob would get at least 1% higher expected returns by managing a growth fund? Is there strong evidence that fund managers with MBA perform worse than fund managers without MBA? What is held constant in this comparison? Based on your analysis of the case, which candidate do you support for AMBTPM’s job opening: Bob or Putney? Discuss

Case: Nils Baker •  Is the presence of a physical Bank Branch creating

demand for checking accounts?

Thank You