## Abstract

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.

This is a preview of subscription content, access via your institution.

## References

AERA, APA, NCME (1999).

*Standards for educational and psychological testing*. Washington: American Educational Research Association.Aguinis, H., Boik, R.J., & Pierce, C.A. (2001). A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression.

*Organizational Research Methods*,*4*, 291–323.Aguinis, H., Boik, R.J., Pierce, C.A., & Beaty, J.C. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review.

*Journal of Applied Psychology*,*90*, 94–107.Aguinis, H., Culpepper, S.A., & Pierce, C.A. (2010). Revival of test bias research in preemployment testing.

*Journal of Applied Psychology*,*95*, 648–680.Aguinis, H., & Stone-Romero, E.F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power.

*Journal of Applied Psychology*,*82*, 192–206.Aitkin, M. (1964). Correlation in a singly truncated bivariate normal distribution.

*Psychometrika*,*29*, 263–270.Arnold, B.C., Beaver, R.J., Groeneveld, R.A., & Meeker, W.Q. (1993). The nontruncated marginal of a truncated bivariate normal distribution.

*Psychometrika*,*58*, 471–488.Balakkrishnan, N., & Lai, C. (2009).

*Continuous bivariate distributions*. New York: Springer.Barr, D.R., & Sherrill, E.T. (1999). Mean and variance of truncated normal distributions.

*The American Statistician*,*53*, 357–361.Bartlett, C.J., Bobko, P., Mosier, S.B., & Hannan, R. (1978). Testing for fairness with a moderated regression strategy: an alternative to differential analysis.

*Personnel Psychology*,*31*, 233–241.Birnbaum, Z.W., Paulson, E., & Andrews, F.C. (1950). On the effect of selection performed on some coordinates of a multi-dimensional population.

*Psychometrika*,*15*, 191–204.Borsboom, D., Romeijn, J., & Wicherts, J.M. (2008). Measurement invariance versus selection invariance: is fair selection possible?

*Psychological Methods*,*13*, 75–98.Cleary, T.A. (1968). Test bias: prediction of grades of Negro and White students in integrated colleges.

*Journal of Educational Measurement*,*5*, 115–124.Culpepper, S.A. (2010). Studying individual differences in predictability with gamma regression and nonlinear multilevel models.

*Multivariate Behavioral Research*,*45*, 153–185.Culpepper, S.A. (in press). Evaluating EIV, OLS, and SEM estimators of group slope differences in the presence of measurement error: the single indicator case.

*Applied Psychological Measurement*.Culpepper, S.A., & Aguinis, H. (2011). Analysis of covariance (ANCOVA) with fallible covariates.

*Psychological Methods*,*16*, 166–178.Culpepper, S.A., & Davenport, E.C. (2009). Assessing differential prediction of college grades by race/ethnicity with a multilevel model.

*Journal of Educational Measurement*,*46*, 220–242.DeCorte, W., Lievens, F., & Sackett, P.R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact.

*Journal of Applied Psychology*,*92*, 1380–1393.Dorans, N.J. (2004). Using subpopulation invariance to assess test score equity.

*Journal of Educational Measurement*,*41*, 43–68.Finch, D.M., Edwards, B.D., & Wallace, J.C. (2009). Multistage selection strategies: simulating the effects on adverse impact and expected performance for various predictor combinations.

*Journal of Applied Psychology*,*94*, 318–340.Gratz v. Bollinger (2003). 539 U.S. 244.

Grutter v. Bollinger (2003). 539 U.S. 306.

Holmes, D.J. (1990). The robustness of the usual correction for restriction in range due to explicit selection.

*Psychometrika*,*55*, 19–32.Humphreys, L. (1952). Individual differences.

*Annual Review of Psychology*,*3*, 131–150.Humpreys, L.G. (1986). An analysis and evaluation of test and item bias in the prediction context.

*Journal of Applied Psychology*,*71*, 327–333.Kuncel, N.R., Hezlett, S.A., & Ones, D.S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance.

*Psychological Bulletin*,*127*, 162–181.Lautenschlager, G.J., & Mendoza, J.L. (1986). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction.

*Applied Psychological Measurement*,*10*, 133–139.Lawley, D.N. (1943). A note on Karl Pearson’s selection formulae.

*Proceedings of the Royal Society of Edinburgh. Section A, Mathematical and Physical Sciences**62*, 28–30.Linn, R.L. (1983). Pearson selection formulas: implications for studies of predictive bias and estimates of educational effects in selected samples.

*Journal of Educational Measurement*,*20*, 1–15.Linn, R.L. (1984). Selection bias: multiple meanings.

*Journal of Educational Measurement*,*21*, 33–47.Linn, R.L., & Werts, C.E. (1971). Considerations for studies of test bias.

*Journal of Educational Measurement*,*8*, 1–4.Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance.

*Psychometrika*,*58*, 525–543.Millsap, R.E. (1997). Invariance in measurement and prediction: their relationship in the single-factor case.

*Psychological Methods*,*2*, 248–260.Millsap, R.E. (1998). Group differences in regression intercepts: implications for factorial invariance.

*Multivariate Behavioral Research*,*33*, 403–424.Millsap, R.E. (2007). Invariance in measurement and prediction revisited.

*Psychometrika*,*72*, 461–473.Millsap, R.E., & Kwok, O.M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations.

*Psychological Methods*,*9*, 93–115.Muthen, B. (1990). Moments of the censored and truncated bivariate normal distribution.

*British Journal of Mathematical & Statistical Psychology*,*43*, 131–143.Pearson, K. (1903). Mathematical contributions to the theory of evolution—xi. On the influence of natural selection on the variability and correlation of organs.

*Philosophical Transactions*,*CC.–A 321*, 1–66.Ricci v. DeStefano (2009). 557 U.S. 244.

Rosenbaum, S. (1961). Moments of a truncated bivariate normal distribution.

*Journal of the Royal Statistical Society. Series B. Methodological*,*23*, 405–408.Saad, S., & Sackett, P.R. (2002). Investigating differential prediction by gender in employment-oriented personality measures.

*Journal of Applied Psychology*,*87*, 667–674.SIOP (2003).

*Principles for the validation and use of personnel selection procedures*(4th ed.). Bowling Green: Society for Industrial and Organizational Psychology, Inc.Wicherts, J.M., & Millsap, R.E. (2009). The absence of underprediction does not imply the absence of measurement bias.

*The American Psychologist*,*64*, 281–283.Young, J.W. (2001).

*Differential validity, differential prediction, and college admission testing: a comprehensive review and analysis*(Research Report No. 6). College Entrance Examination Board.

## Author information

### Affiliations

### Corresponding author

## Additional information

An erratum to this article is available at http://dx.doi.org/10.1007/s11336-013-9345-1.

## Appendix: Additional Derivations

### Appendix: Additional Derivations

### A.1 Observed Prediction Invariance Under the Criterion-Predictor Factor Model

This subsection derives the conditions in (13) and (14) for observed prediction invariance under the criterion-predictor factor model. The condition for slope invariance is fairly straightforward to derive. Specifically, the observed OLS slope for group *i* is the ratio of the covariance between *Z*
_{
i
} and *Y*
_{
i
} to the variance of *Z*
_{
i
} from (12),

Consequently, the condition for slope invariance in (14) arises when *β*
_{1i
}=B_{1}.

Subgroup intercepts are defined as *β*
_{0i
}=*E*(*Y*
_{
i
})−*β*
_{1i
}
*E*(*Z*
_{
i
}). Substituting the definitions of *E*(*Y*
_{
i
}), *E*(*Z*
_{
i
}), and *β*
_{1i
} into the definition of *β*
_{0i
} and rearranging terms yields,

Subgroup intercept invariance occurs when *β*
_{0i
}=B_{0}, which confirms (13).

### A.2 Derivations to Support Theorem 1

An expression for *β*
_{1s
}(*HR*) is defined in the first element of \(\varSigma_{zGs}^{-1}\varSigma_{zGys}\) where *Σ*
_{
zGs
} is the variance–covariance matrix of *Z*
_{
s
} and *G* and *Σ*
_{
zGys
} is a vector of covariances between *Z*
_{
s
} and *Y*
_{
s
} and *G* and *Y*
_{
s
}. Multiplying the elements in \(\varSigma_{zGs}^{-1}\varSigma_{zGys}\) implies that *β*
_{1s
}(*HR*) is defined as:

Results concerning finite mixtures imply that *σ*
_{
zys
} and \(\sigma_{zs}^{2}\) can be written as functions of *p*
_{
s
} and subgroup covariances and variances. Specifically, *σ*
_{
zys
} and \(\sigma_{zs}^{2}\) are defined as

Consequently, *β*
_{1s
}(*HR*) is defined as

Observed subgroup slopes will equal the common slope only in the absence of observed slope differences. For instance, *β*
_{11s
}=*β*
_{1s
}(*HR*) if and only if *β*
_{11s
}=*β*
_{12s
}. More formally, *β*
_{11s
}=*β*
_{1s
}(*HR*) when,

Cross multiplying in (A.7) and simplifying terms implies that *β*
_{11s
}=*β*
_{1s
}(*HR*) only when *β*
_{11s
}=*β*
_{12s
}. Additionally, *β*
_{11s
}>*β*
_{1s
}(*HR*) if *β*
_{11s
}>*β*
_{12s
} and vice versa.

### A.3 Derivation of Correlations Among Variables

This subsection includes additional derivations to support several equations used to compute \(\sigma_{W_{y}}^{2}\) and to compute the probability of detecting intercept and slope differences with OLS. Note that the derivations employ the use of finite mixtures and the fact that *G* is a categorical variable that codes group membership using zero or one. The derivations of correlations proceed by specifying covariances between variables as the expected value of the product of the random variables less the product of their expected values. The first subsection presents derivations for computing correlations necessary to compute \(\sigma_{W_{y}}^{2}\) and the second subsection includes derivations to compute Type I error and power rates of OLS tests of intercept and slope differences.

#### A.3.1 Derivations of Correlations for Computing \(\sigma_{W_{y}}^{2}\)

This section derives expressions for \(\sigma_{W_{z}G}^{2}\), \(\rho_{W_{z},W_{z}G}\), and \(\rho_{G,W_{z}G}\). First, the variance of the product *W*
_{
z
}
*G* is,

Substituting the definitions of *ϕ*
_{1} and *κ*
_{1} from (5) and (7) into the above equation yields

By definition, the *E*(*W*
_{
z
})=0 and \(E (W_{z}^{2} )=1\) across the two groups, so the correlation between *W*
_{
z
} and *W*
_{
z
}
*G* is,

The correlation between *G* and *W*
_{
z
}
*G* can be found in a similar manner. Specifically, \(\rho_{G,W_{z}G}\) is,

#### A.3.2 Derivations of Correlations Among *Z*
_{
s
}, *G*, *Z*
_{
s
}
*G*, and *Y*
_{
s
}

This section derives expressions for *ρ*
_{
z,zGs
}, *ρ*
_{
G,zGs
}, and *ρ*
_{
zG,ys
}. Note that expressions for *ρ*
_{
z,Gs
} and *ρ*
_{
y,Gs
} are not derived, because these correlations are simply biserial correlations. It is important to first derive \(\sigma_{zGs}^{2}\), which is equal to

where the last equality was obtained by substituting *μ*
_{
z1s
}=*μ*
_{
zs
}+(1−*p*
_{
s
})Δ*μ*
_{
zs
} into (A.12) and factoring out a 1−*p*
_{
s
} from the similar terms.

The correlation between *Z*
_{
s
} and *Z*
_{
s
}
*G* is,

Substituting the definition of *μ*
_{
zs1} into (A.13) and rearranging terms produces the result in (41). The correlation between *G* and *Z*
_{
s
}
*G* is similarly expressed as the covariance divided by respective standard deviations:

where the last equation was established by substituting for *μ*
_{
z1s
} and factoring out 1−*p*
_{
s
}.

## Rights and permissions

## About this article

### Cite this article

Culpepper, S.A. Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression.
*Psychometrika* **77, **561–580 (2012). https://doi.org/10.1007/s11336-012-9270-8

Received:

Revised:

Published:

Issue Date:

### Key words

- selection
- prediction bias
- measurement bias
- type I error
- power