Anne Anastasi- Psychological Testing I

ANNE~NASTASI Professor of Psychology, Fordham Universiry

Psyclwlvgical Testing

MACMILLAN

PUBLISHING

New York Collier Maonillan Publishers London

CO.,

INC.

All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the Publisher. Earlier editions copyright 1954 and © 1961 by Macmillan Publishing Co., Inc., and copyright © 1968 by Anne Anastasi. MACMILLAN PUBLISHING Co.,

INC.

866 Third Avenue, New York, New York 10022 COLLIER MACMILLAN CANADA, LTD.

Librarlj of Congress Cataloging in Publication Data Anastasi, Anne, (date) Psychological testing. Bibliography: p. Includes indexes. 1. Mental tests. 2. Personality tests. I. Title. [DNLM: 1. Psychological tests. WM145 A534P] BF431.A573 1976 153·9 75-2206 ISBN O-2-30298entialcomponent of knowledge about contemporary testing. For these reasons, specific tests are again examined and evaluated in Parts 3, 4, and 5. These tests have been chosen either because they are outstanding examples with which the student of testing should be familiar or because they illustrate some special point of test construction or interpretation. In the text itself, the principal focus is on types of tests rather than on specific instruments. At the same time, Appendix E contains a classified list of over 250 tests, including not only those cited in the text but also others added to provide a more representative sample. As for the differences-they loomed especially large during the preparation of this edition. Much that has happened in human society since the mid-1960's has had an impact on psychological testing. Some of these developments were briefly described in the last two chapters of the third edition. Today they have become part of the mairn;tream.;()fpsychological' testing and have been accordingly incorpo~i-ted in the apprqpqate sections throughout the book. Recent changes in psychological Jesting that are reflected in the present edition can be delpribed on three levels: (1) general orientation toward testing, (2) Stlbm,IJ,tiveand inethod()l~ical developments, and (3) "ordinary progress" w1)Q as the publiciitibn of new tests and revision of earlier tests.

Preface

Preface

; An example of changes on the first level is the increasing awareness of ~e ethical, social, and legal implications of t~sting. In the present edilon, this topic has been expanded and treated 111a separate chapter early b the book (Ch. 3) and in Appendixes A and B. A cluster of related evelopments represe~ts a bro~dening of.test u~es..Beside~ the tradi~ion~l 'pplications of tests 111 selectwn and diagnosIs, 111creasmg attention IS eing given to administering tests for self-kuowledge and self-develop~entl and to training individuals in the use of their own test res?lts. in ,lJecisionmaking (Chs. 3 and 4). In the same category are the contmumg ~eplacementof global scores with multitrait profiles and the application bf classificationstrategies, whereby "everyone can be above average" in bne or more socially valued "ariables (Ch. 7). From another angle, rffortsare being made to modify traditional interpretations of test scores, ~n bothcognitive and noncognitive areas, in the light of accumulating psychological knowledge. In this edition, Chapter 12 brings together 'psychologicalissues in the interpretation of intelligence test scores, :touchingon such problems as stability and change in intellectual level .overtime; the nature of intelligence; and the testing of intelligence in :earlychildhood, in old age, and in different cultures. Another example is provided by the increasing emphasis on situational specificity and person-by-situationinteractions in personality testing, stimulated in large partbythe social-learning theorists (Ch. 17). T~e second level, -covering substantive and methodological changes, is illustratedby the impact of computers on the development, administra"tioll,scoring, and interpretation of tests (see especially Chs. 4, 11, 13, 17, 18, W). The use of computers in administering or managing instructional pro/ramshas also stimulated the development of criterion-referenced t~~~ although other conditions have contributed to the upsurge of 'i!restin such tests in education. Criterion-referenced tests are discussed '1 ,. 'pally in Chapters 4,5, and 14. Other types of lllstruments that have to prominence and have received fuller treatment in the present n include: tests for identifying specific learning disabilities (Ch. inventories and other devices for use in behavior modification pro-' (Ch. 20), instruments for assessing early ch~ldhOod education 14), Piagetian "ordinal" scales (Chs. 10 and 14), basic education literacy tests for adults (Cbs. 13 and 14), and techniques for the ment of environments (Ch. 20). Problems to be considered in the , ment of minority groups, including the question of test bias, are ined from different angles in Chapters 3, 7, 8, and 12. the third level, it may be noted that over 100 of the tests listed in edition have been either initially pUblished or revised since the ication of the preceding edition (1968). Major examples include the arthy Scales of Children's Abilities, the WISC-R, the 1972 Stanfordnorms (with all the resulting readjustments in interpretations),

l..

I

c

•

vii

Forms Sand T of the DAT (including a computerized Career Planning Program), the Strong-Campbell Interest Inventory (merged form of the SVIB), and the latest revisions of the Stanford Achievement Test and the Metropolitan Readiness Tests. It is a pleasure to acknowledge the assis~nce received from many sources in the preparation of this edition. The completion of the project was facilitated by a one-semester Faculty Fellowship awarded by Fordham University and by a grant from the Fordham University Research Council covering principally the services of a research assistant. These services were performed by Stanley Friedland with an unusual combination of expertise, responSibility, and graciousness. I am indebted to the many authors and test publishers who provided reprints, unpublished manuscripts, specimen sets of tests, and answers to my innumerable inquiries by mail and telephone. For assistance extending far beyond the interests and responsibilities of any single publisher, I am especially grateful to Anna Dragositz of Educational Testing Service and Blythe Mitchell of Harcourt Brace Jovanovich, Ioc. I want to acknowledge the Significant contribution of John T. Cowles of the University of Pittsburgh, who assumed complete responSibility for the preparation of the Instructor's Manual to accompany this text. For informative discussions and critical comments on particular topics, I want to convey my sincere thanks to Willianl H. Angoff of Educational Testing Service and to several members of the Fordham University Psychology Department, including David R. Chabot, Marvin Reznikoff, Reube~ M. Schonebaum, and 'Warren, W. Tryon. Grateful acknowledgment IS also made of the thoughtful recommendations submitted by course instructors in response to the questionnaire distributed to current users of the third edition. Special thanks in this connection am due to Mary Carol Cahill for her extensive, constructive, and Wide-ranging suggestions. I wish to express my appreciation to Victoria Overton of the Fordham University library staff for her efficient and courteous assistance in bibliographic matters. Finany, I am happy to record the contributions of my husband, John Porter Foley, Jr., who again participated in the solution of countless problems at all stages in the preparation of the book. A.A.

CONTENTS

PART 1 CONTEXT

OF PSYCHOLOGICAL

TESTING

1. FUNCTIONS AND ORIGINS OF PSYCHOLOGICAL TESTING

Q

Current uses of psychological tests Early interest in classification and

training

retarded 5 The first experimental psychologists 7 Contributions of Francis Galton 8 Cattell and the early "mental tests" 9 Binet and the nse of intelligence tests

3 of

the

mentally

10

Group testing 12 Aptitude testing 13 ~ Standardized achievement tests 16 Measurement of personality 18 Sources of information about tests 20

2. NATURE AND USE OF PSYCHOLOGICAL TESTS What is a psychological test? 23 Reasons for controlling the use of psychological tests Test administration 32 Rapport 34 Test anxiet\' 37 Examiner ~nd situational variables 39 Coaching, practice, and test sophistication

3. SOCIAL AND ETHICAL OF TESTING

41

IMPLICATIONS "

User qualifications 45 Testing instruments and procedures 47 Protection of privacy . 49 Confidentiality 52 Communicating test results 56 Testing and the civil rights of minorities ix

57

PART 3 TESTS OF GENERAL

INTELLECTUAL

LEVEL 9. INDIVIDUAL 4. NORMS AND THE INTERPRETATION TEST

OF

SCORES

Statistical concepts 68 Developmental norms 73 Within-group norms 77 Relativity of norms 88 Computer utilization in tile interpretation Criterion-referenced

testing

10. TESTS of test scores

TESTS

Stanford-Binet Intelligence Scale 230 Wechsler Adult Intelligence Scale 245 Wechsler Intelligence Scale for Children 2.'55 Wechsler Preschool and Primary Scale of Intelligence

94

FOR SPECIAL

POPULATIONS

Infant and preschool testing 266 Testing the physically handicapped Cross-cultural testing 287

96

281

5, RELIAB ILITY The correlation coefficient 104 Types of reliability 110 Reliability of speeded tests 122 Dependence of reliability coefficients on the sample tested Standard error of measurement 127 Reliability of criterion-referenced tests 131

Content validity 134 Criterion-related validity Construct validity 151 Overview 158

140

AND

Validity coefficient and error of estimate 163 Test validity and decision theory 167 Moderator variabll;;s 177 Combining information from different tests 180 Use of tests for cl.assification decisions 186 Statistical analyses of test bias 191

ANALYSl-S

Item difficulty 199 Item validity 206 Internal consistency 215 Item analysis of speeded tests Cross validation 219 Item-group interaction 222

12. PSYCHOLOGICAL INTELLIGENCE

ISSUES IN TESTING

Longitudinal studies of intelligence 327. Intelligence in early childhood 332 Problems in the testing of adult intelligence Problems in cross-cultural testing 343 Nature of intelligence 349

7. VALIDITY: MEASUREMENT INTERPRET ATION

8. ITEM

125

Croup tests versus individual tests 299 Multilevel batteries 305 Tests for the college level and beyond 318

PART 4 TESTS OF SEPARATE 13. MEASURING

14. EDUCATIONAL

217

AInLJTIES

MULTIPLE

Factor analysis 362 Theories of trait organization MUltiple aptitude batteries Measurement of creativity

337

APTITUDES 369

378

388

TESTING

Achievement tests: their nature and uses General achievement batteries 403 Standardized tests in separate subjects Teacher-made classroom tests 412

398 410

260

20. OTHER Diagnostic and criterion-rdt:renced tests Specialized prognostic tests 423 Assessment in early childhood education

~ OCCUPATIONAL \V Validation of industrial

417 425

TESTING

tests 435 Short screening tests .for industrial personnel Special aptitude tests 442 Testing in the profeSSions 458

ASSESSMENT

439

Diagnostic use of intelligence tests 465 Special tests for detecting cognitive dysfunction Identifying specific learning disabilities 478 Clinical judgment 482 Report writing 487

B. Guidelines on Employee Selection Procedures (EEOC) Guidelines for Reporting Criterion-Related and Content Validity (OFCC)

PART 5 PERSON ALITY 17. SELF-REPORT

TESTS

INVENTORIES

Content validation 494 Empirical criterion keying - 496 506 Factor analysis in test development 510 Personality theory in test development 515 Test-taking attitudes and response sets Situational specificity 521 Evaluation of personality inventories

18. MEASURES OF INTERESTS, AND VALUES

ATTITUDES,

Interest inventories 528 Opinion and attitude measurement 543 Attitude scales 546 Assessment of values and related variables

19. PROJECTIVE

TECHNIQUES

"Objective" performance tests 588 Situational tests 593 SeH-concepts and personal constructs 598 Assessment techniques in behavior modification programs Observer reports 606 Biographical inventories 614 The assessment of environments 616

;527

552

TECHNIQUES

Nature of projective techniques 558 Inkblot techniques 559 Thematic Apperception Test and related instruments Other projective techniques 569 Evaluation of projective techniques 576

PART

C01ltext of . Psychological Testing

1

CHAPTER

1

Functions and 01~igiTlS of Psycllological TeStiTlg

'

NYONE reading this book today could undoubtedly illush'ate what . is meant by a psychological test, It would be easy enough to recall . a test the reader himself has taken in school, in college, in the armed services, in the counseling center, or in the personnel office. Or perhaps the reader has served as a subject in an experiment in which standardized tests were employed. This would certainly not have been the case fifty years ago. Psychological testing is a relatively young branch of one of the youngest of the sciences.

A

Basically, the function of psychological tests is to measure ,9.:iffe~~~.n~L_ 1Jetween individuals or between the reactions of the same individual on different occasions. One of the first problems that stimulated the development of psychological tests was the identification of the mentally retarded. To this day, the detection of int~i1ectual deficiencies remains an Important application of certain types of psychological tests. Related clinical uses of tests include the examination of the emotionally disturbed, the delinquent, and other types of behavioral deviartts. A strong impetus to the early development of tests was likewise provided by problems arising in education, At present, schools are among the largest test users. The classifica.tiOIlOfchildren with reference to their ability to profit from different types of school instruction, the identi£ication of the intellectually retarded on the one hand and the gifted on the other, the diagnosis of academic failures, the educational and vocational counseling of high school and college students, and the s~~ction of applicants for professional and other special schools are among the many educational ~uses of tests. The selection and classification of industrial personnel represent another major application of psychological testing. From the assembly-line

4

COllfcl't

of Psychological Testing

operator or filing clerk to top management, there is scarcely a type of job for which some kind of psychological test has not proved helpful in such matters as hiring, job assignment, transfer, promotion, or termination. To be sure, the effective employment of tests in many of these situations, es eciiill-"Tri('Onnection with high-level jobs, usuall • re uires that the t!.:ts he used as an adjunct to s -i u interviewing, so that test scores may be properly int~rpreteaTnt1leli ht of other back ound' rmatiQn a out the m IVI un. evertheless, testing constitutes an important part ~ total personnel program. A closely related application of psychological testing is to be found in the selection and classification of military personnel. From simple beginnings in "Vorld 'War I, the scope and variety of psychological tests employed in military sihlations underwent a phenomenal increase during World War II. Subsequently, research on test development has been continuing on a large scale in all branches of the armed services, The use of tests in counseling has gradually broadened from a narrowly defined guidance regarding educational and vocational plans to an involvement with all aspects of the person's life. Emotional wellbeing and effective interpersonal relations have become increasingly prominent objectives of counseling. There is growing emphasis, too, on the use of tests to enhance self-understanding and personal development. Within this framework, test scores are part of the information given to the individual as aids to his own decision-making processes. It is clearly evident that psychological tests are currently being employed in the solution of a wide range of practical problems. One should not, however, lose sight of the fact that such tests are als? serving important functions in basic research Nearly all problems in differential psychology, for example, require testing procedures as a means of gathering data. As illustrations, reference may be made to studies on the nature and extent of individual differences, the identification of psychological traits, the measurement of group:' differences, ~nd the investigationfijo]ogical and cUltural factors associated WIth 6ehavioral differences. For all such areas of research-and for many others-the precise mt>.asurement of individual differences made possible by well-constructed tests is an essential prerequisite. Similarly, psycholOgical tests provide standardized tools for investigating such varied problems as life-span developmental changes within the individual, the relative effectiveness of different educational procedures, the outcomes of psychotherapy, the impact of community programs, and the influence of noise on performance. From the many different uses of psychological tests, it follows that some knowledge of such tests is needed for an adequate understanding of most fields of contemporary psychology. It is primarily with this end in view that the present book has been prepared. The book is not designed to

make the individual either n skilled examiner and test administrator or an"experf on test construction. It is directed, not to the test specialist, but to the general student of psychology. Some acquaintance with the lead·' ing current tests is necessary in order to understand references to the use of such tests in the psychological literature. And a proper evaluation and interpretation of test results must ultimately rest on a knowledge of how the tests were constructe Goodenough (1949, pp. 50-51) notes that in 1881, 2l y~aTs befor~ the appearance of the 1908 Binet-Simon Scale, S. E. Chaille publi!iheq in the New Orleans Medical a~d Surgical Journal a series of tests for infan~ 11l7anged according to the a!1:eat whIch the tests are commonly passed. Partly because' of the limited circulation of the journal 'nd partly, perhaps, because the scientific ~Om!J1l1nity was not ready for it, the significance of this age-scale concept passed unnoticed at the time. Binet's own scale was in~ed by the work oE some oE ~is contemporaries, notably Blin and Damaye, who prepared a set of oral questions from which they derived a single global score Eor eaclrdiild (T. H. Wolf, 1973). .

12

Context

Functions and Origins of Psyc1101ugical Testing

of Psyc11010gical Testing

attention among psychologists throughout the world. Translation~ and adaptations appeared in many lang;uages. In Americ;l, a number of diHerent revisions were prepa.red, the most famous of which is the one developed under the direction of L. ~t Terman a.t Stanford University, and known as the Stanfmd-Binet (Terman, 1916). It was in this test that the intelligence quotient (IQ), or mtio between mental age and chronological age, was first used. The latest revision of this test is widely employed today and will be more full\' considered in Chapter 9. Of special interest, too. is the first Kuhlmann-Binet revision, which extended the scale downward to the age level of 3 months (Kuhlmann, 1912). This scale represents one of the earliest efforts to develop preschool and infant tests of intelligence.

The Binet tests, as well as all their revisions, are indil;iclual scales in the sense that the\" can be administered to onlY one person at a time. Man\' of the tests in these scales require .oral re~ponses from the subject or n~cessitate the manipulation of materials. Some call for individual timing of responses. For these and other reasons, such tests are not adapted to group administration. Another characteristic of the Binet type of test is that" it requires a highly trained examiner. Such tests are essentiallv clinical instruments, suited to the intensive study of individual J

.'

•

cases. Group testing, like the first Binet scale, was developed to meet a pressing practical need. When the United States entered l)!orld 'Var I in 1917, a committee was appointed by the American Psychological Association to consider ways in which psychology might assist in the conduct of the war. This committee, under the direction of !lobert 1.•. 1. Yerkes, recognized the need for the rapid classification of the million and a ha1f recruits with respect to general intellectual level. Such informati~.~~~va:s relevant to many admmistrative decisions, including rejection or discharge from military service, assignment to different types of sel'vicei, or admission to officer-training camps. It was in this setting that the first group intelligence test was developed. In this task, the Ar-m~' psychologists drew on all available test materials, and especially on an unpublished group intelligence test prepared by ~rthur S. Otis, which hc turned over to the Army. A major contribution of Otis's test, which he designed while a student in one of Terman's graduate courses, was the introduction of multiple-choice and other "objective" item types. The tests finally developed by the Army psychologists came to be known as the ~rm""yAlpha and the Army Beta The former was designed

13

fo~ g~n~ral routine te~ting; t~e latter was a nonlanguage scale employed WIth Illiterates and wIth foreign-born recruits who were unable to take a tcst in English. Both test~ w~re suitable for administratio~ to large groups. Shortly af~e~ the temunatlOn of "Vorld War I, the Army tests were released for cmhan use. Not only did the Army Alpha and Army Beta themselves pass through many revisions, the latest of which are even now in use, b.ut they also sVVed as ~dels for most group intelligence tests. The te~ting .movement underwent a tremendous spurt of growth. Soon group mtelhgence tests were being devised for all ages and types of ~ersons, from preschool children to graduate students. Large-sc~le test109 progra~ns: previously impossible, were now being launched with ~est~ul optimIsm. Because group. tests were designed as mass testing lUsh uments, they not only permItted the simultaneous examination of large groups but also simplified the instructions and adminish'ation procedu~es so as to demand a minimum of training on the part of the exammer. Schoolteachers began to give intelligence tests to thcir classes. Coll~ge studen~s were routinely examined prio~ to admission. Extensive studies of specIal adult groups, such as prisoners, were undertaken. And soon the general public became IQ-conscious. "--T~e application of such group intelligence tests far outran their technical Improvement. That the tests were still crude instruments was often f?rgotten in the rush of gathering scores and drawing practical conduslO~Sfrom the ~esults. 'Vhen. ~he tests failed to meet unwarranted expectations" skepticism and hostiht)' toward all testing often resulted. JJ1US. the testi boom of the twenties, based on the indiscriminate use of tests i? ma~ have ~one as much to retai' as to ad\'ance the progress of psvcho- --logical test mg. ~

IS

~lthough intelligence tests were originally designed to sample a wide vanety of ~unctions in order to estimate the individual's general intelIectua~ level, It soon became apparent that such tests were quite limited in theIr .cove~age. Not all important functions were represented. IJ:!. fact, most mtelhgence tests were primarily measures of verbal ability and. to a lesser extent, of the ability to handle numerical and other abstract and symb~~ic re~ations. Gr~dually psychologists eame to recogni~e that the ~erm . Il1telhgence test was a misnomer, since only certain aspects of mtelligence were measured by such tests. To be sure, th~ tests cov~red abilities ,t~t are ot p.rime importance in our culture. B~ It was. realized that more'precise designations, in terms of the type of mformation these tests are able to yield, w\'eadand indiscriminate use durinlJ0 the twenties also revealed another , lIote"iOlthy fact: an individual's erformance on ' test often -showed mar -c variation. This ,yas especially apparent on gl'OUptests, 111whlch the items ar~mmonly segregated into subtests of relath'e1\- homogeneous content. For example, a person might score relatively high on a verbal subtest and low on a numerical subtest, or vice versa, To some extent, such internal variability is also discernible on a test like the Stanford-Binet, in which, for example, all items involving words might prove difficult for a particular individual, whereas itcms employing pictures or geometric diagrams may place him at an advantage, Test users. and especially clinicians, frequently utilized such interc~l11parisons in order to obtain 1110reinsight into the individual's psychological make-up. Thus, not only tllC'IQ or other global score but also scores on subtests wonld lJt' examined in the e\'aluation of the indhidual case, Such a practice is not to be general1~' recommended, ho,~,('ver. ~)eeaus~ intellig('J]ce tests were not designed for the purpose of ,dIHerel,~h,~11 aphtude anal;'sis. Often the subtests heing compared contain t0o,14C\\' items to yield a stable or reliable estimate of a specific ability:;jis'a result, the obtained diffl:'rence betwcen subtest scores might be reversed if the individual were retestE'd on a different day or with another foml of the same test. If such intraindividual comparisons are to be made, tests are needed that are specially designed to reveal differences in performance in various functions. While the practical apl)lication of tests demonstrated the l1~.ed for differential aptitude tests, a parallel development in the stu,d)' of trait organization was gradually providing the means for constructing SUC? tests. Statistical studi('s on the nature of intelligence had been explonng the iflterrelatiol1s among scores obtained by many persons on a ,,,ide variety of different tests, Such investigations were begun by the English ,psychologist Charles Spearman (1904, 1927) during the £lrst decade of the

Functions and OrigillS of PSljchological Testing

15

present century. Subsequent methodological developments, based on the work of such American psychologists as T. L. ReIley (1928) and L. L. !hurs~one (1935, 194i), as well as on that of other American and English ll1veshgators, have come to be known as "factor analvsis." The contributions that the methods of factor ana'lysis have made to test c'Onstruction will be more fully examined and ill~strated in Chapter 1:3. For the present, it will suffice to note that the data gathered by such procedures have indicated the presence of a Dumber of rebtiyely ;ndeJ)endent factors. or traits. Some of these traits were represen'ted, in vary~ng proportions, in the traditional intelligence tests. Verbal comprehenSIOn and numerical reasoning are examples of this tvpe of trait. Others, such as spatial, perceptual, and mechanical aptitude~, were found more often in special aptitude tests than in intelligence tests. One of the chief practical outcomes of factor analysis was the development of multiple aptitude batteries. These batteri('s arc desiuned to provide a measure of the individual's standing in each of a number of traits. In place of a total score or IQ, a separate score is obtained for such traits as "erhal comprehension, numerical aptitude, spatial visualization, arithm~tic re~soning, and perce~tual speed, Such batteries thus provide a SUItable mstrument for makinJiminary demonstra: ~ns, ways of handling queries from subjects. and evel,\, other ~ the testing situation. :Many other, more subtle factors may influence the subject's performance on certain tests. Thus, in giving instructions or, presenting problems orally, consideration must be given to the rate of speaking, tone of voice, inflection, pauses, and faCj~1 e}pression. In a test involving the detection of absurdities, tot eX;lnit>le, the correct an~wer may be given away by smiling or paY~jlg wh~n the crucial word J~.read .. Stand~rdized testing p.rocedure, ~r:,~i[ th~\. ex.aminer's point of \1:w, Will be dJscussed further m a later sect~g~ of-, ",hi e t lose who are customarilv hiil tests may be unduly affected by test anxiety. In a thorough ana::4ontrol1ed investi. gationof this question, French (1962) compar~d Jhf'p,erformancc of high school students on a test given as part of the fe-gular administration of the SAT with performance on a parallel form of the test administered at ,a different time under "relaxed" conditions, The instructions on the latter , occasion specified that the test was given for 'research purposes only and scores would not be sent to any college. The results showed that performance was no poorer during the standard administration than during the relaxed administration. Moreover, the concurrent validitv of the test scores against high school course grades did not differ signifi~antly under the two conditions.

f

Comprehensive surveys of the effects of examiner and situational variables on test seores'lmve been prepared by S. B. Sarason (1954), Masling (l~60), ~foliarty (1961, 1966), Sattler and Theye (1967), Palmer (19,0), and Sattler (1970, 1974). Although some effects have been demonstrated with objective group tests, most of the data have been obtained with either projective techniques or individual intelligence tests. These extraneous factors are more likely to operate with unstructured and ambiguous stimuli, as well as "ith difficult and nO"el tasks, than with clearly defined and well-learned functions. In general, children are more susceptible to examiner and situational influences than are adults; in the examination of preschool children, the role of the examiner is especially cruCiaL. Emotionally disturbed and insecure persons of an\' age are also mClre likely to be affected by such conditions than are well-adjusted persons, There is considerable evidence that test results may vary systematically as a function of the examiner (E. Cohen, 1965; ~'Iasling, 1960). These differences may he related to personal characteristics of the examiner, such as his, age, sex, race, professional or socioeconomic status, training and expenence, personality charaderistics, and appearance. Se\'eral studies of thes~ examiner variables, however, have yielded misleading or illconcluSl\'e results because the experimental designs failed to control or isolate the influence of differcnt examiner or subject characteristics. Hence thp l:'ffeds of two or more variables ma\, be confounded. The examiner's behavior before and during test auministration has also heen s~lown to affect test results, For example, controlled investigations ha\'e YIelded significant differences in intelligence test performance as a res~lt of a "warm" versus a "cold" interpersonal relation between examllJer and examinees, or a rigid and aloof versus a natural manner on the part of the examiner (Exner, 1966; Masling, 1959). Moreover, there may be Significant interactions between examiner and examinee' charact " , h e~lstJCs,III t e sen~e that the same examiner characteristic or testing mannel may have a dIfferent effect on different examinees as a function of the examinee's Own personality characteristics. Similar interactions may occur '~ith task variables, such as the nature of th,e test, the purpose of the testing, and the instructions given to the subjects. Dyer (1973) adds even more variables to this list, calling attention to the possible inHirence of th t t· , d . ," . . c es gIVers an the test takers' diverse perceptions of the funetiglls and goals of testing.' ' St'll ' • '. I. an,other way in which an examin8r may inadvertently affect the ~x~~m~e s responses is through ~is own 'cexpectations, This is simply a P clal mstance of the self-fulfilhng prophecy (Rosenthal, 1966; Rosen-

40

Context of Psycholog.ical Testing

thaI & Rosnow, 1969). -An experiment conducted with the Rorschach will illustrate this effect (Masling, 1965). The examiners were 14 graduate student volunteers, 7 of whom were told, among other things, that experienced examinel's elicit more human than animal responses from the subjects, while the other 7 were told that experienced examiners elicit more animal than human responses. Under these conditions, the two groups of examiners obtained significantly diHerent ratios of animal to human responses from theh subjects. These differences occurred despite the fact that neither examiners nor subjects reported awareness of any influence attempt. ~foreover, tape recordings of all testing sessions revealed no evidence of verbal influence on the part of any examiner. The examiners' expectations apparently operated through subtle postural and facial cues to which the subjects responded. Apa~ from the examiner, other aspects of the testing situation may Significantly affect test performance. Military recmits, for example, are often examined shortly after induction, during a period of intense readjustment to an unfamilim' and stressful situation. In one investigation designed to test the effect of acclimatization to such a situation on test performance, 2,724 recruits were given the Navy Classification Battery during their ninth day at the ~a\'al Training Center (Gordon & Alf, 1960). When their scores were c'Ompared with those obtained by 2,180 recruits tested at the conventional time, during their third day, the 9-day group scored Significantly higher on all subtests of the battery. The examinees' activities immediately preceding the test may also affect their performance, especially when such activities produce emotional disturbance, fatigue, or other- handicapping conditions. In an investigation with third- and fourth-grade schoolchildren, there was some evidence to suggest that IQ on the Draw-a-Man Test was influenced Qrthe children's preceding classroom activity (McCarthy, 1944). On one occasion, the class had been engaged in writing a composition on "The" Best Thing That Ever Happened to Me"; on the second occasion, they had again been writing, but this time on "The Wo~sLThing That Ever'Happened to Me." The IQ's on the second test, fOllowing what may have been an emotionally depressing experience, averaged 4 or 5 points lo\ver than on the first test. These findings were corroborated in a later investigation specifically designed to determine the effect of immediately preeeding experience on the Draw-a-Man Test (Reichenberg-Hackett, 1953). In this study, children who had had a gratifying experience involving the successful solution of an interesting puzzle, followed by a reward of toys and candy, snowed more improvement in their test scores than those who had undergone neutral or less gratifying experiences. Similar results were obtained by W. E. Davis (1969a, 1969b) with college students. Perfonnance on an arithmetic reasoning test was significantly poorer when preceded by a failure experience on a verbal comprehension test than it

Natufa aile! Use

of

Psychological Tests

41

was in a control group given no preceding test and in one that had taken a standard verbal comprehension test under ordinary conditions. Several studies have been concerned with the effects of feedback regarding test scores on the individual's subsequent test performance. In a particularly well-designed investigation with seventh-grade students, Bridgeman (1974) found that "success" feedback was followed by significantly higher performance on a similar test than was "failure" feedhack in subjects who had actually performed equally well to begin with. This type of motivational feedback may operate largely through the goals the subjects set for themselves in subsequent performance and may thus represent another example of the self-fulfilling prophecy. Such general motivational feedback, however, s1)ould not be confused with corrective feedback, 'whereby the individual is informed about the specific items he missed and given remedial instruction; under these conditions, feedback is much more likely to improve the performance of initially low-scoring persons. The examples cited in this section illustrate the wide diversity of testrelated factors that may affect test scores. In the majority of well-administered testing programs, the influence of these factors is negligible for practical purposes. Nevertheless~ the skilled examiner is constantly on guard to detect the possible operation of such factors and to mipimize their influence. When circumstances do not permit the control of these conditions, the conclusions drawn from test performance should be qualified.

In evaluating the eHect of coaching or practice on test scores, a fundamental question is whether the improvement is limited to the specific items included in the test or whether it extends to the broader area of ~ehavior that the test i~gned to p;edict. The answer to this ques~ represel1ts the difference between coacmng and education. Obviously any educational experience the indiVidual undergoes, either formal or informal, in or out of school, should be reflected in his performance on tests sampling the relevant aspects of behavior. Such broad influene.es will in no way invalidate the test, since the test score presents an aar:a,tate piCture of the individual's standing in the abilities under conside~n. The difference is, of course, one of degree. Influences cannot..:..be~dassified as either. narrow or broad, but obviously vary widely in scop~~f;om those ~ffecting only a single a~lllinis~tj~n of a.,single test, throu~hJib.~se. affect~ng'p~rformance on all Items ;()fi,ca /:crtUln,type, to those mtfUencmg the mdl vidual's performance in the large .Irtai9rity of his activities. From the standpOint of effective testing, however, a workable distinction can be

COlltext of P~yc1lOlogic(/l

Testing

e. Thus, it can be stated that a test score is inmlidated only when a ':'cular experience raises it withont appreciably affecting the criterion ~Lviorthat: the test is deSigned to predict.

:";{CHIKC. 'the effects of coaching on test scores have been widely ingated. Many of these studies were conducted by British psycholo,with special reference to the effects of practice and coaching on the brinerly used in assigning ll-year-old children to different types of 'Ilrv;,schools (Yates et aI., 195:3-1954). As might be expected, the ~~ovement depends on the ability and earlier educational; 'ences of'the examinees, the nature of the tests, and the amount and 'of coaching provided. Individuals with deficient educational backunds are more likely to benefit from special coaching than are those 'ihave had superior educational opportunities and are already pre, to do well on the tests. It is obvious, too, that the closer the re,blance between test content and coaching material, the greater will the improvement in test scores. On the other hand, the more closely truction is restricted to specific test content, the less likely is improve:nt to extend to criterion performance. "n America, the College Entrance Examination Board has been conhed about the spread of ill-advised commercial coaching courses for lege applicants. To clarify the issues, the College Board conducted veral well-controlled experiments to determine the effects of coaching 'its Scholastic Aptitude Test and surveyed the results of similar studies other, independent investigators (Angoff, 19711>;Conege Entrance 'amination Board, 1968). These studies covered a variety of coaching ethods and included students in both public and private high schools; e investigation was conducted with black students in 15 urban and '"ral high schools in Tennessee. The conclusion from all"these studies is ':at intensive drill on items similar to those on the SAT is unlikelY to 'oduce appreciably greater gains than occur students are rete~ted 'th the SAT after a year of regular high schot;il instruction. On the basis of such research, the Trustees of the College Board issued .formal statement about coaching, in which the fonowing points were ade, among others (College Entrance Examination Board, 1968, p.8-9):

and is taught. Rather, this particular Scholastic Aptitude Test is a measure of abilities that seem to grow slowly and stubb(lrnl~'. profoundly influcllced by conditions at home and at school over thc years, but not responding to hasty attempts to relive a young lifetime. \in'S

It should also be noted that in its test construction procedures, the Col. lege Board im'estigates the susceptibility of new item types to coaching (:\ngoH, 1971b; Pike & Evans, 1972). Item types on which perfo.rma1lce can be appreciably raised by short-term drill or instruction of a narrowly limited nature are not included in the operational forms of the tests..

ot

wrJ/i

e results of the coaching studies which ha,'e thus far been completed inte that average increases of less than 10 points on a 600 point scale can ,expected. It is not reasonable to believe that admissions decisions can be ected by such small changes in scores. This is especially true since the tests merely supplementary to the school record and other evidence taken into . unt b'): admissions officers. . . , As the College Board uses the term, apitude is not something flxed and impervious to influence by the way the child

PRACTICE.The effects of sheer repetition, or practice, on test performance are similar to the effects of coaching, but usuaIl~' less pronounced. It should be noted that practice, as well as coaching, may alter the nature of the test, since the subjects may emplo~' different work methods in solving the same problems. Moreover, certain types of items may be much easier when encountered a second time. An example is 'provided by problems requiring insightful solutions which, once attained, can be applied directly in solving the same or similar problems in a retest. Scores on such tests, whether derived from a repetition of the identical test or from a parallel form, should therefore be carefully scrutinized. A number of studies have been concerned ~,'ith the effects of the identical repetition of intelligence tests over periods ranging from a few days to se,'eral years (see Quereshi, ] 968). Both adults and children, and both normal and mentally retarded persons have been employed. The studies have covered individual as well as group tests. All agree in showing significant mean gains on retests. Nor is improvement necessarily limited to the initial repetitions. \Vhether gains persist or level off in successive administrations seems to depend on the difficulty of the test and the abilit~· level of the subjects. The implications of sucll findings are il- \ lustrated by the results obtained in annual retests of .3,500 schoolchildren with a Yariety of intelligence tests (Dearborn & Rothnev, 1941). When the same test was readministered in successive years, th~ median IQ of the group rose from 102 to 113, but it dropped to 104 when another test w~s substituted. Becaus~ of the retest gains, the meaning of an IQ obtamed on an initial and later trial proved to be quite different. For example, .a~ ~Q of 100 fell approximately at the average o£'lhe distribution on the Im~lal trial, -but in the lowest quarter On a retest~S\ldl iQ's, though numencally identical and derived from the same te~ 1l;!ightthus signify normal ability in the one instance and inferior ability#},(,the other. G~ins in score are also found on retesting with pili:dIel -forms lishers specify reqUlr~ments that must be met by purchasers, Usually ~pdividuals with a mast~r s degree in psychology or its equivalent qu~l.i~~' -SO'rtle publishers claSSIfytheir tests into levels with reference to user qt;al~fi~~ions, ranging from educational achievement and vocational proficiency tests, through

'Context of Psychological Testing , , 'entories to such clinical instrultelligence tests and mterest In\ t 'ersonalit tests, Distincs individual intelligence tests al ldmOhsPers alld a~thorized instid' 'idua 1 [lUre as e' alsohma db'e ofetween In ,1\ t . Graduate students who may appropnate tes s, , h Pure asers " must have t e . , f I ignment or for research , . articular test or ~ c ass a~s h "ehology instructor, who as" order countersigned by t elf ps~ , 'b'l' f' th oller use of the test. , sponsl 1 Ity 01 e'pr, f h a dual objective: secunty to restrict the distn~uboll o· ~ests ;~: Ethical Standards state: ' .1 d prevenhon of mIsuse, 1 , t atena san ' , I' 't d to persons \\1,'th professional mteres s to such deVices IS ImI e , , 1 13)' "Test scores like test ' d h' "( Pnnclp e, , ~ll safeguar t elr use who arc ualifled to interpret and q als, are rele::sed ~nl~ to perso~:sshould be noted that although test m properly (Prmciple 14)" I t these obJ'cctives, the conk ' , efforts to Imp emen 'b'l' utors ma 'e SllleCIe , '1 limited, The major responsl 1 Ity Yare able to exert IS neeessan y h ' d' 'dual uscr or institution f 'd in t e 111 IVi proper use 0 tests resl es h t MA degree in psychology ~ed,It is evident, ~or exampleA~;p a~i 'lorna-do not necessarily en a PhD, state hc~nse, a~ld P articular test or that his a ' hat the indi\'idualls quah~ed ~o u;e ti ;: of the results obtained is relevant to the proper mtel pre a 0 at test. . . l' 'bilihr concern.s the Il1arketing of psvcho, er professIOna lcsponsl '} h Id - t be released preI d blishers Tests s Oll no tests by aut lOrs an pu , I' be made regardincr the ' ' 1 N" h Id anv c aUllS b V for ns (see, e.g., American Psychological Association, 1969; Cleary, Humphreys, Kendrick, & \Ves-

Ie tlthou~h ~omen repre)'lnt a statistical majorltyjn the nati~~al population. ga I.y,~c~upalJonallY'in in otlu~r ways.they have s~ed Jllany of the problems of mmoTlhes. Hence w the term "minority" is use(i "fu tnis section it will be understood to includj) men. ' I

I

I II

1

II

Social and Etllicallmplications . 'onlcxt

of Ps!}clIOlogica1Testing

Tl::

'5' Deutsch Fishman, Kogan, North, & Whiteman, 1964; '1Jl~use of t~sts 1972). A brief but cogent paper b~ F~augh

Iso helps to cle~r away some preval~nt S~ll;C~So~;:; ~~I~~~ural 'of the concern centers on the lowenng 0 es sc . d . t rns that ma)' have affected the devel~p;ne~t lofc~;:~~e:;;ti: eoE otivation, attitudes, and other psyc ~ O~IC: for the problem ou members. Some of the propose so u Ions . al mi~nlrstandings about tIle nature anddfllnfction of ps~chdol'Vll?j~~ls . . I b kgroun s 0 groups or 10 iflerencesin the expenentia ac hI' 1 test ~itably manifested in test performanlce. Ev:rytPsbychaoVl~o~C~ts in. 1 I f as Cll ture alIec s e , res a beh:wlOl' samp e. nso ar If 1 ut aU cultural will and should be detected by tests. .we ~ e. 0 as a measure I1tialsHom a test, we may th.ereb Ylower Its ;ah~?t case the test behavior domain it was deslgnc d to assess. n 'fail to provide the kind of information needed to correct the very 'ionsthat impaired performance. . 1 citron the ause the testing of minorities repr~sent\a sP:~~~l ~~; :heoretical .er problem of cross-cultural te.stmg, t e U full) in Cha ter 12. naleand testh~g procedures ar: ~1~~~:~e~i:'?7s giv:n in Ch~pter 7, chnicalanalysIs of the concep 0 h t h ter our interest is , .h l'd't In t e presen c ap , llnnectlOl1Wit test va I I y. ., ., f inDrity groUp wily in the basic issues and SOCialImplications 0 m

·ng.

~~d

FACTORS. In testing culturally di"h·elt·seffPerst °bno~h i:e~~ . . b cultural factors t a a ec rtant to differentiate etween . t . t d to the test It is I' d th hDse in uence is res nc e . ·terionbe laVlor an ose w d ~ Ex~mples of such atter, tSst-related actors that. re l\~e va 1 .; ion to erEorm to~sinclude previous experience m ~akmg tests, mo~;t. variable; ,~_ r ~veJlon tests, rappDrt with the exammer, an~ an y 0 tet ~_-i 'n of the Civil Rl?hts Act o. a~ ;nfottement is vested in the sponsibility for Implementation an ., (EEOC) When charges , 0 rtunity C ommlSSlon . qual Employment ppo. h plal'nt and if it finds the charges , " h EEOC' shgates t e com , -arefiled, t e lllve t th 'tuation through conferences and . '6 d'" first to correc e Sl to be lush e , u1.es d f '1 EEOC may proceed to r If these proce ures al, voluntary com~ lance. d d . t orders and finally bring action in hold hearings, ISsue cease an eSlS , AL REGULATIONS.

, e.

al developmentssince midcentury, including rt decisions, can be found in Fincher legislativeactions, executive orders, an cou .

1

'A brief summary of ~he major e~ d

(1973).

of Testing

63

the federal courts. In states having an approved FEPC, the Commission will defer to the local agency and will give its Bndings and conclusions "substantial weight." The Office of Federal Contract Compliance (OFCC) has the authority to monitor the use of tests for employment purposes by government contractors. Colleges and universities are among the institutions concerned with OFCC regulations, because of their many research and training grants from such federal sources as the Department of Health, Education, and Welfare. Both EEOC and OFCC have drawn up guidelines regarding employee testing and other selection procedures, which are virtulillly identical in substance. A copy of the EEOC Guidelines on Employee Selection Procedures is reproduced in Appendix B, together with a 1974 amendment of the OFCC guidelines clarifying acceptable procedures for reporting test validity,3 Some major provisions in the EEOC Guidelines should be noted, The Equal Employment Opportunity Act prohibits discrimination by employers, trade unions, or employment agencies on the basis of race, color, religion, sex, or national origin, It is recognized that properly conducted testing programs not only are acceptable under this Act but can also contribute to the "implementation of nondiscriminatory personnel policies." Moreover, the same regulations specified for tests are also applied to all other formal and informal selection procedures, such as educational or work-history requirements, interviews, and application forms (Sections 2 and 13), \Vhen the use of a test (or other selection procedure) results in a significantly higher rejection rate for minority candidates than for nonminority candidates, its utility must be justified by evidence of validity for the job in question. In defining acceptable procedures for establishing validity, the Guidelines make explicit reference to the Standards for Educational and Psychological Tests (1974) prepared by the American PsycholOgical Association. A major portion of the Guidelines covers minimum requirements for acceptable validation (Sections 5 to 9). The reader may find it profitable to review these requirements after reading the more detailed technical discussion of validity in Chapters 6 and 7 of this book. It will be seen that the requirements are generally in line with good psychometric practice. In the final section, dealing with affirmative action, the Guidelines point out that even when selection procedures have been satisfactorily 3 In 1973, in the interest of simplIficationand improved coordination,the preparation of a set of uniform guidelines was undertaken by the Equal Employment Opportunity Coordinating Council, consisting of representatives of E ,the U.S, Department of Justice, the u.s. Civil Service Commission,the U.S'c,rtlJlent of Labor, and the U.S. Commissionon Civil Rights. No'uniform versioD,o~t Id not • . ce 'I·nfouI,th grade arithmetic. It certam. Iy COU lOrpenorman - • . d 'h fc I. med that he has the prerequi~ites for seventh-gra e ant me I ~ adc norms tend to be incorrectly regarded as performan~l A sixth-grade teacher, for example: may assume tha.t all class should fall a! or close to tl~e sixth-grade ,n?rm In ac rade ests This misconception is certamly not surpnsmg when g h iare ~sed Yet individual differences within any onc grade ar~ suc ·.,:therange' of achJevement test scores will inevitably exten over

h~!:e~

;df.

pal grades, 1 t developmental norms derives Another approac 1 0 1 b t' f behavior , hI' E Ipirica 0 serva Ion 0 research in chIld psyc 0 og~, . n . 1 d t the description of be'pment in infants and young chlldl;n e. 0 1 omotion sensory typical of successive ages in ,SUC uncti~ns as OCt forma~ion. An

DINAL SCALES.

?

.' inati0t, .lingui~~c dc~~~~~;~~:t:~n~f a~ese~lo:~e£ his associates at p eAxame1913s et ~l. 1940; Gesell & Amatruda, 1947; H~lver( mes" , h d I h th apprOXImate 1933) The Gesell Developmental Sc e U es s 0''0 e h ff r lopm~ntallevel in months that the child has attained in eadc 0 °aul d ptive lan.."INN

"''''

"''''0

"'~Lll NM.,.

N"'"(predictive validity) . . . Because ~he criterion for concurrent validity is always available at the hme of testmg, we might ask what function is served bv the test in such situa~ions. B~sicalIy, such tests provide a simpler, quicker, or less ex~ensive subs.htute for the criterion data. For example, if the criterion conSIStsof continuous observation of a patient during a two-week hospital- , ization period, a test that could sort out normals from neurotic and ' ?oubtful cases would appreciably reduce the number of persons requirmg such extensive observation.

'CURREI'.:T AND PREDICTIVE

• ~RITERION CO~TAMINATION. An essential precaution in finding the vahdlty of a test IS to make certain that the test scores do not themselves influence any individ~ars c~terion. status. For example, if a college illranalysis is to simplify the description of behavior by reducing the er of categories from an initial multi licit of test vari bles to a few 1

ac ;Aft~rthe factors have been idcntified, they can be utilized in describing e factorial composition of a test. Each test can thus be cl1afacterized in rmsof the l1)a)or factors determining its scores, together with the weight r loading of each factor and the correlation of the test with each facto~. uch a correlation is known as the factorial validity of the test. Thus, lf he verbal comprehension factor has a weight of .66 in a vocabulary test, he factorial validity of this vocabulary test as a measure of the trait of erbal comprehension is .66. It should be noted that factorial validity is entially the correlation of the test with whatever is common to a group of tests or other indices of behavior. The set of variables analyzed can, ofcourse, include both test and nontest data. Ratings and other criterion 'measurescan thus be utilized, along with other tests, to explore the factorial validity of a particular test and to define the common traits it measures. INTERNAL CONSISTENCY. In the published descriptions of certain tests, especially in the area of personality, the statement is made that the test has been validated by the method of internal consistency. The essential characteristic of this method is that the criterion is none other than the -total score on the test itself. Sometimes an adaptation of the contrasted . grOUpmethod is used, extr'"eme groups being selected on the basis of the total test score. The performance of the upper criterion group on each test item is then compared with that of the lower criterion group. Items that fail to show a significantly greater proportion of "passes" in the upper than in the lower criterion group are considered invalid, and are either ~liminated or revised. Correlation~l pr.qcedures may also be employed for this purpose. For example, the biserial'correlation between ."pass-f~il" .on each item and total test score can be computed. Only those Items )'leldmg significant item-testcorr~fliJi.Pns would be retained. A test whose items were selected by this meth,qd can be said to show internal consistency, since each item differentiates in the same direction as the entire test. Another application of the criterion of internal consistency involves the

i

EFFECT OF EXPERIYENTAL VARIABLES ON TEST SCORES' A further source of data forconstmct validation is provided by ex-periments on the effect of selecte(;I~ariables on test scores. In checking the validitv of a ~riterionreferellce'O test for use in an individualized instruction~l program, for example, one approach is through a comparison QE pretest and posttest scor~s.. The rationale of such a test calls for low scores on the pretest, admlms~ered b~fore ~he relevant instruction, and high scores on the posttest. ThiS relationshIp can also be checked for individual items in the te~t (Po.pharo, 1971). Ideally, the largest proportion of examinees should fall an Item ?n the pretest and pass it on the posttest. Items that are commonly falled on both tests are too difficult, and those passed on both tests ~oo easy, for t~e purposes of such a test. If a sizeable proportion of exa~mees pass an ltem on thc pretest and fail it on the posttest, there is obvlOusly something wrong with the item, or the instruction, or both. A. test designed to measure anxiety-proneness can be administered to sub!ects who are subsequently put through a situation designed to arouse amQe.~, such as .t~~ng an examination under distracting and stressful conditions. The lDltlal anxiety test scores can then be correlated with phySiolog!cal. and other indices of an~iety expression du~pg and after the exammatIon. A different hypothesis regarding an anxietY· test could ?e evalua~ed by admini~tering the test before and after an anxiety-arousmg expen:~ce an~ seemg whether test scores rise Significantly on the retest. PosItive flndmgs from such an experiment would indicate that the test scores. reBect current anxiety level. In a similar w,lI.y;' exper4;h.lents can be. designed to test any other hypothesis regarding th.~;;tfait ~~SUred by a gIVen test.' .'

TABLE

In a thoughtful analysis nstruet validation, D. T. Campbell (1960) points out that in order emollstrate construct validity we must show not only that a test cores highly with other variables with whi~h .it should ~heoret.ically elate, but also that it does not correlate sIgmficantly wIth van abIes which it should differ. In an earlier article, Campbell and Fiske ) described the former process as convergent validation and the er as discriminant validation. Correlation of a mechanical aptitude with subsequent grades in a shop course would be an example of vergent validation. For the same test, discriminant validity would be rated by a low and insignificant correlation with scores. on a .reading prehension test, since reading ability is an irrelevant varIable m a test gnedto measure mechanical aptitude. ., . t will be recalled that the requirement of low correlatlOn WIth trrelet variables was discussed in connection with supplementary and pretionary procedures followed in content validation. Discrin;inant va~ionis also especially relevant to the validation of personality tests, In ich irrelevant variables may affect scores in a variety of ways. ampbell and Fiske (1959) proposed a systematic experimental deSign the dual approach of convergent and discriminant validation, which ey called the multitrait-multimet1lOd J7latrix. Essentially, this procedure quiresthe assessment of two or more traits by tw.o Qr ~ore metho~s. A pathetical example provided by Campbell and FIske WIll serve to IUUSate the procedure. Table 12 shows all possible correlations among the ores obtained when three traits are each measured by three methods. , e three traits could represent three personality characteristics, such as A) dominance, (B) sociability, and (C) achievement motivation. The hreemethods could be (1) a self-report inventory, (2) a projective tech'iquc,and (3) associates' ratings. Thus, Al would indicate dom~na~ce oreson the self-report inventory, A2 dominance scores on the projective est,C3 associates' ratings on achievement motivation, and so forth. The hypothetical correlations giv~n in Ta~le 12 include reli.ability cofficients (in parentheses, along principal dIagonal) and validity coefcients (in boldface, along three shorter diagonals). In these validity coefficients,the scores obtained fc",~~~p same trait by different methods arecorrelated; each measure "is.thu~ 'being checked against other, independent measures of the same'::'trait, ~s'.in the familiar validati~n procedure. The table also includes correlations between different traIts measured by the same riJ":.thod'(in solid triilngles) ~nd corrclati.ons between different traitsllleasured by different methods (Ill broken trIangles). For satisfactory construct validity, the validity coefficients should obviously be higher than the correlations between different traits measured by different methods; they should also be higher than the correlations beNVERGENT AND DISCRIMINANT

VALIDATiON.

12

A Hypothetical

Multitrait-M:ultimethod

Matrix

(From Campbell & Fiske, 1959, p. 82.)

Traits

A, Method 1

B,

C,

A, Method 2

B. C. ,...56::-:- ...~22---:11:

67-'---42-------:

',. .33 I: :.23'".58"',)2: :.43 '".~6',,:.34: II..... ',',

I

.•.

"...

I

1'"

I : ......•...... , ....•....•..

....,~

t

••••••

I

l ~~1 :..~~:~~~45L~~ .:~~~::::~58~ .58 •...• ·::;"'-~"':'_~~~'~:;~'::>'.-~;~I;;,';"_:~~~~

(.85)

..•_.:;.::~:=.t;,..~~~':Q.~IM&~)

Note: Le~tersA. B, C refer to traits, subSCripts1,2,3 to methods. Validity coefficients (rnon~tralt-heteromethod) are the three diagonal sets of boldface numbers; reliability c~efficlents (~ono~ralt-rnonomethod) are the numbers in parentheses along principal diagonal. Sohd tnangles enclose heterotrait-monomethod correlations; broken triangles enclose heterotrait-hcteromethod correlations. 'l

tween different traits measured by the same method. For example, the COITf:lationbetween dominance scores from a self-report inventory and dOITt~ijancescores from a projective test should be higher than the correlatIon between dominance and sociability scores from a self-report in. ventor~. If ~he l~tter correlation, representing common method variance, :-rere hIgh, It mIght inllicate, for example, that a person's scores on this Inventory are unduly affected by some irrelevant common factor such as ability to understand the questions or desire to make oneself appear in a favorable light on all traits. Fiske (1973) has added still another set of correlations that should be checke~, esp~cially in the construct validation of personality tests. These ~rrelab~ns Involve the same trait measured by the"same method, but With a dlffer~nt test. For examplc, two il)vestigators may each pliepare a self-report Inventory designed to assesseIl,durance. Yet the end~rance scores obtained with the two inventories may show quite diffe~~nt. patterns of correlations with measures of other personality traits. Under these

Principles

Validity: Basic Conc('pts

of Psychological Testing

.ditions,it cannot be concluded tllat both inventories measure the same ·sonalityconstruct of endurance. ., . t might be noted that within the framework of the mnlhtrmt-mulhhod matrix, reliability represents agreement between two measures of same trait obtained through maximally similar methods, such as alle! forms of the same test; validity represents agreement between measures of the same trait obtained by maximally different methods, chas test scores and supervisor's ratings. Since similarity and difference methods arem~tters of degree, theoretically reliability and validity can regarded as falling along a single continuum: O~~inarily, ho\~'e~er, the hniques actually employed to measure rehabllIty and validIty corond to easily identifiable regions of this continuum.

We have considered several ways of asking, "How valid is this test?" Topoint up the distinctive features of the different types of validity, let us apply each in turn to a test consisting of 50 assorted arithmetic problems.Four ways in which this test might be employed, together with the type of validation procedure appropriate to each, a:e illustra:ed ~n Table 13. This example highlights the fact that the chOIce of valIdahon pro; cedure depends on the use to be made of the test scores. The same test, when employed for different purposes, should be validated in different ways.If an achievement test is useet to predict subsequent performance

TABLE

13

Validationof a Single Arithmetic Test for Different Purposes Illustrative Question . Achievement test in elementary school aritlunetic Aptitude test to predict performance in high school mathematics Technique for diagnosing learning disabilities Measure of logical reasoning

Type of Validity

How much has Dick learned in the past? How well will Jim learn in the future?

Criterion-related: predictive

Does Bill's performance show specific disabilities? How can we describe Henry's psychological functioning?

Criterion-related: concurrent

159

at a higher educational level, as when selectinO' school students for b hiO'h t:< college admission, it needs to be evaluated against the criterion of subsequent college performance rather than in terms of its content validity. The examples given in Table 13. focus on the differences among the various types of validation procedures. Further consideration of these procedures, however, shows that content, criterion-related, and construct validity do not correspond to distinct or lOgically coordinflte categories. On the contrary, construct validity is a comprehensive concept, which includes the other types. All the specific techniques for establishing content and criterion-related validity, discussed in earlier sections of this chapter, could have heen listed again under construct validity. Comparing the test performance of contrasted groups, such as neurotics and normals, is one way of checking the construct validity of a test designed to measure emotional adjustment, anxiety, or other postulated traits. Comparing the test scores of institutionalized mental retardates with those of normal schoolchildren is one way to investigate the construct validity of an intelligence test. The correlations of a mechanical aptitude test with performance in shop courses and in a wide variety of jobs contribute to our understanding of the construct measured by the test. Validity against various practical criteria is commonly reported in test manuals to aid the potential user in understandin~ what a test measures. Although he may not be directly concerned with the prediction of any of the specific criteria employed, by examining such criteria the test user is able to build up a concept of the behavior domain sampled by the test. Content validity likewise enters into both the construction and the subsequent evaluation of all tests. In assembling items for any new test, the test constructor is guided by hypotheses regarding the relations between the type of content he chooses and the behavior he wishes to measure. All the techniques of criterion-related validation, as well as the other techniques discussed under construct validation, represent ways of testing such hypotheses. As for the test user, he too relies in part on content validity in evaluating any test. For example, he may check the vocabulary in an emotional adjustment inventory to determine whether some of the words are too difficult for the persons he plans to test; he may conclude that. the scores on a particular test depend too much on speed for his purposes; or he may notice that an intelligence test developed twenty years ago contains many obsolescent items unsuitable for use today. All these observations about content are relevant to the construct validity of a test. In fact, there is no information provided by any validation procedure that is not relevant to construct validity. The term construct validity was officially introduced into the psychome~rist's lexicon in 1954 in the Technical RecommenN4a{ions for Psyc11010glcal Tests and Diagnostic Techniques, which constituted the first edition of the current APA test Standards (1974). Although the validation

160

Pritlci,Jles of PSlJchological Testing

procedures subsumed under construct validity were not new at the time, the discussions of construct validation that followed served to make the . implications of these procedures more explicit and to provide a systematic ,; rationale for their use. Construct validation has focused attention on the role of psychological theory in test construction and on the need to formulate hypotheses that can be proved or disproved in the validation process. It is particularly appropriate in the evaluation of tests for use in research. In practical contexts, construct validation is suitable for investigating ; the validity of the criterion measures used in traditional criterion-related " test validation (see, e.g., James, 1973). Through an analysis of the correlations of different criterion measures with each other and with other , relevant variables, and through factorial analyses of such data, one can learn more about the meaning of a particular criterion. In some instances, the r~sults of such a study may lead to modification or replacement of the criterion chosen to validatc a test. Under any circumstances, the results will enrich the interpretation of the test validation study. Another practical application of construct validation is in the evaluation of tests in situations that do not permit acceptable criterion-related validation studies. as in the local validation of some personnel tests for industrial use. The difficulties encountered in these situations were discussed earlier in thi.s chapter, in connection with synthetic validity. Constr~ct validation offers another alternative approach that could be followed in evaluating the appropriateness of published-tests for a particular job. Like synthetic validation, this approach requires a systematic job analysis, followed by a description of worker qualifications expressed in ;.:''terms of relevant behavioral constructs. If, now, the test has bcen subjected to sufficient research prior to publication, the data cited in the manual should permit a specification of the principal constructs measured by the test. This information could be used directly in assessing the relevance of the test to the required job functions, if the correspondence of constructs is clear enough; or it could serve as a basis for computing a J-coefficient or some other quantitative index of synthetic validity. . Construct validation has also stimulated the search for novel ways of gathering validity data. Although the principal techniques employed in investigating construct validity have long been familiar, the field of operation has been' expanded to admit a \\rider variety of procedures. This very multiplicity of data-gathering techniques, however, presents certain hazards. It is possible for a test constructor to try a large number of different validation procedures, a few of which will yield positive results by chance. If these confirmatory results were then to be reported without mention of all the validity probes that yielded negative results, a very misleading impression about the validity of a test could be created. Another possible danger in the application of construct validation is that I

I

Validity: Basic Concepts 161 it may open the way for s b" . validity. Since . ~ J~chve, unvenfled assertions about test cept, it has bE':~~~~~~ v;~~~~;s ~uc~ asbroad and loosely dcflned canconstructors Seem to ~r . . rs 00. ome textbook writers and test psychological trait na~lescelVe It as content validity expressed in terms of subjective accounts of ~h~:~~e, t~e~ present as construct validity purely A further source of ossibl ey e ~ve (o~ hope) the test measures. construct validation "is ; I e Co~fuslOn anses from a statement that d a measure of some at .~vo ve w e~ever ~ test is to be interpreted as oned'." (Cronbach & ~:e~tel~r quahty whIch is not 'operationally depublished analysis of the co' 282). Appearing in the first detailed often incorrectl acce ted :c~p ~nstruct "alidity, this statement was the absence of ~ata ~hat t~ Justifrng a claim for construct validity in such an interpretati;n is i1lus:a~~t ors of .the sta~e~ent did. not intend article, that "unless the n t k d b ytheIr own inSIstence, III the same e war ma k es contact with b . construct validation cannot b I' d" 0 servations . . . .. . e c alme (p. 291) In th . h t ey cnhclze tests for wh' h" fi . e same connectIon, been oHered as if it wcre l~al'; t.ne~pun network of rationalizations has l construct, trait or behavio d a I~n (p, 291). Actually, the theoretical ' r omam measured bv rti I b e a d equateI), defined only' th I' h f - a pa cu ar test can validating that test Such I~ Iie. ~g t 0 data gathered in the process of abIes with which th~ test c~ ~ ~lhO~ would take into account the varifound to affect its Scores an~et~ ed SIgnificantly, as well as the conditions scores. These procedures are e ; ~~~ps that diff~r significantly in such butions made bv the co t n fIre :- m aCcord w1th the positive contrl. ncep 0 construct valid'ty I ' the empirical investigation of the r I' h' 1. t IS only through external data that we can d' ehahons IpS of test SCores to other ISCOverw at a test measures.

5;,?f'

HArTER

7

alidity: Measuremel~t and lrlterpretation 6 was concerned with different concepts of validity and their appropriateness for various testing. f~nctions; t~is. chapter deals with quantitative expressions of vahdlty and theIr mterpretation. The test user is concerned with validity at either or both of two stages. First, when considering the suitability of a test for his purposes, he examin~ailable validit)'data reported in the test manual or ot~er p~ed so.Jltces..Through such in~ormation, he arrives at a tentative concept of what psychological fu~ctlOns the test actually measures, and he judges the relevance of such function~ to his p.rop~sed use of t~e test. In effect, when a test user relies on published validation data, he IS dea.ling with construct validity, regardless of the specific pro?ed~res used m gathering the data. As we have seen in Chapter 6, the cntena employed in published studies cannot be assumed to be iden?cal. with th~se the test user wants· to predict. Jobs bearing the same title m two dIfferent companies are rarely identical. Two courses in freshman English taught in different colleges may be quite dissim~1;l.r· i Because of the specificity of each criterion, te~t users are .us~ally advised to check the validity of anv chosen, 'test agamst local cnterla whenever possible. Although publishe'd dat~ay str~ngl~ sugg~st that a given test should have high validity in a particular sltuatio~, dlTee: corrobo~ation is always desirable. The dete:t'inination of validJ!Y agamst specific local criteria represents the second stage in the test ~r's evaluation of valKTfty.The teChnIques ttr'1le dIscussed 1~ this chapter are esp~cially relevant to the analysis of validity data obtamed by ~e test u.ser hlms~1f. Most of them are also useful, however, in understanding and mterpretmg the validity data reported in test manuals. HAPTER

C .

J6z

MEASUREMEXT OF RELATIONSHIP. A validity coefficient is a correlation between test score and criterion measure. Because it provides a single numerical index of test validity, it is commonly used in test manuals to report the validity of a test against each criterion for which data are available. The data used in computing any validity coefficient can also be expressed in the form of an expectancy table or expectancy chart, illustrated in Chapter 4. In fact, such tables and charts provide a convenient way to show what a validity coefficient means for the person tested. It will be recalled that expectancy charts give the probability that an individual who obtains a certain score on the test will attain a specified level of criterion performance. For example, with Table 6 (Ch. 4, p. 101), if we know a student's score on the DAT Verbal Reasoning test, "",e can look up the chances that he will earn a particular grade in a hIgh school course. The same data yield a validity coefficient of .66 When both test and criterion variables are continuous, as in this example, the familiar Pearson Product-Moment Correlation Coefficient is applicable. Other types of correlation coefficients can be computed when the data are expressed in different forms, as when a two-fold pass-fail criterion is employed (e.g., Fig. 7, Ch. 4). The specific procedures for computing these different kinds of correlations can be found in any standard statistics text.

COI\"DITIONS AFFECTING VALIDITY COEFFlCIEXTS. As in the case of reliability, it is essential to specify the nature of the group on which a validity coefficient is found. The same test may measure different functions when given to individuals who differ in age, sex, educational level, occupation, or any other relevant characteristic. Persons with different experiential backgrounds, for example, may utilize different work methods to solve the same test problem. Consequently, a test could have high validity in predicting a particular criterion in one population, and little or no validity in another. Or it might be a valid measure of different functions in the two populations. Thus, unless the validation s~ple is repri'!seiififiVe of the population on which the test is to be used, validity should be redetermined on a more appropriate sample. The question of sample heterogeneity is relevant to the measurement of validity, as it is to the measurement of reliability,'.,since both characteristics ale commonly reported in terms of correlation eoefficiElnts. It will be recalled that, other things being equal, the wider the range of scores, the higher will be the correlation. This fact should be kept in

Principles

of

Psychological

mind when interpreting

Testing

the validity coefficients given in test manuals. ,;. Il.special difHcttlt}, encountered in many validation samples arises from preselection. For example, a new test that is being validated for job selee.tionmay be admini$tered to a group of newly hired employees on whom ;criterJIonmeaSures of job performance will eventua11y be available. It is ~likely;however, that such employees represent a superior selection of all "!hosewho applied for the job. Hence, the range of such a group in both .'tests¢ores and criterion measures will be curtailed at the lower end of the :·bdistribution.the effe~t of such preselection will therefore be to lower the 'validity coefficient. In the subsequent use of the test, when it is admindster¢d to all applicants for selection purposes, the validity can be expected to be somewhllt higher. " Validity coefficients may also change over time because of changing .'selection standards. An example is provided by a comparison of validity ,coefficients compll.ted over a 3D-year interval with Yale students (Burn"ham, 1965). Correlations were found between a predictive index based , on College Entrance Examination Board tests and high school records, f onthe one hand, and average freshman grades, on the other. This correlation dropped from .11 to .52 over the 30 years. An examination of the r' bivariate distributions dearly reveals the reason for this drop. Because of ~higher admissibn standards, the later class was more homogeneous than .:the earlier class in both predictor and criterion performance. Consequently, the correlation was lower in the later group, although the act curacy with whkh individuals' grades were predicted showed little ch~nge. In other words, the observed drop in correlation did not indicate . that the predictors were less va-lid than they had been 30 years earlier. Had the difference$ in group homogeneity been ignored, it might have " been 'Wrongly concluded that this was the case. '0' For the propet interpretation of a validity coefficient, attention should alm be given to the form of the relationship between test and criterion. , The computation of a Pearson correlation coefficient a;;sumes that the relationship is linear and uniform throughout the range. There is evidence I I that in certain situations, however, these' conditions may not be met , (Fisher, 1959; Kahneman & Ghiselli, 1962). Thus, a particular job may " require a minimum level of reading comprehension, to enable employees to read instructiorl manuals, labels, and the like. Once this minimum is e:,tceeded, however, further increments in reading ability may be unrelated to degree of job success. This would be an example of a nonlinear relation between test and job performance. An examination of the bivariate distributjon or scat.\:er diagram obtained by plotting reading comprehension scores a!Ylinst criterion measures would show a rise in job perI fprmance up to the minimal required reading ability and a leveling off beyond that point. Hence, the entries would cluster around a curve rather

Validity: Mcasuremcnt

and Interprctation

~65

In other situations th 1" f b individual entries m;y d: .lfIte~ ~st 6t may be a straight line, but the at the lower end of the s:~ e Sart er around this line at the upper than aptitude test is a necClisa; ~ut u~:se that 'performa~c::e on a scholastic achievement in a course Th t' h a tufficlent condItion for successful poorly in the cOU"se' bl!lt'a' a IS t, h~ how-scoring students will perform • ,. mong the Ig -scor' t d . fonn well in the course . d th mg s u ents, some WIll permotivation. In this situat~:n ~h ers :erf~rm poorly because of low performance among the ·h.'h ere. WI e WIder variability of criterion dents, This condition in ~g ~sco~g t~an. am?ng the low-scoring stuscedasticih.' Th p. bwanate dIstrIbution is known as hctero'J' e earson correlatio h variability throughout tb ~ assum:s ?moscedasticity or eqll.al present example, the bivae..r:n~ o'b th~ bIVanate distribution. In the at the upper end and n na e shtn utIon would be fan-shaped-wide ' , arrow at t e lower end A . ' . b Ivanate distribution itsdf ill 11 . . II exammation of the nature of the relations·hip b 'tV usua y give a, good indication of the e ween test and 't' E and expectancy charts also I cn erIOn. xpectancy tables the test at different levels. correct y reveal the relative effectiveness of

",:::1

MAGNITUDE

OF

A V.Aj.LIDITY·COEFFr

•

coefficient be? No gener I CIE~T. How hIgh should a validity . . a answer to thIS gr' . mterpretation of a validit ffi . ues lOll IS pOSSIble, since the of concomitant circumsta; coe clent ~ust take into account a number be high enough to be sta~~~·' o~tamed correlation, of course, should such as the 01 or 05 level' d~8.Ica !Jds~gnificant at some acceptable level . . . s ISCusse in Cha t 5 I h ' drawing any conclusions about th • lid' per . not er words, before sonably certain that the obt' d el~d~ Ity of a test, we should be rea~ . ame va I Ity coeffi' t ld throug~ chance fluctuatip.tls of sam Ii . Clen cou not ,have arisen Havmg establjshed a signiflcant p ng fro.m a true correlation of zero. criterion, however, we need to e correlat1~n between test Scores and light of the uses to be m d f v~luate the SIZeo~ the correlation in ~he vidual's exact criterion s~ e 0 ~le test. If we WIsh to predict an indiwill receive in college the as:: grade-point average a student of the standard erro; of estimare coe .clen.t may be interpreted in terms measurement discussed in : whl7h IS analogous to the ,error of that the errOr of measure.:~;~c~~n WIth reliability. It wl"ll be recalled pected in an individual's n Icates the margin,. of error to be ex~irni1ar1y, the eITor of esti=~: :~ a res~t of th~ unreliability of the t~t. m the individual's predicted 't o~s t e margm of ~r,rotto be expe~tec:l validity of the test. cn erIon score, as a ~lwf the imper{~(;t

n;~

~:fi~~t

The error of estimate is found b th f 11 . Y e 0 owmg fOfn,ula:

";"'"

6

Prillciples of Psychological Testing

: whichr2 >'V is the square of the validity coefficient and Uv is th~ ~tandard eviatiol1 of the criterion scores. It will be noted that if the vahdlty were erfect(r >'V ;::: 1.00), the error of estimate would be zero. On the other and, with a test having zero validity, the error of estimate is as large as e standard deviation of the criterion distribution (ucBr.;::: ulIVI -0 = v), Under these conditions, the prediction is no better. than. a ~ues~; and he range of prediction error is as wide as the enbre distnbutIOn of criterionscores. Between these two extremes are to be found the errors ofestimate corresponding to tests of varying validity. Reference to the formula for cr •• t. will show that the term VI - r'''11 , servesto indicate the size of the error relative to the error that wou~ , result from a mere guess, i.e., with zero validity. In other words, lf v'l- r'xv ig equal to 1.00, the error of estimate is ~s .lar~e as it would be if we were to guess the subject's score. The predlc~ve lmprove~~nt attributable to the use of the test would thus be rol. If the validlty coefficientio; .80, then VI - "XI/ is equal to .60, and -the error is 60 percent aslarge as it would be by chance. To put it diffe:ently, the use of s~ch a test enables us to predict the individual's critenon performance wlth a marginof error that is 40 percent smaller than it would be if we were to guess. . . . . It would thus appear that even with a validlty of .80, whl~h 1S unusu~lIy high, the error of predicted scores is.conside~abl~..u th,e pnmary ~~ctl~n of psychological tests were to predlct each mdIvl~ual ~ exact l?OSlhO~in the criterion distribution, the outlook would be qUite dlscouraglOg. \\ hen examined in the light of the error of estimate, mos~ t~sts do not appear very efficient. In most testing situations, ho~ev~r,. lt IS not necessary to predict the specific criterion performance of mdlvl~ual .c~ses, but rather to determine which individuals will exceed a certam mlmmum standard of performance, or cutoff point, in the cri:erion. What are the ch.an:es that Mary Greene will graduate from medIcal school, tI:at Tom Hlggms ",'in pass a course in calculus, or that Beverly ~ruce WIll succeed as an astronaut? Which applicants are likely to be satlsfactory clc::rks,salesmen, or machine operators? Such information is ~seful not only fo~ ~roup i selection but also for individual career planmng. For example, lt 15 advantageous for a student to know that he has a gOO? chanc~ of pas~ing all courses in law school, even if we are unable to estimate WIth certamty whether his grade average will be 74 or ~I: . ., A test may appreciably improve predIctive effiCIency If It sho~s a~1J significant correlation with the criterion, however 10w..Un.der.certa~n Cltcumstanees even validities as low as .20 or .30 may Justify lncluslon of the test in ~ selection program. For many testing purposes, evaluation .of tests in terms of the error of estimate is unrealistically stringent. Conslderation must be given to other ways of evaluating the contribution of a

Validity: AI easuremellt and Interpretation

167

test, which take into account the types of decisions to be made from the scores. Some of these procedurcs will be illustrated in the following section.

BASIC APPROACH. Let us suppose that 100 applicants have been given fln aptitude test and followed up until each could be evaluated for success on a certain job. Figure 17 shows the bh'ariate distribution of test scores and measures of job success for the 100 subjects. The correlation between these two variables is slightly below .70. The minimum acceptable job performance, or criterion cutoff point, is indicated in the diagram by a heavy horizontal line. The 40 cases falling below this line would represent job failures; the 60 above the line, job successes. If all 100 appli~ants are hired, thereforc, 60 percent will succeed on the job. Similarly, if a smaller number were hired at random, without reference to test scores, the proportion of successes would probably be close to 60 percent. Suppose, however, that the test scores are used to select the 45 most promising applicants out of the 100 (selection ratio;::: .45). In such a case, the 45 individuals falling to the right of the heavy vertical line would be chosen. Within this group of 45, it can be seen that there- arc 7 job failures, or false acceptances, falling below the heavy horizontal line, and 38 job successes. Hence, the percentage of job successes is now.84 rather than 60 (i.e., 38/45 .84). This increase is attributable to the use of the test as a screening instrument. It will be noted that errors in predic:te:d criterion score that do not affect the decision can be ignored. Opl)' those prediction errors that cross the cutoff line and hence place the individual in the wrong category will reduce the selective effectiveness of the test . . For a complete evaluation of the effectiveness of the test as a screening mstrument, another category of cases in Figure 17 must also be examined. This is the category of false re;ections, comprising the 22 persons who score below the cutoff point on the test but above the criterion cutoff. From these data we would estimate that 22 percent' of the total applicant sample are potential job successes who will be lost if the test is used as a screening device with the present cutoff point. These false rejects in a personnel selection situation correspond to the false positives in clinical evaluations. The latter term has been adopted frO,J:lk medical practice, in whi~ .a t~st for a pathological condition is reported ~positive if the condltion 1S present and negative if the patient is Dormal. A false positive thus refers to ~ case in ~hich the test erroneously 4l~~atf,(~-1:hepresence ?f ~ ?athologJ~1 condition, as when brain damage~,-~ mdicated in an mdlVldual who lS actually normal. This terminology is likely to be COD-

=

Validity: Measurement and Interpretation

, I

openin s, and the ur ('nc or seed with which t , filled.

'~

I

I

I

Job

Successes

Criterion

Cutoff

Job failures

Low Low

Test Score ~'FIC.

17. Increase in the Proportion of "Successes" Resulting from the Use of

, a Selection Test.

fusingunless we remember that in clinical practice a positiv~ result po a , test denotes pathology and unfavorable diagnosis, whereas In pers~n~el . selectiona positive result conventionally refers to a favorabJ~ prediCtIon : regardingjob performance, academic achievement, and the lI~e. . In settin on a test, attention should be ven to the percentage of false rejects (or false positives as we as to the .erc:nt-i ) a cesses an ai ures wit in t~_se eete grou.!} In certam SItu;; ations,the cutoff point should be set sufficiently higt, to e~clu?e all but ',' a few possible failures. This would be the case when t~~',;obIS of such !: a nature that a poorly qualified worker could cause senous loss or dami age. An example would be a commercial airline. pilot. Under o.ther :' circumstances, it may be more important to admit. as many qualIfied ~ personsas possible, at the risk of including more fallures .. In the latter ',> case the number of false rejects can be reduced by the choice of a lower ,~,cutoffscore. Other factors that normally determine the position of ~he ."i, cutoffscore include the available personnel snpP4:, the number of job r.'

'i.

.

"

169

In many personnel decisions, the selection ratio is determined by the practical demands of the situation. Because of supply and demand in filling job openings, for example, it may be necessary to hire the top 40 percent of applicants in one case and the top 75 percent in another. When the selection ratio is not externall,T imposed, the cutting smre 011 a test can be set at that point giving the maximum differentiation be. tw~ Clilelioll grouEs. TIus can be done roughly by comparing the distrl ution of test scores in the two criterion groups. More precise mathematical procedures for setting optimal cutting scores have also been worked out (Darlington & Stauffer, 1966; Guttman & Raju, 1965; Rorer, Hoffman, La Forge, & Hsieh, 1966). These procedures make it possible to take into account other relevant parameters, such as the relative seriousness of false rejections and false acceptances. In the terminology of decision theofy, the example given in Figure 17 illustrates a simple strategy, or plan for deciding which applicants to accept and which to reject. I~ mor.e.~eral terms, a strategy is a technique for utilizing information in order to reach a decision about individuals. In tTllscase, the strategy was to accept the 45 persons with the highest te; scores. The increase in percentage of successful employees from 60 to 84 could be used as a basis for estimating the net benefit resulting from the use of the test. Statistical decision theory was developed by Wald (1950) with special reference to the decisions required in the inspection and quality control of industrial products. Many of its implications for the construction and interpretation of psychological tests have been systematically worked out by Cronbach and GIeser (1965). Essentially, decision theory is an attempt to put the decision-making process into mathematical form, so thdt available information may be used to arrive at the most effective decision ~nder .s~ecified circumstances. The mathematical procedures employed lD. d~c1Slon.th~ory a~e often quite complex, and few are in a form permItting theIr Immediate application to practical testing problems. Some of the basic concepts of decision theory, however, are proving helpful in the reformulation and clarification of certain questions about tests. A few of these ideas were introduced into testing before the formal development of statistical decision theory and were later recognized as Dtting into that framework.

A precursor of decision theory ini.psychologi. ca.1testing is ~o b~ found in the Taylor-Russell table~( 193,~),which permIt a detennmation of -the net gain in selection acc~racy atbibutable to the use of the test. ~ information required inc1\ip s'the validity coPREDICTION OF OUTCOMES.

l

Validity: Measurement and Interpretation o

Principles of Psychological

Testing

cient of the test, the proportion of applicants who m~~t be acclep~e~ lection ratio), and the proportion of successfu~ app lc~n~ :: ~~r:e thout the use of the test (base rate). A change many 0 t I" ctors can alter the predictive efficiency of the test. For urposes of illustration, one of the Taylor-Russell tables has been e rod~eed in Table 14. This table is designed for us~ when the base .aie or ercenta e of successful applicants selected pnor to the use of he test 1s 60. Ot~er tables are prOVided by Taylor and Russe~l for ~t~~r base ra~es Across the top of the table are given different va ues ~ .e selection ;atio, and along the side are the tes~ validities. The entnes 111 the' body of the table indicate the proportion of successful· persons

TABLE

14

i

f G'

(

Proportionof "Successes" Expected through the Use 0 Test Validityand Given Selection Ratio, for Base Rate .60. (From Taylor and Russell, 1959, p. 576) ~':~~,",7"'J2'-':UliH~~'.:>,JI;~~,.:~!M.r

0

lven

=-"

.

••_.:::·..:;.':5.~~~

Selection Ratio .30

.40

.50

.60 .70 .80 .90

.95

.60 .60 .60 ·60 .60 .61 .61 .61 .60 .60 .63 .62 .61 .61 .60 .64 .63 .62 .61 .61 .65 .64 .63 .62 .61

.66 .65 .63 .62 .61 .68 .66 .64 .62. .6J .69 .70

.75 .80

.99 1.00

.85

1.00 1.00 1.00 1.00

.90

.95 1.00

.99 1.00 1.00 1.00

.96 .98 .99 1.00 1.00

.93 .95 .97 .99

.90 .92

.95 .97

.86 .88 .91 .94

1.00

1.00

1.00

1.00

.99 1.00

1.00

.99

.97

.67 .68

.65 .66

.63 .62 .63.62

.72 .69 .66 .64

.62

.73 .70 .67 .64 .75 .71 .68 .64 .76 .73 .69 .65 .78 .74 .70 .65 .80 .75 .71 .66

.62

.62 .63 .63

.63

.81 .77 .71 .66 .63 .83 .78 .72 .66 .63 .86 .80 .73 .66 .63 .88 .82 .74 .67 .63 .92 .84 .75 .67 .63 1.00 .86 .75 .67 .63

171

selected after the use of the test. Thus, the difference between .60 and anyone table entry shows the increase in proportion of successful selections attributable to the test. Obviously if the selection ratio w.ere 100 percent, that is, if all applicants had to be accepted, no test, howen'r valid, could improve the selection process. Reference to Table 14 sho\\'s that, when as many as 95 percent of applicants must be admitted, even a test with perfect validity ( r = 1.00) would raise the proportion of successful persons by only 3 percent (.60 to .63). On the other hand, when only 5 percent of applicants need to be chosen, a test with a validity coefficient of only .30 can raise the percentage of successful applicants selected from 60 to 82. The rise from 60 to 82 represents the incremental vaUdity of the test (Sechrest, 1963), or the increase in predictive validity attributable to the test. It indicates the contribution the test makes to the selection of individuals who will meet the minimum standards in criterion performance. In applying the Taylor-Russell tables, of course, test validity should be computed on the same sort of group used to estimate percentage of prior successes. In other words, the contribution of the test is not evaluated against chance success unless 'applicants were preViously selected by chance-a most unlikely circumstance. If applicants had been sele