Raven's Apm International Technical Manual

 

                   

Raven’s Advanced Progressive Matrices International Technical Manual

                                 

         

Copyright © 2011 NCS Pearson, Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the copyright owner. Pearson, the TalentLens, Watson-Glaser Critical Thinking Appraisal, and Raven’s Progressive Matrices are trademarks in the U.S. and/or other countries of Pearson Education, Inc., or its affiliate(s). Portions of this work were previously published. Produced in the United Kingdom

Contents Chapter 1 Introduction .......................................................................................................1 Development of the 23-Item Form....................................................................................................2 Internal Consistency Reliability...........................................................................................................3 Content Validity.....................................................................................................................................3 Convergent Validity ..............................................................................................................................3 Criterion-Related Validity...................................................................................................................4 Equivalency Information ......................................................................................................................5 Global Applicability...............................................................................................................................7 Development of Raven’s APM International Versions ..................................................................7

Chapter 2 Australia/New Zealand (English) ...................................................................9 Translation/Adaptation Process.................................................................................................................9 Sampling Procedure ......................................................................................................................................9 Item/Test Difficulty .......................................................................................................................................11 Distribution of Scores ................................................................................................................................11 Evidence of Reliability .................................................................................................................................12

Chapter 3 France (French)........................................................................................................13 Translation/Adaptation Process................................................................................................................13 Sampling Procedure ....................................................................................................................................13 Item/Test Difficulty .....................................................................................................................................14 Distribution of Scores ...............................................................................................................................15 Evidence of Reliability ................................................................................................................................15

Chapter 4 India (English) .........................................................................................................17 Translation/Adaptation Process.......................................................................................................17 Sampling Procedure ............................................................................................................................17 Item/Test Difficulty .............................................................................................................................18 Distribution of Scores .......................................................................................................................19 Evidence of Reliability ........................................................................................................................19

Chapter 5 The Netherlands (Dutch) ...............................................................................21 Translation/Adaptation Process...........................................................................................................21 Sampling Procedure ................................................................................................................................21 Item/Test Difficulty .................................................................................................................................23 Distribution of Scores ...........................................................................................................................23 Evidence of Reliability ............................................................................................................................20

Chapter 6 The UK (English) .................................................................................................25 Translation/Adaptation Process...........................................................................................................25 Sampling Procedure ................................................................................................................................25 Item/Test Difficulty .................................................................................................................................27 Distribution of Scores ...........................................................................................................................28 Evidence of Reliability ............................................................................................................................28

Chapter 7 The US (English) ................................................................................................29 Sampling Procedure ..............................................................................................................................29 Item/Test Difficulty ...............................................................................................................................30 Distribution of Scores .........................................................................................................................31 Evidence of Reliability ..........................................................................................................................31

Appendix A Best Practices in Administering and Interpreting the APM .................................33 Administrator’s Responsibilities ........................................................................................................33 Assessment Conditions .......................................................................................................................33 Answering Questions ..........................................................................................................................34 Administration.......................................................................................................................................34 Understanding the Scores Reported ...............................................................................................35 Maintaining Security of Results and Materials ................................................................................35 Sources of Additional Best Practice Information...........................................................................35 Instructions for Administering the APM Online............................................................................37 APM Short Test Administration Instructions – Paper and Pen..................................................41

References ..........................................................................................................43 Tables 1.1 Descriptive Statistics of the APM by Test Version and Administration Order....................6 1.2 Reliability Estimates by APM Test Version and Administration Order .................................6 1.3 Descriptive Statistics and Internal Consistency Reliability Estimates for Raven’s APM Across Countries........................................................................................................................7 2.1 Demographic Information for the Australia/New Zealand Sample .......................................10 2.2 Raven’s APM Item Analysis Information for the Australia/New Zealand Sample ................11 2.3 Distribution of APM Scores in the Australia/New Zealand Sample ......................................11 2.4 Internal Consistency Reliability Estimates in the Australia/New Zealand Sample..............12 3.1 Demographic Information for the France Sample .....................................................................13 3.2 Raven’s APM Item Analysis Information for the France Sample..............................................14 3.3 Distribution of APM Scores in the France Sample ....................................................................15 3.4 Internal Consistency Reliability Estimates in the France Sample ...........................................15 4.1 Demographic Information for the India Sample.........................................................................17 4.2 Raven’s APM Item Analysis Information for the India Sample .................................................18 4.3 Distribution of APM Scores in the India Sample .......................................................................19 4.4 Internal Consistency Reliability Estimates in the India Sample...............................................19 5.1 Demographic Information for the Netherlands Sample ..........................................................22

5.2 Raven’s APM Item Analysis Information for Netherlands Sample .........................................23 5.3 Distribution of APM Scores in the Netherlands Sample .........................................................23 5.4 Internal Consistency Reliability Estimates in the Netherlands Sample ................................24 6.1 Demographic Information for the UK Sample ..........................................................................26 6.2 Raven’s APM Item Analysis Information for the UK Sample...................................................27 6.3 Distribution of APM Scores in the UK Sample .........................................................................28 6.4 Internal Consistency Reliability Estimates in the UK Sample.................................................28 7.1 Demographic Information for the US Sample ...........................................................................29 7.2 Raven’s APM Item Analysis Information for the US Sample....................................................30 7.3 Distribution of APM Scores in the US Sample ..........................................................................31 7.4 Internal Consistency Reliability Estimates in the US Sample.................................................. 31

Chapter 1 Introduction

The Raven’s Progressive Matrices have been used in many countries for decades as a measure of problem-solving and reasoning ability (Raven, Raven, & Court, 1998a). The various versions of the Raven’s Progressive Matrices have been studied in over 45 countries on samples totalling over 240,000 participants (Brouwers, Van de Vigver, & Van Hemert, 2009). This manual describes the adaptation/translation of the latest 23-item version of the Raven’s Advanced Progressive Matrices (APM) for the US, Australia/New Zealand, France, India, the Netherlands and the UK. From an international perspective, several enhancements were made to facilitate cross-country score comparisons, and to standardise the testing experience for administrators and participants. These enhancements include: 

Use of a uniform test format and common test content across countries



Uniform scoring and reporting of scores across countries



Availability of local manager norms for each country, based on a common definition of manager across countries



Implementation of a common set of items and administration time across countries (i.e., 23 items; 40 minutes).

Description of the Raven’s Advanced Progressive Matrices The Raven’s Advanced Progressive Matrices (APM) is a nonverbal assessment tool designed to measure an individual’s ability to perceive and think clearly, make meaning out of confusion and formulate new concepts when faced with novel information. The APM score indicates a candidate’s potential for success in such positions as executive, director, general manager, or equivalent high-level technical or professional positions in an organisation. These categories of positions typically require high levels of

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

1

clear and accurate thinking, problem identification, holistic situation assessment, and evaluation of tentative solutions for consistency with all available information. Each item in the APM comprises a pattern of diagrammatic puzzles with one piece missing. The candidate’s task is to choose the correct missing piece from a series of possible answers.

Development of the Current 23-Item Form The current revision of the APM was undertaken to provide customers with a shorter version of the assessment that maintains the essential nature of the construct being measured and the psychometric features of the assessment. The APM is a power assessment rather than a speeded assessment, even though it has a time limit. Speeded assessments are typically composed of relatively easy items and rely on the number of correct responses within restrictive time limits to differentiate performance among candidates. In contrast, the APM items have a wide range of difficulty and a relatively generous time limit, which makes it a power assessment. The 42-minute administration time for the current APM (40 minutes for 23 operational items in Part 1; 2 minutes for 2 experimental items in Part 2) maintains the APM as an assessment of cognitive reasoning power rather than speed. N.B. The paper and pencil format does not contain Part 2 experimental items

Classical Test Theory (CTT) and Item Response Theory methodologies were used in the analyses of the APM data for item selection. Specifically, for each of the 36 items in the previous APM version, the following indices were examined to select items: item difficulty index (p value), corrected item-total correlation, IRT item discrimination (a) parameter, and IRT item difficulty (b) parameter. Because the APM was designed to differentiate among individuals with high mental ability, less discriminating items were dropped from the current version of the APM. For the current APM revision, data were used from 929 applicants and employees in a number of positions across various occupations. These individuals took the APM within the period May 2006 through to October 2007. Five hundred and nine of these individuals provided responses about their current position levels (e.g., “Executive,” “Director,”

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

2

“Manager,” and “Professional/Individual Contributor”). See the Appendix of separate document ‘APM Development’ 2007, for more details regarding the composition of the sample.

Internal Consistency Reliability The internal consistency reliability estimate (split-half) for the APM total raw score was .85 in the U.S standardisation sample (n=929). This reliability estimate for the 23-item version of the APM indicates that the total raw score on the APM possesses good internal consistency reliability. Internal consistency reliability estimates for each country-specific Manager norm group in the global data-collection effort are summarised in each country-specific chapter within this manual.

Content Validity In an employment setting, evidence of content validity exists when an assessment includes a representative sample of tasks, behaviours, knowledge, skills, abilities, or other characteristics necessary to perform the job. Evidence of the content-related validity of the APM should be established by demonstrating that the jobs for which the APM is to be used require the problem solving skills measured by the assessment. Such evidence is typically documented through a thorough job analysis.

Convergent Validity Evidence of convergent validity is provided when scores on an assessment relate to scores on other assessments that claim to measure similar traits or constructs. Years of previous studies on the APM support its convergent validity (Raven, Raven, & Court, 1998b). In a sample of 149 college applicants, APM scores correlated .56 with math scores on the American College Test (Koenig, Frey, & Detterman, 2007). Furthermore, in a study using 104 university students, Frey and Detterman (2004) reported that scores from the APM correlated .48 with scores on the Scholastic Assessment Test (SAT). Evidence of convergent validity for the current version of the APM is supported by two findings. First, in the standardisation sample of 929 individuals, scores on the current APM correlated .98 with scores on

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

3

the previous APM. Second, in a subset of 41 individuals from the standardisation sample, the revised APM scores correlated .54 with scores on the Watson-Glaser Critical Thinking Appraisal®—Short Form (Watson & Glaser, 2006).

Criterion-Related Validity Criterion-related validity addresses the inference that individuals who score better on an assessment will be more successful on some criterion of interest (e.g., job performance). Criterion-related validity for general mental ability tests like the APM is supported by validity generalisation. The principle of validity generalisation refers to the extent that inferences from accumulated evidence of criterionrelated validity from previous research can be generalized to a new situation. There is abundant evidence that measures of general mental ability, such as the APM, are significant predictors of overall performance across jobs. For example, in its publication on the Principles for the Validation and Use of Personnel Selection Procedures, SIOP (2003) notes that validity generalisation is well-established for cognitive ability tests. Schmidt & Hunter (2004) provide evidence that general mental ability “predicts both occupational level attained and performance within one's chosen occupation and does so better than any other ability, trait, or disposition and better than job experience” (p. 162). Prien, Schippmann, and Prien (2003) observe that decades of research “present incontrovertible evidence supporting the use of cognitive ability across situations and occupations with varying job requirements” (p. 55). Many other studies provide evidence of the relationship between general mental ability and job performance (e.g., Kolz, McFarland, & Silverman, 1998; Kuncel, Hezlett, & Ones, 2004; Ree & Carretta, 1998; Salgado, et al., 2003; Schmidt & Hunter, 1998; Schmidt & Hunter, 2004).

In addition to inferences based on validity generalisation, studies using the APM over the past 70 years provide evidence of its criterion-related validity. For example, in a validation study of assessment centres, Chan (1996) reported that scores on the Raven’s Progressive Matrices correlated with ratings of participants on “initiative/creativity” (r=.28, p< .05). Another group of researchers (Gonzalez,

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

4

Thomas, and Vanyukov, 2005) reported a positive relationship between scores on the Raven’s APM and performance in decision-making tasks. Fay and Frese (2001) found that APM scores were “consistently and positively associated with an increase of personal initiative over time” (p. 120). Recently, Pearson (2010) conducted a study of 106 internal applicants for management positions in which APM scores were positively correlated with trained assessor ratings of “thinking, influencing, and achieving.” In addition, manager applicants scoring in the top 30% of APM scores were 2-3 times more likely to receive above average ratings for the “Case Study/Presentation Exercise”, “Thinking Ability”, and “Influencing Ability” than applicants in the bottom 30% of APM scores.

The APM Manual and Occupational User’s Guide (Raven, 1994; Raven, Raven, & Court, 1998b) provide additional information indicating that the APM validly predicts the ability to attain and retain jobs that require high levels of general mental ability. The validity information presented in this manual is not intended to serve as a substitute for locally obtained validity data. Local validity studies, together with locally derived norms, provide a sound basis for determining the most appropriate use of the APM. Therefore, users of the APM should study the validity of the assessment at their own location or organisation.

Equivalency Information Occasionally, customers inquire about the equivalence of online versus paper administration of the APM. Studies of the effect of the medium of test administration have generally supported the equivalence of paper and computerised versions of non-speeded cognitive ability tests (Mead & Drasgow, 1993). To ensure that these findings held true for the Raven’s Advanced Progressive Matrices, Pearson TalentLens conducted an equivalency study using paper-and-pencil and computer-administered versions of the test. The psychometric properties of the two forms were compared to determine whether the mode of administration impacted scores and whether decision consistency could be assured across modes of administration.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

5

In this study, a counter-balanced design was employed using a sample of 133 adult participants from a variety of occupations. Approximately half of the group (n=68) completed the paper form followed by the online version, while the other participants (n=65) completed the tests in the reverse order. The interval between test sessions ranged from 13 to 91 days (M=22.9, SD=12.1). Table 1.1 presents means, standard deviations, and correlations obtained from an analysis of the resulting data. Analyses of the test modes revealed that there was no significant difference in the examinees’ raw scores on the APM between paper (M= 11.9, SD= 4.9) and online versions (M= 11.7, SD= 4.8), t(132) = -0.95, p = .34). The total scores from the different versions were highly correlated (r= .78). Table 1.1 Descriptive Statistics of the APM by Test Version and Administration Order Administration Order Paper First Online First Total

N 68 65 133

APM Test Version Paper Online M SD M SD 12.4 4.8 13.0 5.5 11.5 4.8 10.3 3.7 11.9 4.8 11.7 4.9

r .85 .73 .78

Table 1.2 displays the reliability estimates of the paper and online versions from the different test administration groups. These estimates demonstrate that reliability estimates of APM scores fall within the acceptable range regardless of test modes or administration order, providing additional support for equivalence. Table 1.2 Reliability Estimates by APM Test Version and Administration Order APM Test Version Paper Administration Order Paper First Online First Total

Online

rsplit .88 .86 .86

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

ralpha .83 .82 .83

6

rsplit .88 .75 .85

ralpha .87 .70 .83

Global Applicability The nonverbal aspect of the APM minimizes the impact of language skills on performance on the assessment. The fact that the Raven’s shows less influence of cultural factors than more verbally-laden assessments has made it very appealing as global measure of cognitive ability. The global exposure of the Raven’s abstract reasoning format has several important advantages for inclusion in a global selection strategy. Specifically, its’ familiarity increases the likelihood of local management support, it promotes positive participant reactions and helps ensure that scores aren’t unduly influenced by language or culture (see Ryan & Tippins, 2009 for more information on implementing a consistent and effective global selection system). The following chapter provides important information in helping multinational companies incorporate the APM into a global selection system, and to make informed comparisons of applicants’ performance on the APM across countries and cultures.

Development of Raven’s Advanced Progressive Matrices (APM) International Versions Development for each country-specific version of the APM followed a uniform process focused on adapting and translating the test instructions in a way that ensured consistent measurement of the construct across countries. The international versions are based on the same 23-item set, including the practice items, matrix stimuli, and response options. Table 1.3 presents a summary of results by country, including the number of managers tested, characteristics of the score distribution, and total score reliability. Table 1.3 Descriptive Statistics and Internal Consistency Reliability Estimates for Raven’s APM Across Countries Country Australia/New Zealand France India Netherlands UK US

N 128 106 100 103 101 175

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

Mean 11.95 14.33 9.51 13.01 12.38 12.23

SD 4.17 4.09 4.22 4.53 4.72 4.14

7

Skewness

Kurtosis

0.02 -0.34 0.19 -0.10 0.04 -0.03

-0.39 -0.09 -0.80 -0.64 -0.71 -0.13

ralpha .77 .74 .79 .81 .83 .77

rsplit .78 .79 .82 .83 .85 .81

Results showed that internal consistency reliability estimates across countries were adequate (e.g., rsplit = .79-.85), and that sample homogeneity and differences in prior exposure to cognitive ability testing may account for observed differences in raw score means across countries. Detailed information on the collection and analyses of country-specific norms is provided throughout the remainder of this Manual.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

8

Chapter 2 Australia/New Zealand (English) Translation/Adaptation Process Instructions for the APM were reviewed and adapted by a team of test-development experts from the Sydney, Australia Pearson TalentLens office.

Sampling Procedure The Sydney, Australia office of Pearson TalentLens recruited and administered the online version of the APM to 128 manager-level examinees across various industries. These individuals took the APM under timed (40-minutes) and proctored (i.e. supervised) conditions within the period November, 2009 through to September, 2010. Table 2.1 provides the demographic data for the final sample of N=128.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

9

Table 2.1 Demographic Information for the Australia/New Zealand Sample N 128 Education Level Year 11 or equivalent Year 12 or equivalent Certificate III / IV Diploma Advanced Diploma Bachelor Graduate Certificate Graduate/Postgraduate Diploma Master Doctorate Other Not Reported Sex Female Male Not Reported Age 21-24 25-29 30-34 35-39 40-49 50-59 60-69 Not Reported Years in Occupation 15 Not Reported

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

Percent 100.0

2 4 2 3 2 28 1

1.6 3.1 1.6 2.3 1.6 21.9 0.8

20 47 3 4 12

15.6 36.7 2.3 3.1 9.4

61 55 12

47.7 43.0 9.4

1 9 19 25 28 29 5 12

0.8 7.0 14.8 19.5 21.9 22.7 3.9 9.4

18 6 27 17 14 14 18 14

14.1 4.7 21.1 13.3 10.9 10.9 14.1 10.9

10

Item/Test Difficulty Classical Test Theory (CTT) and Item Response Theory (IRT) methodologies were used in the analysis of the APM data collected in Australia and New Zealand. Specifically, for each of the 23 items in the APM, the following indices were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item discrimination (a) parameter, item difficulty index (p value), and item-total correlation. Results are presented in Table 2.2. Table 2.2 Raven’s APM Item Analysis Information for the Australia/New Zealand Sample APM Item Number

Item Difficulty (b) Parameter (IRT)

Item-Ability Correlation (IRT)

Discrimination (a) Parameter (IRT)

Item Difficulty Index (p value; CTT)

Item-Total Correlation (CTT)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

-3.38 -3.05 -1.82 -1.51 -1.24 -0.55 -1.24 0.33 0.05 -0.14 0.05 0.36 0.48 0.68 1.15 1.01 0.80 0.17 1.06 1.28 0.64 1.24 3.63

0.25 0.37 0.43 0.40 0.30 0.44 0.44 0.41 0.46 0.46 0.41 0.31 0.43 0.42 0.39 0.35 0.49 0.42 0.37 0.41 0.42 0.42 0.43

1.03 1.11 1.12 1.05 0.85 1.07 1.10 0.91 1.12 1.13 0.91 0.53 1.00 0.99 0.93 0.80 1.21 0.96 0.86 0.99 0.98 1.00 1.09

0.95 0.94 0.83 0.80 0.76 0.64 0.76 0.46 0.52 0.56 0.52 0.46 0.43 0.39 0.31 0.33 0.37 0.50 0.32 0.28 0.40 0.29 0.05

0.22 0.33 0.37 0.34 0.22 0.36 0.36 0.31 0.38 0.38 0.31 0.21 0.33 0.33 0.28 0.24 0.42 0.32 0.27 0.32 0.33 0.31 0.31

Distribution of Scores Characteristics of the distribution of APM raw scores for the Australia/New Zealand sample are provided in Table 2.3. Table 2.3 Distribution of APM Scores in the Australia/New Zealand Sample N 128

Minimum 3

Maximum 23

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

M 12.0

11

SD 4.2

Skewness 0.02

Kurtosis -0.39

Evidence of Reliability Split-half (rsplit), Cronbach’s alpha (ralpha), and standard error of measurement (SEM) were calculated using the Australia/New Zealand Sample. Results are presented in Table 2.4. Internal consistency reliability estimates were consistent with the values found in the US standardisation sample and confirm that the APM demonstrates adequate reliability in the Australia/New Zealand sample.

Table 2.4 Internal Consistency Reliability Estimates in the Australia/New Zealand Sample Raven’s APM Total Score

N 128

rsplit .78

Note. SEM was calculated based on split-half reliability.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

12

ralpha .77

SEM 1.96

Chapter 3 France (French) Translation/Adaptation Process Instructions for the APM were translated by a team of test-development experts from the Paris, France office of Pearson TalentLens (Les Editions du Centre de Psychologie Appliquee).

Sampling Procedure The Paris, France office of Pearson TalentLens recruited and administered the online version of the APM to 100 manager-level examinees across various industries. These individuals took the APM under timed (40-minutes) and proctored (i.e. supervised) conditions within the period November, 2009 through to March, 2010. Table 3.1 provides the demographic data for the final sample of N=100. Table 3.1 Demographic Information for the France Sample N 100 Education Level 11 ans (1ière, CAP-BEP) 13-14 ans (Bac +1 et 2) 15-16 ans (Bac +2 et 3) 17-18 ans (Bac + 4 et 5) Plus de 18 ans (Doctorat) Not Reported Sex Female Male Age 21-24 25-29 30-34 35-39 40-49 50-59 60-69 Not Reported

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

1 7 11 74 4 3

Percent 100.0 1.0 7.0 11.0 74.0 4.0 3.0

51 49

51.0 49.0

13 25 17 10 18 15 1 1

13.0 25.0 17.0 10.0 18.0 15.0 1.0 1.0

13

Item/Test Difficulty Classical Test Theory (CTT) and Item Response Theory (IRT) methodologies were used in the analysis of the APM data collected in France. Specifically, for each of the 23 items in the APM, the following indices were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item discrimination (a) parameter, item difficulty index (p value), and item-total correlation. Results are presented in Table 3.2. Table 3.2 Raven’s APM Item Analysis Information for the France Sample APM Item Number

Item Difficulty (b) Parameter (IRT)

Item-Ability Correlation (IRT)

Discrimination (a) Parameter (IRT)

1 2* 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Item Difficulty Index (p value; CTT)

Item-Total Correlation (CTT)

-3.38 0.05 0.90 0.97 0.00 -1.04 0.34 0.97 0.80 0.25 -1.97 0.32 1.03 0.90 0.26 -0.76 0.39 1.03 0.76 0.32 -0.45 0.15 0.45 0.71 0.02 -0.70 0.32 0.90 0.75 0.23 -0.70 0.45 1.13 0.75 0.37 -0.07 0.44 1.10 0.64 0.34 -0.63 0.49 1.21 0.74 0.43 0.34 0.38 0.88 0.56 0.26 -0.17 0.48 1.21 0.66 0.40 -0.07 0.25 0.55 0.64 0.13 0.49 0.50 1.32 0.53 0.41 0.98 0.40 0.92 0.43 0.27 1.18 0.42 0.99 0.39 0.30 0.04 0.52 1.35 0.62 0.44 0.34 0.57 1.56 0.56 0.50 0.78 0.24 0.31 0.47 0.13 0.93 0.45 1.08 0.44 0.32 0.88 0.43 1.03 0.45 0.32 0.93 0.48 1.20 0.44 0.38 3.06 0.32 0.98 0.12 0.24 *100 percent of the sample obtained a perfect score on Item 2, making the IRT and CTT values inestimable.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

14

Distribution of Scores Characteristics of the distribution of APM raw scores for the France sample are provided in Table 3.3. Table 3.3 Distribution of APM Scores in the France Sample N 100

Minimum 3

Maximum 22

M 14.3

SD 3.9

Skewness -0.31

Kurtosis -0.16

Evidence of Reliability Split-half (rsplit), Cronbach’s alpha (ralpha), and standard error of measurement (SEM) were calculated using the France Sample. Results are presented in Table 3.4. Internal consistency reliability estimates were consistent with the values found in the US standardisation sample and confirm that the APM demonstrates adequate reliability in the France sample. Table 3.4 Internal Consistency Reliability Estimates in the France Sample Raven’s APM Total Score

N 100

rsplit .79

Note. SEM was calculated based on split-half reliability.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

15

ralpha .74

SEM 1.80

Raven’s Advanced Progressive Matrices (APM)

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

16

Chapter 4 India (English)

Translation/Adaptation Process Instructions for the APM were reviewed and adapted by a team of test-development experts from the Pearson TalentLens Bangalore, India office.

Sampling Procedure The Bangalore, India office of Pearson TalentLens recruited and administered the online version of the APM to 100 manager-level examinees across various industries. These individuals took the APM under timed (40-minutes) and proctored (i.e. supervised) conditions within the period February, 2010 through to April, 2010. Table 4.1 provides the demographic data for the final sample of N=100. Table 4.1 Demographic Information for the India Sample Education Level 10th 12th Bachelors Masters Doctoral Other Sex Female Male Age 20-24 25-29 30-34 35-39 40-44 45-49 Not Reported

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

N 100

Percent 100.0

1 3 49 39 2 6

1.0 3.0 49.0 39.0 2.0 6.0

22 78

22.0 78.0

2 27 33 17 10 4 7

2.0 27.0 33.0 17.0 10.0 4.0 7.0

17

Item/Test Difficulty Classical Test Theory (CTT) and Item Response Theory (IRT) methodologies were used in the analysis of the APM data collected in India. Specifically, for each of the 23 items in the APM, the following indices were examined: IRT item difficulty (b) parameter, item-ability (theta) correlation, item discrimination (a) parameter, item difficulty index (p value), and item-total correlation. Results are presented in Table 4.2. Table 4.2 Raven’s APM Item Analysis Information for the India Sample APM Item Number

Item Difficulty (b) Parameter (IRT)

Item-Ability Correlation (IRT)

Discrimination (a) Parameter (IRT)

Item Difficulty Index (p value; CTT)

Item-Total Correlation (CTT)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

-2.85 -2.27 -1.85 -2.05 -1.49 -1.43 -1.21 0.12 -0.57 -0.57 0.64 0.01 0.52 1.42 1.34 1.59 0.17 0.76 0.52 0.83 1.42 1.03 3.94

0.39 0.53 0.54 0.43 0.53 0.52 0.46 0.40 0.57 0.52 0.55 0.38 0.43 0.29 0.33 0.23 0.40 0.30 0.51 0.45 0.32 0.35 0.01

1.05 1.21 1.21 1.01 1.17 1.13 0.96 0.83 1.29 1.14 1.26 0.73 0.98 0.88 0.93 0.83 0.82 0.74 1.16 1.07 0.94 0.91 0.90

0.86 0.79 0.73 0.76 0.67 0.66 0.62 0.37 0.50 0.50 0.28 0.39 0.30 0.17 0.18 0.15 0.36 0.26 0.30 0.25 0.17 0.22 0.02

0.30 0.43 0.46 0.31 0.43 0.42 0.34 0.31 0.48 0.43 0.48 0.27 0.35 0.22 0.25 0.15 0.30 0.20 0.43 0.37 0.26 0.26 -0.02

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

18

Distribution of Scores Characteristics of the distribution of APM raw scores for the India sample are provided in Table 4.3. Table 4.3 Distribution of APM Scores in the India Sample N 100

Minimum 2

Maximum 19

M 9.5

SD 4.2

Skewness 0.19

Kurtosis -0.80

Evidence of Reliability Split-half (rsplit), Cronbach’s alpha (ralpha), and standard error of measurement (SEM) were calculated using the India sample. Results are presented in Table 4.4. Internal consistency reliability estimates were consistent with the values found in the US standardisation sample and confirm that the APM demonstrates adequate reliability in the India sample.

Table 4.4 Internal Consistency Reliability Estimates in the India Sample Raven’s APM Total Score

N 100

rsplit .82

Note. SEM was calculated based on split-half reliability.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

19

ralpha .79

SEM 1.79

Raven’s Advanced Progressive Matrices (APM)

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

20

Chapter 5 Netherlands (Dutch)

Translation/Adaptation Process Instructions for the APM were translated into Dutch by an independent translator, contracted by the Amsterdam, Netherlands office of Pearson TalentLens. Two Dutch-speaking test-development experts from the Pearson TalentLens Amsterdam office reviewed the translation and refined the final translation.

Sampling Procedure The Amsterdam, Netherlands office of Pearson TalentLens recruited and administered the online version of the Dutch APM to 138 manager-level examinees across various industries. These individuals took the APM under timed (40-minutes) and proctored (i.e. supervised) conditions within the period September through to October 2009.Thirty-five participants were eliminated from the sample after a review of self-reported job titles revealed that they did not qualify as a Manager. Table 5.1 provides the demographic data for the final sample of N=103.

Copyright © 2011 NCS Pearson, Inc or its affiliate(s)

21

Table 5.1 Demographic Information for the Netherlands Sample N 103 Education Level