Designing classroom language tests Chapter 3

FOR EDUCATIONAL USE ONLY YESDİL – Eskişehir’in İÖLP Markası

http://www.yesdil.com/v1/pdf/testing_language_assessment.pdf CHAPTER 3 DESIGNING CLASSROOM LANGUAGE TESTS In this chapter, we will examine test types, and we will learn how to design tests and revise existing ones. To start the process of designing tests, we will ask some critical questions. The following five questions should form the basis of your approach to designing tests for your classroom. Question 1: What • • • •

is the purpose of the test? Why am I creating this test? For an evaluation of overall proficiency? (Proficiency Test) To place students into a course? (Placement Test) To measure achievement within a course? (Achievement Test)

Once you have established the major purpose of a test, you can determine its objectives. Question 2: What are the objectives of the test? • What specifically am I trying to find out? • What language abilities are to be assessed? Question 3: How will the test specifications reflect both the purpose and objectives? • When a test is designed, the objectives should be incorporated into a structure that appropriately weights the various competencies being assessed. Question 4: How will the test tasks be selected and the separate items arranged? • The tasks need to be practical. • They should also achieve content validity by presenting tasks that mirror those of the course being assessed. • They should be evaluated reliably by the teacher or scorer. • The tasks themselves should strive for authenticity, and the progression of tasks ought to be biased for best performance. Question 5: What • • •

kind of scoring, grading, and/or feedback is expected? Tests vary in the form and function of feedback, depending on their purpose. For every test, the way results are reported is an important consideration. Under some circumstances a letter grade or a holistic score may be appropriate; other circumstances may require that a teacher offer substantive washback to the learner.

www.yesdil.com

-20-

YESDİL – Eskişehir’in İÖLP Markası TEST TYPES Defining your purpose will help you choose the right kind of test, and it will also help you to focus on the specific objectives of the test.

Below are the test types to be examined: 1. 2. 3. 4. 5.

Language Aptitude Tests Proficiency Tests Placement Tests Diagnostic Tests Achievement Tests

1. Language Aptitude Tests      



They predict a person’s success prior to exposure to the second language. A language aptitude test is designed to measure capacity or general ability to learn a foreign language. Language aptitude tests are ostensibly(görünüşte olan) designed to apply to the classroom learning of any language. Two standardized aptitude tests have been used in the US. The Modern Language Aptitude Test(MLAT), and the Pimsleur Language Aptitude Battery(PLAB) Tasks in MLAT includes: Number learning, phonetic script, spelling clues, words in sentences, and paired associates. There’s no unequivocal(su götürmez bir şekilde) evidence that language aptitude tests predict communicative success in a language. (Yani bu tür test sonuçları ile öğrencilerin dil öğrenme süreçleri arasında genelde bir tutarlılık olsa da bu testlerin mutlak olduğu söylenemez.) Any test that claims to predict success in learning a language is undoubtedly flawed because we now know that with appropriate self-knowledge, and active strategic involvement in learning, virtually everyone can succeed eventually.

2. Proficiency Tests  

     

A proficiency test is not limited to any one course, curriculum, or single skill in the language; rather, it tests overall ability. It includes: standardized multiple choice items on grammar, vocabulary, reading comprehension, and aural(işitsel) comprehension. Sometimes a sample of writing is added, and more recent tests also include oral production. Such tests often have content validity weaknesses. Proficiency tests are almost always summative and norm-referenced. They are usually not equipped to provide diagnostic feedback. Their role is to accept or to deny someone’s passage into the next stage of a journey. TOEFL is a typical standardized proficiency test. Creating these tests and validating them with research is a time-consuming and costly process. To choose one of a number of commercially available proficiency tests is a far more practical method for classroom teachers.

3. Placement Tests   

The ultimate objective of a placement test is to correctly place a student into a course or level. Certain proficient tests can act in the role of placement tests. A placement test usually includes a sampling of the material to be covered in the various courses in a curriculum.

www.yesdil.com

-21-

YESDİL – Eskişehir’in İÖLP Markası  

  

In a placement test, a student should find the test material neither too easy nor too difficult but appropriately challenging. The English as a Second Language Placement Test (ESLPT) at San Francisco State University has three parts. Part 1: students read a short article and then write a summary essay. Part 2: students write a composition in response to an article. Part 3: multiple-choice; students read an essay and identify grammar errors in it. The ESL is more authentic but less practical, because human evaluators are required for the first two parts. Reliability problems are also present but are mitigated(hafifletmek) by conscientious training of all evaluators of the test. What is lost in practicality and reliability is gained in the diagnostic information that the ESLPT provides.

4. Diagnostic Tests   





A diagnostic test is designed to diagnose specified aspects of a language. A diagnostic test can help a student become aware of errors and encourage the adoption of appropriate compensatory strategies. A test of pronunciation, for example, might diagnose the phonological features of English that are difficult for learners and should therefore become part of a curriculum. Usually such tests offer a checklist of features for the administrator to use in pinpointing difficulties. Another example: a writing diagnostic would elicit a writing sample from students that would allow the teacher to identify those rhetorical and linguistic features on which the course needed to focus special attention. A typical diagnostic test of oral production was created by Clifford Prator(1972) to accompany a manual of English pronunciation. In the test; • Test-takers are directed to read a 150-word passage while they are taperecorded. • The test administrator then refers to an inventory(envanter, deftere kayıtlı eşya) of phonological items for analyzing a learner’s production. • After multiple listenings, the administrator produces a checklist for errors in five separate categories. o o o o o

Stress and rhythm, Intonation, Vowels, Consonants, and Other factors.

This information can help teacher make decisions about aspects of English phonology. 5. Achievement Tests  



An achievement test is related directly to classroom lessons, units, or even a total curriculum. Achievement tests should be limited to particular material addressed in a curriculum within a particular time frame and should be offered after a course has focused on the objectives in question. There’s a fine line of differences between a diagnostic test and an achievement test.  Achievement tests analyze the extent to which students have acquired language features that have already been taught. (Geçmişin analizini yapıyor.)  Diagnostic tests should elicit information on what students need to work on in the future. (Gelecek ile ilgili bir analiz yapılıyor.)

www.yesdil.com

-22-

YESDİL – Eskişehir’in İÖLP Markası 





The primary role of an achievement test is to determine whether course objectives have been met – and appropriate knowledge and skills acquired – by the end of a period of instruction. Achievement tests are often summative because they are administered at the end of a unit or term of study. But effective achievement tests can serve as useful washback by showing the errors of students and helping them analyse their weaknesses and strengths. (Tam bir washback örneği.) Achievement tests range from five- or ten-minute quizzes to three-hour final examinations, with an almost infinite variety of item types and formats.

IMPORTANT: • New and innovative testing formats take a lot of effort to design and a long time to refine through trial and error. Traditional testing techniques can, with a little creativity, conform to the spirit of an interactive, communicative language curriculum. • Your best tack(yol, gidiş) as a new teacher is to work within the guidelines of accepted, known, traditional testing techniques. • Slowly, with experience, you can get bolder in your attempts. In that spirit, then, let us consider some practical steps in constructing classroom tests: A) Assessing Clear, Unambiguous Objectives Before giving a test; examine the objectives for the unit you’re testing. Your first task in designing a test, then, is to determine appropriate objectives. (Objective olarak: “Tag questions” ya da “Students will learn tag questions.” eksiktir. Çünkü “testable” değildir. Örneğin; spoken olarak mı, writing olarak mı öğrenecekleri belli değil. Dahası context olarak conversation’ da mı, essay’ de mi, academic lecture’ da mı olacağı belli değil.) Olması gereken objective: “Students will recognize and produce tag questions, with the correct grammatical form and final intonation pattern, in simple social conversations.” For more see the original book pg. 50 B) Drawing Up Test Specifications (Talimatlar) Test specifications will simply comprise a) a broad outline of the test b) what skills you will test c) what the items will look like This is an example for test specifications based on the objective stated above: “Students will recognize and produce tag questions, with the correct grammatical form and final intonation pattern, in simple social conversations.” Test specifications Speaking (5 minutes per person, previous day) Format: oral interview, T and S Task: T asks questions to S Listening (10 minutes) Format: T makes audiotape in advance, with one other voice on it Tasks: a. 5 minimal pair items, multiple choice b. 5 interpretation items, multiple choice

www.yesdil.com

-23-

YESDİL – Eskişehir’in İÖLP Markası Reading (10 minutes) Format: cloze test items (10 total) in a story line Tasks: fill in the blanks Writing (10 minutes) Format: prompt for a topic: why I liked/didn’t like a recent TV sitcom Task: writing a short opinion paragraph

These informal classroom-oriented specifications give you an indication of  the topics(objectives) you will recover  the implied elicitation and response formats for items  the number of items in each section  the time to be allocated for each C) Devising Test Tasks As you devise your test items, consider such factors as  how students will perceive them(face validity)  the extent to which authentic language and contexts are present  potential difficulty caused by cultural schemata In revising your draft, you should ask yourself some important questions: 1. 2. 3. 4. 5.

Are the directions to each section absolutely clear? Is there an example item for each section? Does each item measure a specified objective? Is each item stated in clear, simple language? Does each multiple choice have appropriate distractors; that is, are the wrong items clearly wrong and yet sufficiently “alluring” that they aren’t ridiculously easy? 6. Is the difficulty of each item appropriate for your students? 7. Is the language of each item sufficiently authentic? 8. Do the sum of the items and the test as a whole adequately reflect the learning objectives? In the final revision of your test,  imagine that you are a student taking the test  go through each set of directions and all items slowly and deliberately. Time yourself  if the test should be shortened or lengthened, make the necessary adjustments  make sure your test is neat and uncluttered on the page, reflecting all the care and precision you have put into its construction  if there is an audio component, make sure that the script is clear, that your voice and any other voices are clear, and that the equipment is in working order before starting the test.

D) Designing Multiple-Choice Test Items There’re a number of weaknesses in multiple-choice items:  The technique tests only recognition knowledge.  Guessing may have a considerable effect on test scores.  The technique severely restricts what can be tested.  It is very difficult to write successful items.  Washback may be harmful. (Nasıl olsa cevabı tahmin ederim. Atsam bile tutar gibi düşüncelerle negatif bir washback olabilir.)  Cheating may be facilitated.

www.yesdil.com

-24-

YESDİL – Eskişehir’in İÖLP Markası However, The two principles that stand out in support of multiple-choice formats are, of course, practicality and reliability. Some important jargons in Multiple-Choice Items: 





Multiple-choice items are all receptive, or selective, that is, the test-taker chooses from a set of responses rather than creating a response. Other receptive item types include true-false questions and matching lists. Every multiple-choice item has a stem(soru kökü), which presents several options(şıklar/ usually between three and five) or alternatives to choose from. One of those options, the key (doğru cevap), is the correct response, while the others serve as distractors (çeldirici).

IMPORTANT!!! Consider the following four guidelines for designing multiple-choice items for both classroom-based and large-scale situations: 1. Design each item to measure a specific objective. (Örneğin; aynı anda hem modal bilgisini hem de article bilgisini ölçme.) see pg. 56 2. State both stem and options as simply and directly as possible. Do not use superfluous (lüzumsuz) words, and another rule of succinctness (az ve öz) is to remove needless redundancy (gereksiz bilgi) from your options. see pg. 57 3. Make certain that the intended answer is clearly the only correct one. Eliminating unintended possible answers is often the most difficult problem of designing multiple-choice items. With only a minimum of context in each stem, a wide variety of responses may be perceived as correct. 4. Use item indices (indeksler) to accept, discard, or revise items: The appropriate selection and arrangement of suitable multiple-choice items on a test can best be accomplished by measuring items against three indices: a) item facility(IF), or item difficulty b) item discrimination (ID), or item differentiation, and c) distractor analysis a) Item facility (IF) is the extent to which an item is easy or difficult for the proposed group of test-takers. (Çok çok kolay ya da çok çok zor olan sorular altın ve kömür değerindeki öğrencileri birbirinden ayırt edebilmemize hizmet etmez. Bu yüzden bu “item facility” önemli bir parametre olarak karşımıza çıkmaktadır.) 20 öğrenciden 13 doğru cevap geldiyse; 13/20=0,65(%65) Oranın ne olması gerektiği hakkında kesin bir bilgi olmamasına rağmen %15 - %85’in kabul edilebilir oranlar olduğu söylenebilir. Note: Two good reasons for occasionally including a very easy item (%85 or higher) are to build in some affective feelings of “success” among lower-ability students and to serve as warm-up items. And very difficult items can provide a challenge to the highestability students.

www.yesdil.com

-25-

YESDİL – Eskişehir’in İÖLP Markası b) Item discrimination (ID) is the extent to which an item differentiates between high- and low-ability test-takers.  An item on which high-ability students and low-ability students score equally well would have poor ID because it did not discriminate between the two groups.  An item that garners(toplamak) correct responses from most of the high-ability group and incorrect responses from most of the low-ability group has good discrimination power. 30 öğrenciyi en iyiden en düşüğe kadar üç eşit parçaya ayır. En yüksek notu alan 10 öğrenci ile en düşük notu alan 10 öğrenciyi bir item’da aşağıdaki gibi ayırdığımızı farz edelim: Item # High-ability students (top 10) Low-ability students (bottom10)

Correct 7 2

Incorrect 3 8

ID: 7-2=5/ 10= 0,50 → The result tells us that us that the item has a moderate level of ID. High discriminating level would approach 1.0 and no discriminating power at all would be zero.  In most cases, you would want to discard an item that scored near zero.  As with IF, no absolute rule governs the establishment of acceptable and unacceptable ID indices. c) Distractor efficiency (DE) is the extent to which  the distractors “lure” a sufficient number of test-takers, especially lower-ability ones, and  those responses are somewhat evenly distributed across all distractors. Example: Choices High-ability students (10) Low-ability students (10)

A 0 3

B 1 5

C* 7 2

D 0 0

E 2 0

*Note: C is the correct response. The item might be improved in two ways: a) Distractor D doesn’t fool anyone. Therefore it probably has no utility. A revision might provide a distractor that actually attracts a response or two. b) Distractor E attracts more responses (2) from the high-ability group than the low-ability group (0). Why are good students choosing this one? Perhaps it includes a subtle reference that entices the high group but is “over the head” of the low group, and therefore the latter students don’t even consider it. The other two distractor (A and B) seem to be fulfilling their function of attracting some attention from the lower-ability students.

www.yesdil.com

-26-

YESDİL – Eskişehir’in İÖLP Markası SCORING, GRADING AND GIVING FEEDBACK A) Scoring As you design a classroom test, you must consider how the test will be scored and graded. Your scoring plan reflects the relative weight that you place on each section and items in each section. (Lesson objective hangi beceriyi daha çok önemsemişse o beceriye daha fazla puan vermek gerekir.) Listening ve speaking’ e yoğunlaşan reading ve writing’ e daha az önemseyen bir lesson objective ya da curriculum varsa puan dağılımı şöyle olabilir: Oral production %30, Listening %30, Reading %20 ve Writing %20 şeklinde. B) Grading Grading doesn’t mean just giving “A” for 90-100, and a “B” for 80-89. It’s not that simple. How you assign letter grades to a test is a product of     

the country, culture, and context of the English classroom, institutional expectations (most of them unwritten), explicit and implicit definitions of grades that you have set forth, the relationship you have established with the class, and student expectations that have been engendered(cause) in previous tests and quizzes in the class.

C) Giving Feedback Feedback should become beneficial washback. Those are some examples of feedback: 1. 2. 3. 4.

5.

6.

7. 8. 9.   

a letter grade a total score four subscores (speaking, listening, reading, writing) for the listening and reading sections a. an indication of correct/incorrect responses b. marginal comments for the oral interview a. scores for each element being rated b. a checklist of areas needing work c. oral feedback after the interview d. a post-interview conference to go over the results on the essay a. scores for each element being rated b. a checklist of areas needing work c. marginal and end-of-essay comments, suggestions d. a post-test conference to go over work e. a self-assessment on all or selected parts of the test, peer checking of results a whole-class discussion of results of the test individual conferences with each student to review the whole test Options 1 and 2 give virtually no feedback. The feedback they present does not become washback. Option 3 gives a student a chance to see the relative strength of each skill area and so becomes minimally useful. Options 4, 5, and 6 represent the kind of response a teacher can give that approaches maximum feedback. - END OF CHAPTER 3 -

www.yesdil.com

-27-

YESDİL – Eskişehir’in İÖLP Markası

CHAPTER

3

EXERCISE 1: Decide whether the following statements are TRUE or FALSE. 1. A language aptitude test measures a learner’s future success in learning a foreign language. 2. Language aptitude tests are very common today. 3. A proficiency test is limited to a particular course or curriculum. 4. The aim of a placement test is to place a student into particular level. 5. Placement tests have many varieties. 6. Any placement test can be used at a particular teaching program. 7. Achievement tests are related to classroom lessons, units, or curriculum. 8. A five-minute quiz can be an achievement test. 9. The first task in designing a test is to determine test specification.

EXERCISE 2: Decide whether the following statements are TRUE or FALSE. 1. 2. 3. 4. 5. 6.

It is very easy to develop multiple-choice tests. Multiple-choice tests are practical but not reliable. Multiple-choice tests are time-saving in terms of scoring and grading. Multiple-choice items are receptive. Each multiple-choice item in a test should measure a specific objective. The stem of a multiple-choice item should be as long as possible in order to help students to understand the context. 7. If the Item Facility value of a multiple-choice item is .10(% 10), it means the item is very easy. 8. Item discrimination index differentiates between high and low-ability students.

ANSWERE KEY EXERCISE 1: 1. TRUE 2. FALSE 3. FALSE 4. TRUE 5. TRUE 6. FALSE (Not all placement tests suit to every teaching program.) 7. TRUE 8. FALSE 9. FALSE (The first task is to determine appropriate objectives.) EXERCISE 2: 1. 2. 3. 4. 5. 6. 7. 8.

FALSE FALSE TRUE TRUE TRUE FALSE FALSE TRUE

(It seems easy, but is not very easy.) (They can be both practical and reliable.)

(It should be short and to the point.) (An item with an IF value of .10 is a very difficult one.)

www.yesdil.com

-28-