psychometric tradition

Bambang Ruby S, NIM: 0203513037, Smester 1 PPs Unnes Pendidikan Bahasa Inggris (Kelas Khusus) 2013/2014

HANDOUT 4 Research in English Language Teaching (The Psychometric Tradition)

Activity 1 What are variables, samples, and populations, and why are they important in research? Variable

: Nunan (1992) stated that a variable, as the term itself suggests, is anything which does not remain constant. In our case, it includes language proficiency, aptitude, motivation, and so on.

Population

: Saleh (2012) stated that population is the group of people or objects that the researcher wants to know about by conducting the study.

Sample

: Saleh (2012) stated that the sample is representative of the population defined.

They are important components of research because of the significant impact that they can have on the quality of the results/findings. The variables used as main resources while population and sample used to know the influence of variables. What are the basic principles of sound experimental design? The basic principles of experimental designs are randomization, replication and local control. Each of them is described in the following subsections. (1) Randomization. The first principle of an experimental design is randomization, which is a random process of assigning treatments to the experimental units. The random process implies that every possible allotment of treatments has the same probability. An experimental unit is the smallest division of the experimental material and a treatment means an experimental condition whose effect is to be measured and compared. The purpose of randomization is to

remove bias and other sources of extraneous variation, which are not controllable. Another advantage of randomization (accompanied by replication) is that it forms the basis of any valid statistical test. Hence the treatments must be assigned at random to the experimental units. Randomization is usually done by drawing numbered cards from a well-shuffled pack of cards, or by drawing numbered balls from a well-shaken container or by using tables of random numbers. (2) Replication. The second principle of an experimental design is replication; which is a repetition of the basic experiment. In other words, it is a complete run for all the treatments to be tested in the experiment. In all experiments, some variation is introduced because of the fact that the experimental units such as individuals or plots of land in agricultural experiments cannot be physically identical. This type of variation can be removed by using a number of experimental units. We therefore perform the experiment more than once, i.e., we repeat the basic experiment. An individual repetition is called a replicate. The number, the shape and the size of replicates depend upon the nature of the experimental material. A replication is used to: (i) Secure more accurate estimate of the experimental error, a term which represents the differences that would be observed if the same treatments were applied several times to the same experimental units; (ii) Decrease the experimental error and thereby to increase precision, which is a measure of the variability of the experimental error; and (iii) Obtain more precise estimate of the mean effect of a treatment, since , where denotes the number of replications. (3) Local Control. It has been observed that all extraneous sources of variation are not removed by randomization and replication. This necessitates a refinement in the experimental technique. In other words, we need to choose a design in such a manner that all extraneous sources of variation are brought under control. For this purpose, we make use of local control, a term referring to the amount of balancing, blocking and grouping of the experimental units. Balancing means that the treatments should he assigned to the experimental units in such a way that the result is a balanced arrangement of the treatments. The main purpose of the principle of local control is to increase the efficiency of an experimental design by decreasing the experimental error. The point to remember here is that the term local control should not be confused with the word control. The word control in experimental design is used for a treatment. Which does not receive any treatment but we need to find out the effectiveness of other treatments through comparison.

What do we mean by inferential statistics? Inferential statistics are data which are used to make generalizations about a population based on a sample. They rely on the use of a random sampling technique designed to ensure that a sample is representative. The first type of inferential statistic is a t-test. A t-test is used to compare the average scores between two different groups in a study to see if the groups are different from each other. The second basic type of inferential statistics is called an analysis of variance or ANOVA. An analysis of variance is a test that compares the average scores between three or more different groups in a study to see if the groups are different from each other. In other words, an ANOVA is exactly the same as a t-test, but it can analyze multiple groups at once. For example, a teacher believes that different teaching styles result in different scores when children take a test over the material. He might try lecturing for one group of students, versus worksheets with a second group of students. He then gives everyone the same test, and wants to compare the results. If he only had these two groups, he would use a t-test to compare the scores. However, if he wanted to add a third teaching style, which was having the students learn the material on their own and then teach it to each other. If he now wants to compare all three teaching styles to each other, he would use an analysis of variance, or an ANOVA test. When is it appropriate to use the following statistical procedures: t-test analysis variance, correlation, Chi-square? T-test: when researcher need to compare the average scores between two different groups in a study to see if the groups are different from each other. Analysis of variance (ANOVA): when researcher need to compares the average scores between three or more different groups in a study to see if the groups are different from each other. Correlation: when researcher need to determine relationship between two variables. Chi-square: when researcher need to compares observed frequencies to expected frequencies.

What is the difference between true experiments, quasi-experiments, and freeexperiments? True experiments: Tuckman (1978) stated that it provides completely adequate controls for all sources of internal invalidity. They represent no compromise between experimental design requirements and the nature and reality of the situation in which a study is being undertaken. Quasi-experiments: Tuckman (1978) stated that quasi experimental design exist for situations in which complete experimental control is difficult or impossible. Free-experiments: Tuckman (1978) stated that free experimental designs do not qualify as legitimate experimental designs because they do not control adequately against the sources of internal invalidity.

Activity 2 The Two Ways of Classifying Variables Nunan (1992) stated that a variable, as the term itself suggests, is anything which does not remain constant. In our case, it includes language proficiency, aptitude, motivation, and so on. In such a case, it is customary to distinguish between the two variables by giving them different labels. The label given to the variable that experimenter expects to influence the other is called the independent variable. The variable upon which the independent variable is acting is called the dependent variable. Based on the explanation above, it is clearly known that variables in English language teaching and learning divide into two categories. First, independent variable that is teaching method. Secondly, dependent variable that is the result of the students’ achievement. The kind of situation requires experiment as an appropriate way of gathering data The teacher found the new innovative method in teaching and learning English. He/she has used it and made development to his/her students. The teacher want to persuade his/her new method is better than traditional method. So, he/she should collect evidences from the teaching method and it correlation with the students’ development.

The difficulties and solutions to conduct experiment in a school setting Nunan (1992) argued that while your research design is becoming more rigorous, it is still no rigorous enough to allow you to claim that there is a casual relationship between the independent variable and the dependent variable. There is always possibility that some factor other than the experimental materials has brought about the observed differences in the scores. From the explanation above, we know that there is some factor apart from dependent and independent variables. We can’t conclude the result only from the correlation of both variables, but there is possibility we should observed some factor which make our research acceptable. Nunan (1992) stated that in order to appreciate the logic behind the procedures, one must be familiar with the following statistical concepts: Mean, standard deviation, normal distribution, and standard error. The explanation above shows us to know the solution to conduct the experiment in a school. We should follow the statistical concepts logically, those are: Mean, standard deviation, normal distribution, and standard error. After we done the concepts we could reasonably claim that our research is in true experiment. The important of standard deviation for studying numerical data Nunan (1992) stated that the standard deviation on the other hand, is the most important measure of dispersion, giving us information on the extent to which a set of scores varies in relation to the mean. Based on the explanation, Standard deviation helps researcher in measuring the variability of a mean. It used in evaluating values in records set to the mean and measuring of dispersion. For example, we have two classes whose mean reading scores are the same. With only that information, we would be inclined to teach the two classes in the same way. In our research, for example, we have many students throughout the entire range of performance in the first class. In this situation we will need to have teaching strategies more challenged. But in the second class they show low Standard deviation, it means we don't have any challenged students. They're all average, and our teaching strategy will be entirely different.

ANOVA as the best technique of data analysis Nunan (1992) stated that when comparing more than two means, or more than two groups, the appropriate test is the f-test, which is based on a procedure called analysis of variance (ANOVA). The ability of ANOVA is to test the significance of interactions between different variables. Based on the explanation above, ANOVA can test more than one treatment is a major advantage over other statistical analysis such as the t-test, it opens up many testing capabilities. ANOVA’s use an F-ratio as its significance statistic which is variance because it is impossible to calculate the sample means difference with more than two samples. Activity 3 Intact- group comparison

Improved control for history but weak on external validity

Pretest – posttest control group design

does not control for history or selection

Posttest – only control group design

True design with no testing effect

One- shot case study

does not control for selection

Equivalent time-samples design

True design with possible testing effect

Separate-sample pretest posttest

one – group design repeated twice

Nonequivalent control group design

Imperfect control of selection but better than nondesign

Posttest-only control group design

true design with no pretest bias

Time-series design

inadequate control of history

One-group pretest-posttest design

imperfect control of history

Factorial design

true design for dealing with multiple independent variables

Patched-up design

combination of two non design

Three design in terms of controlling for history bias Most adequate

pretest-posttest control group design

Next most adequate

time-series design

Least adequate

one-group pretest-posttest design

Four design in terms of controlling for selection bias Most adequate

posttest-only control group design

Next most adequate

patched-up design

Next Least adequate

nonequivalent control group design

Least adequate

intact-group comparison

Prediction: student teachers who are randomly assigned to urban schools for experience are more likely to choose urban schools for their first teaching assignment than student teachers who are randomly assigned to nonurban schools. Construct an experimental design to test this prediction. Based on the prediction above we should be better to use quasi experimental design because the situations in which complete experimental control is difficult or impossible. There are times when a comparison or control group cannot be included in an experiment. One kind of quasi experimental design that suitable for this case is time series design. It can be diagramed as follows: O1O2O3O4 X O5O6O7O8

Prediction: students given programmed math instruction will gain more in math achievement than students not given this instruction, but this effect will be more pronounced among high math aptitude students than among low. Construct an experimental design to test this prediction. Based on the prediction above it can be concluded that the best design is Nonequivalent Control Group Design. There is possibility of bias because there is not enough randomized of Ss. This design is identical to the previous one in all

respects except for the random assignment of subjects to conditions. It can be diagramed below: O1

X

O2

----------------------O3

O4

The procedures of this design are the same as for a true design except that intact groups rather than randomly assigned ones are used, creating a control problem in terms of selection bias. It has ability in controlling of selection of bias, especially by using a pretest. Which of the following circumstances necessitate the use of a quasiexperimental design? Experimenter cannot assign Ss to condition A patched up design has been created where this years’ first graders serve as the control group for a treatment being tried on this years’ second graders. Which validity threat is not controlled? Maturation and History Prediction: student teachers who choose urban schools for experience are more likely to choose urban schools for their first teaching assignment than student teachers who choose nonurban schools. Based on the prediction above we should be better to use quasi experimental design because the situations in which complete experimental control is difficult or impossible. There are times when a comparison or control group cannot be included in an experiment. One kind of quasi experimental design that suitable for this case is time series design. It can be diagramed as follows: O1O2O3O4 X O5O6O7O8 A school decides to implement a dental hygiene program for all its students. It predicts that cavities will be reduced as a result of this program. a. Why must a quasi-experimental design be employed to test this prediction? Because there is only a single group is available for study and the group pattern of the experience with the treatment is highly predetermined- that is,

the researcher must expose the group to the treatment on some systematic basis. b. construct one The design used for the prediction above is Equivalent Time – samples Design. It is diagramed below X1

O1

X0

O2

X1

O3

X0

O4

The school takes the dental hygiene programme regularly for the week(X 1). The following week the teacher checks student’s teeth (O 1). Then the teacher ask a dentist to check again student’s teeth (X0) followed by the teacher’s checking on the next week (O2). And it is done simultanously until (O4). The analysis of the data in this study is set up as shown above. A comparison of O1,O2,O3 and O4 allows the teacher to compare the experiences. The interaction between the four measurements provides a check on differential changes overtime, Prediction : children from broken homes will create a greater discipline problem in school ( as evidenced by demerits) than children from intact homes. a. Why must a criterion-group design be employed to test this prediction? We must use criterion- group design because the context of the prediction is suitable for the requirements of this design. An attempt is made to determine what characteristics are associated with the criterion group and have presumably preceded and thereby caused the criterion behavior. Illustrated by the directive versus nondirective teacher study, the criterion group approach might better have called a naturalistic study. b. construct one The criterion group design can be diagramed as follows. C

O1

O1

C

O2

C

O1

Prediction : an after-school dance program will improve the physical skills and social skills of first graders. a. why does the testing of this prediction call for Hawthorne control? The prediction above calls for Hawthorne control because the experimental intervention were after-school dance program for first graders and dependent variables are physical skills and social skills. Then Hawthorne intervention might take the form of technique in teaching dance to the children during the same period of the time that the experimental group was experiencing it. The nontreatment control condition, on the other hand, would involve no contact whatever between experimenter and subjects. b. Construct a design for testing it.

X H Ep En

R

O

Ep

O1

R

X

En

O2

R

H

Ep

O3

R

H

En

O4

Experimental (relevant) treatment. Hawthorne control (irrelevant experience) Positive teacher expectation Neutral teacher expectation created.

A researcher has just designed a special program to increase verbal I.Q. It is a series of classroom lesson. She wants to try it out in some schools. a. Why would a Hawthorne control be a good idea? A Hawthorne control is a good idea because it represents a systematic intervention and interaction on the part of experimenter with the subject; the purpose is to introduce a new procedure that is not anticipated to have specific effects related to the effects of the treatment or intervention being evaluated. In Hawthorne control experiment an irrelevant, unrelated intervention is deliberately introduced in order to create the Hawthorne effect which is often associated with intervention.

b. Why would teacher expectancy controls be a good idea? Teacher expectancy controls be a good idea because the teacher would believe that the experimental innovation would be successful. The outcome then would be a combination of the treatment plus the teacher’s expectation to success. c. Construct a design to test this program R

O

Ep

O1

R

X

En

O2

R

H

Ep

O3

R

H

En

O4

X

Experimental (relevant) treatment.

H

Hawthorne control (irrelevant experience)

Ep

Positive teacher expectation

En

Neutral teacher expectation created.