Approaches to Materials Analysis

APPROACHES to MATERIALS ANALYSIS MATERIALS ANALYSIS & PRODUCTION - UNIT 1 Sue Wharton

Goals In this unit we will consider different frameworks for the analysis of materials as text, in order to compare and contrast their emphases. We will consider the case for a standardised approach to materials evaluation. We will look at purposes of, and criteria for, public evaluations of teaching materials. Finally, we will begin to discuss the evaluation of materials in use. By the end of this unit you should be able to: o identify the rationale behind various analytical frameworks; o devise appropriate criteria for materials evaluation in your professional context; o undertake focused evaluations; o present the results of evaluation to different audiences. Introduction In this unit we will look at some texts which aim to suggest, to the teacher, ways of going about the task of materials analysis. When reading a group of such texts together, I think one notices two things: differences of emphasis as to content, and certain broad similarities as to approach. So by considering the differences and similarities together, we will get a sense of an evolving "paradigm" for materials evaluation in our field. In addition to this, I hope that each individual reader will get a sense of their own position vis a vis the ideas presented. A list of the three main texts which will be discussed in this unit can be found with the list of references. You could read them before, during or after the main body of the unit - I would be interested to hear how your choice affects your interaction with the ideas in the main text. The tasks for this unit are intended to encourage you to consider the basis for materials analysis, to critically evaluate some approaches to analysis, and to conduct your own materials evaluations. Reflection tasks are best done before you start work on the unit. Analysis tasks should be done after reading at least the main text of the unit. They appear at the point in the text where they are most relevant, but you don't need to stop and do them before moving on. Research tasks appear at the end of the unit. They will take a while to organise: so think about them well in advance! Let us now begin by discussing some of the different emphases that can be found in texts about materials analysis. I have chosen three perspectives and in each case will suggest that the perspective leads the writer to advocate a particular style of materials analysis. Reflection task 1 Do you feel that teacher-produced materials have any inherent advantages over published materials? Or vice versa? List these. When you have thought about this you may like to read Block (1991). Reflection task 2 Do you feel that the ELT profession needs some sort of standardised approach to materials analysis? Think of at least three advantages and disadvantages of such an approach. Differences of emphasis Focus on language A perspective which emphasises language is likely to examine both the linguistic exposure provided by materials as input, and also the output that they might naturally generate. Thus, questions might be suggested about the source of the language presented: is it "authentic"? Is it based on any sort of corpus? Is there a balance between spontaneous and prepared language, drawn from spoken and written sources? The analysis may make use of certain well-known distinctions, such as that between form and function, to discuss the overall approach to language in materials. It may consider different aspects of language, like vocabulary, phonology, or discourse, in an attempt to discover where materials' main linguistic priorities lie. It may suggest questions about the weighting given to particular areas of language, such as verb group/ noun group/ lexis/ collocation, etc.

A framework may also propose an examination of the linguistic skills covered in materials: this, of course, would add a dynamic dimension to the static view of language we have just mentioned. In focusing on skills it would underline the importance of evaluating potential linguistic output, and guide the analyst to look at potential interactions between materials and learners, learners and teachers, learners and other learners, and learners and the outside world. This kind of analysis attempts to uncover the suppositions about language in teaching materials. But of course, it itself will also be based on certain assumptions about language. For example: if questions are asked about the selection and grading of language items, then there may be an assumption that language can be divided into units like structures or functions, and that some of these are inherently more difficult than others. An example of an analysis with a significant language focus is Cunningsworth (1995) chapters 4, 5 and 6. The questions raised in these chapters indicate a clear concern with the language as a system, and a great many aspects of language are discussed. In all sections the author emphasises the need to analyse the language of the materials in terms of the learners' target situation, rather than as an end in itself. The linguistic criteria for analysis discussed by Cunningsworth may or may not be appropriate to guide materials analysis in different professional contexts. It is our task, as readers of texts like this, to look critically at the approach and to modify it as appropriate before implementing it. Focus on classroom activities A perspective which concentrates on classroom activities combines an emphasis on conditions of learning with an emphasis on process aspects of language. It is often suggested that the success of the learning experience will be influenced by the extent to which learners are able to exercise choice and initiative in the classroom, the extent to which they are able to use language to express real meanings, and the extent to which they perceive classroom activities as relevant. It is also often suggested that knowing a language does not only mean knowing its elements, but also means using it for a real purpose, and interacting with other language users in different situations via different channels. A focus on classroom activities in materials allows us to examine language and learning together. It is easy to suggest that classroom activities are not determined by materials, and this is of course true in the literal sense. However, it is also true that materials carry within them strong suggestions about roles of learners and teachers and roles of learners vis a vis each other. They also contain expectations about how learners should interact with texts within the materials and with the materials as a text. It is therefore appropriate to analyse materials from the perspective of the types of classroom activity which they might stimulate when used as the writer intended. An example of such an approach is Candlin & Breen (1987), who propose a series of questions as a framework for analysis. In phase 1 section 2 of their questionnaire, they seek to engage theories of learning as a tool for understanding materials, though we note that the choice of which theory is seen as a matter for the individual analyst. Phase 1 section 3 encourages the teacher/analyst first to explicate their own views on appropriate teacher roles, and then to match this view with the position of the materials under analysis. Phase 2 section 2 again advocates a matching process, this time between the materials and the teacher/analyst's view of their group's learning styles. Candlin and Breen's emphasis on the particular situation of the analyst, their preference for open-ended questions and their desire to engage the personal values of the teacher/analyst, may at first lead us to think that their framework does not imply its own view about the appropriacy of particular classroom activities. But if we look more closely we will see that this is not the case: there is an emphasis on flexibility, teacher and learner choice and indeed learner centredness, which reflects the values of the writers and perhaps the teaching contexts with which they are most familiar. So this approach to analysis, like Cunningsworth's above, is not a tool that can be used in any circumstances without being subjected to prior critical analysis. Focus on the target speech community This approach to materials analysis has a great deal in common with the "target situation" approach to needs analysis: both rest on the assumption that it is possible to arrive at a sociolinguistic profile of the learner's future language use, and from there to develop a profile of their present learning needs. Such approaches will ask questions, then, about the language-using communities that the learner wishes to join and what their roles and purposes within that community are likely to be. Such information can then be the basis of materials analysis, as tasks in materials are examined in terms of their linguistic and pragmatic "authenticity" vis a vis the target speech community. This approach to both needs analysis and materials analysis is most commonly associated with ESP, but it can also be valid in EGP where many learners have no clear goal in learning the language. This is

because a class of learners in itself constitutes a speech community, whose members have authentic roles and purposes. The more a speech community like this allows and encourages learners to use language authentically: eg to create their own meanings and to initiate and control interaction patterns: the more likely it is to constitute an effective "training ground" for whatever TSCs the learners later decide they want to join. The communicative tasks of a language-learning speech community have their own objectives, and these may be seen as stepping-stones, or enabling objectives, which will give the learner a foundation and enable them to later aspire to more difficult objectives, authentic for their chosen TSC. An example of a framework for analysis based on the concept of TSC is Roe's (1989) Task Analysis, printed here as appendix 1 at the end of this page. It is in the form of a hierarchy of possible enabling objectives, where level 1 represents a task which is fully authentic in the TSC and level 16 is not directly concerned with language use at all. As will be seen, this is an implicational hierarchy where each level also subsumes features of lower levels: so, for example, in participating in a role play (level 5) learners will also engage the objectives of understanding the dictionary meanings of lexical items (level 14) and reproducing the sounds of the language (level 15). The validity of this framework is not, of course, context-free. In particular we can see that the hierarchy assumes that the TSC will be concerned with language use and that it is this assumption that pushes tasks based on usage to the lower points on the scale. Whereas if we imagine that the learner's target is to pass a reading-aloud test of English, then a decontextualised read-aloud task could move from its present level 15 right up to level 4. So far, then, we have looked at three approaches to materials analysis. We have seen that all have their strengths but that none are intended for use without critical refinement on the part of the analyst. Similarities of approach One does not have to read very many articles or books on the subject of materials analysis to start getting a sense of deja vu. Many texts are organised around the principle of a very long set of questions to be asked of the materials under analysis. Two of the three texts discussed above fit this characterisation, as do Sheldon 1988, Hutchinson & Waters 1987 ch9, and McDonagh & Shaw 1993 ch 4. By the length of the lists of criteria, we can assume that differences of emphasis notwithstanding, each writer is aiming for comprehensiveness. Swales (1980) has criticised this tendency, arguing that the more questions one asks of a set of teaching materials, and the more sophisticated they are, the more likely one is to be disappointed. Many of the lists of criteria quoted above would be extremely time-consuming and unwieldy if used unadapted. Some writers (such as Sheldon) attempt to make results from their checklists easily interpretable by suggesting that answers to questions be expressed as points on a scale. Other writers prefer to use open-ended questions which perhaps encourage more interpretation on the part of the analyst. Frameworks for analysis lie on a continuum from open to closed, but do not seem to differ very much in terms of the demands placed on those who might attempt to use them. What, then, is the value of the checklist approach? The case is made by Sheldon (1988) in an article provided with this unit which culminates in the presentation of his own checklist, containing 53 questions organised under 17 major headings. Sheldon argues that systematic criteria for assessment are needed because of the very wide variety of ELT textbooks available: he strongly implies that many such textbooks are of inherently poor quality and so represent a trap for the unwary. In this situation, Sheldon laments the lack of an established approach to materials analysis in our field: he recognises that most teacher-training courses include a materials analysis element, but is unhappy with the "uneven quality" of these "evaluative tools" (p240). Sheldon is no happier with book reviews as guidelines for materials evaluation. He claims that reviews do not include enough practical information or explicit evaluation, that they vary considerably in quality, and that they only cover a small part of what is on the market. Sheldon then goes on to present his own checklist. He makes no claims of universal applicability, stating that any global checklist would require adaptation before being used in particular circumstances. Sheldon's article and checklist, then, would seem to encapsulate the similarities of current approaches to materials analysis: an attempt to arrive at comprehensive and stable criteria which sits, perhaps uneasily, alongside a recognition that "we can be committed only to checklists or scoring systems that we have had a hand in developing, and which have evolved from specific selection priorities" (p 242) Analysis task 1 What adaptations would you need to make to Sheldon's checklist before using it in your teaching situation?

Draft a revised version. Compare your version with that of someone else who teaches in a similar situation to you. Book reviews So far in this unit we have looked at texts which purport to give guidance for materials analysis. I now want to move on to consider book reviews and to examine the differences between these and teachergenerated analyses. A teacher or DOS might typically evaluate materials having a specific context in mind and with a view to purchase or use in that context. Evaluation like this is not intended to have relevance outside the context, or to represent a judgement about the materials as a product. A book reviewer, though, is asked to present an analysis which is informative and fair, while at the same time delivering a judgement on the quality of the book, addressed to unknown people in unknown contexts. To do this, reviewers tend to go back to principles valued in the discipline as a whole, to refer to the current consensus about materials. A close look at a book review can tell us as much about the reviewer's views on teaching and on language, than about the material reviewed. Dougill (1987) has attempted to summarise the criteria which book reviewers most typically use, and he provides a typical list of questions. Some of these relate to privileged concepts or distinctions in EFL: "Is the course linear or cyclical?" "What is the type of syllabus (structural/functional/multi-syllabus etc)" "Is there an inductive or a deductive approach?" At first sight, questions like this appear fairly neutral, implying that the reviewer will provide information, and the teacher will then make judgements as to suitability. But of course, matters are not so simple: if we make judgements on the basis of a review, then we are accepting the reviewer's analytical framework and this in itself places limits on the kinds of judgements we can make. Analysis task 2 In appendix 2 of your hard copy binder there are two book reviews, one from The EFL Gazette February 1997 and one from ELTJ 48/2. Notice the differences between them: What criteria are the reviewers using in each case? What are the goals of each reviewer? Assessing materials in use So far in this unit we have looked at evaluation outside the classroom. We have stressed the importance of context but we have not considered the evaluation of materials at the moment of use; nor have we looked specifically at the contribution learners can make to materials evaluation. Candlin & Breen (1987) discuss the importance of discovering learners' criteria for good materials: not only as part of the needs analysis that precedes the design or purchase of new materials, but also to facilitate the sensitive adaptation of existing materials. Candlin & Breen suggest that learners be asked to work with fairly open-ended questionnaires, designed to promote reflection both about the learners' own aims in language learning, and about the procedures for working in the classroom. Such a questionnaire is clearly a teaching tool in itself. McDonagh & Shaw (1993 ch13) discuss the merits of classroom observation for materials analysis. They discuss some basic techniques which can help the analyst to see how teachers and learners are actually using particular materials. One could look for example at the kinds of language use particular materials generate, or the ways in which teachers and learners adapt materials. There are clearly many advantages in observing someone else's class - freed from the pressure of making constant interactive decisions vis a vis materials as when actually teaching, the analyst can focus on the contribution that the materials themselves make to classroom processes. Dudley Evans & Bates (1987) discuss a long term, in situ materials evaluation project focused on the use of their book Nucleus General Science in Egyptian secondary schools. Their discussion concentrates on the research procedures used, which included teacher questionnaires and group discussions. They show how the research made it possible to identify the mismatches between the materials and the particular context of use, and then to adapt the materials so as to better suit local needs. Ellis (1997) draws a distinction between the"predictive" or "armchair" evaluation which he claims is usually discussed in the literature, and his own proposal which is for the retrospective, empirical evaluation of materials-in-use. This article (provided with the unit) thus illustrates an important direction of materials analysis at the present time. Ellis puts forward a systematic framework with which to conduct a retrospective evaluation. This centres around the evaluation of particular tasks found in teaching materials: so it is very focused, and does not

attempt to evaluate a whole teaching text. The article describes the steps of task evaluation in detail. The central decision to be made is, what are we evaluating FOR? Ellis suggests three possibilities: student based evaluations, concerned with student satisfaction from the tasks: response based evaluation, concerned with the interaction which results from the tasks; and learning based evaluation, which would aim to determine whether any measurable learning took place as a result of the tasks. The choice made has ramifications for the whole of the data-collection process. Ellis justifies his procedure not only as a method for generating contextually situated evaluations but also as a teacher development activity and a possible basis for an action research cycle. Teachers who use it, he suggests, are in control of the criteria for evaluation and of the interpretation of observed events: which is very different from "applying" someone else's criteria to materials-as-written-text. Analysis, choice and design In this unit we have seen that a number of decisions are to be made as one embarks on a materials evaluation project: decisions as to content, focus and purpose of the analysis, as well as decisions about which method(s) to use. The interplay between these factors is vividly portrayed in an article by Chambers (1997) which focuses on procedures, rather than criteria, for coursebook selection within a given institution. Chambers advocates that the whole teaching team be involved in the decision and describes a technique to make this possible. His premise is that all of us carry mental models for "good" materials but that explicating these at a specific level is difficult. We need a more focused tool than simple "discussion". He describes a decisionmaking technique taken from the literature of business. He claims that it permits the quantification (and therefore direct comparison) of factors involved in the decision, but also keeps professional judgement at the centre of the process. The technique is based on a pro-forma which encourages the identification of required and desirable criteria, and allows them to be weighted relative to each other. The proforma is completely empty in terms of the criteria themselves: these must be group generated. Chambers thus emphasises the importance of local criteria. The importance of choosing an appropriate focus for analysis is further underlined if we consider that analysis is frequently not an end in itself, but rather a basis for either the significant adaptation of existing materials or the design of new ones. It is the same criteria which underpin our analysis that will guide our production: in analysis we are matching an existing text with our beliefs about what such a text should be, and in production or adaptation we are attempting to actually translate our theoretical position, to realise it, in teaching materials. In both cases, the starting point is the exploration, questioning and refining of our own beliefs and criteria. This is of course a continuing process, and we hope that interaction with the ideas presented in this unit and with the literature referred to has contributed to the process for you. In the next two units, and with the aim that you will further sharpen your theoretical positions, we will look in more detail at the interactions between materials, teachers and learners. Research task 1 This exercise may take you a while to set up, but please do persevere - groups who have done it have found it rewarding. You may be able to do it with colleagues at school, or with fellow MSc participants. Its stages are as follows: - arrange yourselves in groups of three - ideally, with people who are in a broadly similar teaching situation. - choose a unit of a textbook familiar to you all, and agree on a particular type of learner and situation for whom you will analyse the unit - each person individually analyse the unit from one of the three perspectives discussed at the beginning of the unit -come back together and compare notes. To what extent does the focus of analysis affect your evaluation of the material? -ask yourselves how suitable or useful you found the different frameworks? What changes did you make to adapt them to your situation? Research task 2 As an ongoing project for the MAP component, I would like to suggest that you try to involve yourselves with the piloting and evaluation of materials which have yet to reach the commercial publication stage. Contact the Developmental Editor at a publisher who is active in your part of the world, and offer the services of you and your class!

Participation in piloting is an excellent opportunity to have input into the development of commercial materials. It also puts pressure on you to examine your own criteria for good materials, as you think about what kind of feedback you can most helpfully offer the authors. If you would like to, you can use your participation in piloting as a starting point for your MAP assignment Research task 3 Write a review of a coursebook for the LSU Bulletin. Unit 1 appendix 1 FRAMEWORK FOR TASK ANALYSIS by P.Roe LSU 1989 This framework is intended as a tool for the analysis of tasks in ELT materials - particularly materials which aim to be communicative. A quick glance at the results of a Task Analysis will show you what 'communicative' activities the learners are likely to engage in. It should be possible to assign most tasks to one or more levels on the scale given below. Level 1 There is little or no distinction between enabling and terminal objectives. Tasks are fully authentic and arise as a natural part of the activity of the TSC, possibly under the guidance of a 'mentor', 'knower' or more experienced member of the community, whose job it is to ensure the learners' full linguistic integration. Learning on the job. The TSC is itself the learning environment. You pick up the language as you go along. Felicity conditions are fully met. Level 2 The tasks are 'authentic', but the TSC which generates them is artificial, that is, the learners become the TSC where purposes/outcomes lie in the successful negotiation of agreed answers to tasks set by authority. The TSC is created for the purpose, and the outcomes to be negotiated by it are prescribed by the course-book writer, the teacher, the learners, or any combination of these. In its most basic form, this approach can be characterised by a task-setter who provides both the input data and an instruction/task outcome. The task dynamics can be modelled thus: 1st Cycle Teacher/Task-Setter/Evaluator: Data & Instruction Learner Outcome: Task Process Output Data Teacher's Linguistic Exploitation: Commentary on aspects of language arising during task completion 2nd Cycle Teacher/Task-Setter/Evaluator: Data & Instruction Learner Outcome: Task Process Output Data Teacher's Linguistic Exploitation: Commentary on aspects of language arising during task completion This approach is of course only as rich as the TSC which is called into being. It can be quite rich, but it could also prove restricted and stultifying. Its value as an enabling objective depends on what set of TSCs the learners will eventually need to join. To a certain extent, of course, all classroom teaching involves some element of "classroom management" and ipso facto authentic TSC communication in this sense. For example, the teacher instruction "Put these sentences into the past tense, and hand in your work for me to correct" must qualify as such, but hardly indicates the kind of TSC proposed above. Level 3 Tasks are simulations of normal TSC tasks executed in a normal TSC environment, with the learner in his/her target role and all other participants accepted by the learner as authentically filling their role in the TSC. That is, each task is a 'dummy run' for the benefit of the learner, where all variables, including

sanctions, have a counterpart in the 'real thing'. For example air traffic controllers in a sophisticated simulator, dress rehearsal for a play, 'mock' examinations, a pilot flying with a non-intervening instructor before going solo. As in Level 1, language constitutes only one dimension of difficulty in the task concerned, although other dimensions may be kept artificially low to highlight the linguistic component for pedagogic purposes. This enabling objective prepares the learner for the real thing. Felicity conditions remain reasonably intact. But consider the differences in sanction between the risk of a real and simulated crash, exam failure etc. Level 4 Tasks are simulations of authentic tasks, ie are structurally related to TSC tasks, but in an obviously 'classroom' environment, where the sanctions are language-teacher sanctions rather than those of the TSC. But the learner is only asked to fulfil his/her own target roles in the TSC. Felicity conditions are further weakened. Success at these tasks might be seen as an enabling objective for Levels 1 to 3. Level 5: Role-Play This level of enabling objective represents a further watering down of the authenticity of the task in that learners assume roles other than those they are aiming at in the TSC, or, in extreme cases, could ever be imagined as filling. (Eg an 18 year old male socialist being asked to take the role of Mrs Thatcher in a simulated Cabinet meeting.) Felicity conditions are further weakened (eg "I'll break your pencil if you say that again" might be accepted in the context of the classroom as a genuine threat, but hardly "If you vote against the government at tonight's division you will find yourself on the back benches in the morning.") Level 6 Tasks are dramatisations and role interpretations of scripted, or partially scripted, scenarios. Such tasks call many aspects of discourse strategy into play, especially the control of suprasegmental features of language, and have the virtue of 'feeding' highly contextualised language, but the outcome is one of 'rendition' rather than negotiation. Level 7 Tasks involve the learner not as participant in achieving the purposes of the TSC, but as auditor/reader/observer of text arising in the course of transactions of a TSC and are aimed at an "appreciation" of text (written or spoken). The learners' task essentially boils down to inferring the effect the text might produce in context. Such tasks can be performed on: o Authentic text, by which I mean text which was actually generated by a TSC in the actual negotiation of an outcome, eg a) A recording of a telephone caller arranging an appointment. b) A letter to the tax authorities explaining why the writer didn't declare any unearned income. c) Dickens' Novel: A Tale of Two Cities. o Non-authentic text, by which I mean text produced by teachers/course-book writers as course material for language learners and offered as if it were text produced by third parties, (ie persons outside the classroom situation) eg a) A recording by actors of the script of a telephone conversation written by a course-book author. b) An example of how one might write a letter to the tax authorities. c) An abridged and simplified edition of A Tale of Two Cities written for foreign learners to the 2000 word vocabulary level. Tasks of this nature (Level 7) often fall under the traditional heading of 'global comprehension', and typically involve inference, reading between the lines, and attention to discourse markers in written or spoken text. Level 8 At this level, tasks are aimed at text (spoken or written, authentic or non-authentic) seen not so much as communicative discourse, but as containers of factual/propositional information, and without reference to those who may have originated or received the document. Such tasks may be classed as traditional comprehension in which inference plays little or no part, and frequently involve restatement of information directly contained in the input text. Level 9 Tasks are aimed at the successful construction of clauses into sentences or analysis of sentences into clauses.

Level 10 Tasks are aimed at the construction of clauses by the application of the rules of syntax. Level 11 Tasks are aimed at an awareness of common phrasal patternings in the language. Level 12 Tasks focus on the relationship between the class of common utterances/phrases in the language and the class of possible effects they might be used to produce in others in real discourse. (That is, they focus on functional relationships). But it should be noted that this is not the same thing as actual functional practice which is possible only in the context of a TSC, most genuinely in a context such as those of Level 1, or more artificially Level 2, or, with less and less chance of felicity conditions being met, Levels 3 to 5 of my hierarchy of enabling objectives. Take an example I have witnessed. The teacher (who has written forms of apology on blackboard) takes A's hand, puts it onto B's coat, causing it to fall to the floor. T: "A, you have thrown B's coat on the floor. You shouldn't do that. Apologise to her at once." A: (Reads a form of apology from blackboard) "I am sorry I threw your coat on the floor." (Consider the extent to which the felicity conditions for an apology are met by A's utterances.) Level 13 Tasks in which the criteria of evaluation are morphological. Level 14 Tasks focusing on the meaning (dictionary definition) of words. Level 15 Tasks aimed at skill in reproducing the sounds of the language, both: a) Segmental b) Suprasegmental Level 16 The task consists of reading or listening to information about the target language (grammar rules, statements about what forms realise what functions, what lexical items realise what notions etc). Such information can of course be conveyed largely in the L1, thus effectively reducing the amount of classroom management language. It is meant to be digested and applied in the context of higher level tasks. References The three main texts referred to are: Cunningsworth A 1995: Choosing your coursebook London Heinemann Candlin C & Breen M 1987: "Which materials? A consumers' and designers' guide" in Sheldon L (ed) ELT Textbooks and materials: Problems in evaluation and development ELT Docs 126 London: Modern English Publications Roe P 1989: Framework for Task Analysis Aston University LSU Other references are: Chambers F 1997: "Seeking consensus in coursebook evaluation" ELTJ 51/1 29-35 Block D 1991: "Some thoughts on DIY materials design" ELTJ Vol 45/3 pp 211-217 Dudley Evans A & Bates M 1987: "The Evaluation of an ESP textbook" in Sheldon L (ed) ELT Textbooks and materials: Problems in evaluation and development ELT Docs 126 London: Modern English Publications Dougill J 1987 "Not so obvious" in Sheldon L (ed) ELT Textbooks and materials: Problems in evaluation and development ELT Docs 126 Modern English Publications Ellis R 1997: "The empirical evaluation of teaching materials" ELTJ 51/1: 36-42 Hutchinson T & Waters A 1987: English for Specific Purposes: A learning-centred approach Cambridge CUP

McDonagh J & Shaw C 1993: Materials and Methods in ELT Oxford Blackwell Sheldon L 1988: "Evaluating ELT textbooks and materials" ELTJ 42/4: 237-246 Swales J 1980: "ESP: The textbook problem" ESP Journal vol 1 pp11-23 Supplied reading Ellis R 1997: "The empirical evaluation of teaching materials" ELTJ 51/1: 36-42 Sheldon L 1988: "Evaluating ELT textbooks and materials" ELTJ 42/4 : 237-246