Adversary-Oriented Evaluation (2)

SGDC 5023 EVALUATION OF TEACHING GROUP 5

ADVERSARY-ORIENTED EVALUATION APPROACHES

1

Prepared By: ARWINA SYAZWANI BINTI GHAZALI 820403-11-5560

WAN ANISAH BINTI ABDUL KADIR 810522-11-5572

NOOR FAZLINA YATI BT MAT RAZALI 820208-11-5338

NORIHAN BT. ENDUT 791223-11-5408

ADVERSARY-ORIENTED EVALUATION APPROACHES

3

ADVERSARY-ORIENTED EVALUATION  Adversary-oriented evaluation is based on the judicial metaphor. It is assumed here that the potential for evaluation bias by a single evaluator cannot be ruled out, and, therefore, each “side” should have a separate evaluator to make their case.

ADVERSARY-ORIENTED EVALUATION Purposes of Evaluation:  Providing a balanced examination of all sides of controversial issues or highlighting both strengths and weaknesses of a program.

6

ADVERSARY-ORIENTED EVALUATION Distinguishing Characteristics: Use of public hearing, use of opposing points of view, decision based on arguments heard during  proceedings.

7

DEVELOPERS OF ADVERSARY-ORIENTED EVALUATION APPROACHES AND THEIR CONTRIBUTIONS

8

GUBA [1965]

OWENS [1971]

WOLF [1973]

HISCOX & OWENS [1975]

WOLF [1975]

STENZEL [1976]

LEVINE [1976] NORTHWEST REGIONAL EDUCATIONAL

DEVELOPERS OF ADVERSARY-ORIENTED EVALUATION APPROACHES AND THEIR CONTRIBUTIONS GUBA [1965]

Suggested that educational evaluation might use aspects of legal paradigm

OWEN [1971]

The first real adversary evaluation in education was conducted “modified judicial model”

WOLF [1975]

“judicial evaluation model” (JEM)

11

THREE GENERAL APPROACHES TO ADVERSARY-ORIENTED EVALUATION The Judicial Evaluation Model and other „Pro and Con” Adversary Hearing Adversary Hearing with More than Two Opposing Views

Adversary Debates and Other Forensic Structures

12

THE CHARACTERISTICS OF The Judicial Evaluation Model and other ‘Pro and Con” Adversary Hearing 1. Procedural rules must be flexible 2. There are no strict rules for the assessment of evidence. 3. The only requirement is that the judge(s) must determine  before hand whether evidence is admissible or not. 4. The parties may be asked before the hearing to present all relevant facts, pieces of evidence and names of witnesses/experts to the judges 5. A copy of the complaint must, before the public hearing takes  place, be committed to the judge(s) and the defence. The defence may plead guilty to some charges and deny others.

THE CHARACTERISTICS OF The Judicial Evaluation Model and other ‘Pro and Con” Adversary Hearing 6. Witnesses are able to speak freely and may be subjected to cross-examination. 7. Experts may be summoned for a statement before or during the hearing. 8. Meetings of all parties involved with the judge(s) prior to the  public hearing tend to soften the debate and can be conducive to a joint striving to get to the truth of the matter on the basis of relevant facts. 9. Besides the two parties involved, other stakeholders may also  be allowed to participate.

13

4 STAGES OF ADVERSARY-ORIENTED EVALUATION

Wolf (1979) and Thurston,propose the following four stages for adversary evaluation: THE ISSUE GENERATION STAGE THE ISSUE SELECTION STAGE

THE PREPARATION OF ARGUMENTS STAGE

THE HEARING STAGE ITSELF

STEPS OF ADVERSARY-ORIENTED EVALUATION THE ISSUE GENERATION STAGE

Should the program laid off and pick some new programs suitable alternative?

At this stage, a broad range of issues are identified. Thurston recommends that issues which reflect those perceived by a variety of persons involved in, or affected by the program in question, are taken under consideration in the preliminary stages.

STEPS OF ADVERSARY-ORIENTED EVALUATION THE ISSUE SELECTION STAGE

This stage consists of issue-reduction. Wolf (1979) proposes that issues on which there is no debate, should be eliminated. Thurston states that this reduction may involve extensive analysis (inclusive of content, logic and inference). The object of debate should also be defined and focused during this

STEPS OF ADVERSARY-ORIENTED EVALUATION THE PREPARATION OF ARGUMENTS STAGE PRO

CON

This stage consists of data collection, locating relevant documents and synthesising available information. The data or evidence collected should be relevant to the for and against arguments to be deployed in the hearing

STEPS OF ADVERSARY-ORIENTED EVALUATION THE HEARING STAGE ITSELF

This stage may also be referred to as the clarification forum and involves public presentation of the object of debate (Wolf, 1979). This is followed by the presentation of evidence and panel or jury deliberation.

THE CHARACTERISTICS OF Adversary Hearings With More Than Two Opposing View 1.

2.

Some of the review panels discussed in chapter 8 may hold  public hearings to collect information pertinent to their charge Appointed commission charged with the resolution of controversial issues frequently hold hearings to obtain evidence and opinions relevant to their mission.

THE CHARACTERISTICS OF Adversary Hearings With More Than Two Opposing View 3.

House (1980) has cited as one example the frequent use in England of commission and councils headed by prominent citizens to provide guidance to government policymakers.

4.

Several types of committee hearings are structured to identify and explore all the points of view represented in particular context.

THE CHARACTERISTICS OF Adversary Hearings With More Than Two Opposing View Although not adversarial in the strict sense of world because:

5.



 Smith (1985) has noted they explore a variety of positions.

 prefer to include them because:



a.

they reflect multiple viewpoints

 b.

they frequently use hearing processes, questioning, cross-examination, interaction concerning alternate viewpoints and summary statements of the various positions.

THE CHARACTERISTICS OF Adversary Hearings With More Than Two Opposing View St John (undated) referring to such hearings as the committee approach to evaluation:

6.









All of those with a stake of evaluation A public hearing with testimony, questioning, crossexamination, and summary statements The hearing method consists of public, verbal, face to face interaction. A process of communication and education occurs, and evaluation makes its impact as it is happening.

THE CHARACTERISTICS OF Adversary debates and other forensic structure 1.

Several approaches that qualify as adversary-oriented do not employ hearing processes.

2.

Kourilsky (1973) proposed that pro and con arguments be  presented to a decision maker. ( who would examine the evidence and question the presenter, ultimately arriving at the decision that seemed fair given both positions)

THE CHARACTERISTICS OF Adversary debates and other forensic structure 3. Kourilsky and baker (1976) described an adversary model in which: a. two teams prepared, respectively, affirmative and negative appraisals of that which was evaluated,  b. met to present the views to one another, c. Cross-examining and critiquing one another‟s continents on  prespecified criteria

d. Engaged in open ended discussions until reconciliation of views was attained and translated into written recommendations in a single report

THE CHARACTERISTICS OF Adversary debates and other forensic structure 4.

Levine (1974) proposed that a resident adversary or critic might be assigned to a research project to challenge each bit of information collected, searching for other plausible explanations.

5.

The Stake and Gjerde (1974) strategy of having two evaluators prepare separate reports summing up opposing  positions for and against the program- yet another variant adversarial approach that does not depend on hearing format.

THE CHARACTERISTICS OF Adversary debates and other forensic structure 6.

Donmoyer (undated) proposed “deliberatively” approach to evaluation which focused on assessing and balancing alternative conceptions of reality and the differing value  positions underlying these conceptions.

7.

 Nafziger and others (1977) described an adversary evaluation design employing a modified debate model of presenting data collected in a comprehensive evaluation to ensure that both sides of controversial issues were illuminated.

Clyne ( 1982 ) summarized :1

2

3

4

5

6

•

Summative evaluation

•

Formative evaluation

•

Social Science making

•

Policy analysis and debate

•

School governance and local decision making

•

Issue resolution and policy formation

THE STRENGTHS OF THE ADVERSARY- ORIENTED EVALUATION APPROACH

STRENGTHS OF ADVERSARY-ORIENTED EVALUATION 1. Due to the public nature of the evaluation, openness

and transparency regarding the object of evaluation is encouraged. 2.  As the model takes into account multiple forms of

data (inclusive of statistical fact, opinions, suppositions, values and perceptions), it is argued to do justice to the complex social reality which forms part of the evaluation. 3. The judicial nature of this approach may reduce

political controversy surrounding an object of

STRENGTHS OF ADVERSARY-ORIENTED EVALUATION 4.  As both sides of an argument are presented, the risks

of tactical with holding of information should be minimized. 5. This approach allows for the incorporation of a

multitude of perspectives, this should promote a more holistic evaluation. 6. The presentation of pro and con evidence and a

platform which allows for cross-examination, permits public access to various interpretations of the evidence introduced into the evaluative context .

STRENGTHS OF ADVERSARY-ORIENTED EVALUATION 7. The presentation of rival hypotheses and explanations

may enhance both quantitative and qualitative approaches. 8.  All data must be presented in an understandable and

logical way in order to persuade the jury. Dependent on the jury in question, this can make the data presented more accessible to the public and other stakeholders involved in the evaluation. 9. Finally, this approach is suitable for meta-evaluation

and may be combined with other approaches which are participatory or expertise- oriented.

The object of the evaluation affects many people Resources are available for additional expends required by adversarial strategies

Controversy about the object of the evaluation has created wide interest

SUMMARY Administrators understands the intensity

Decisions are summative

Clear issues are involved

Evaluators are external

THE LIMITATIONS OF THE ADVERSARYORIENTED EVALUATION APPROACH

THE LIMITATIONS OF ADVERSARY-ORIENTED EVALUATION  According to Smith (1985), many of the limitations of this approach relate to its competitive nature, the complexity of the process, and the need for skilled individuals willing to perform the various roles needed for a hearing. Listed are the main limitations of the adversary evaluation: 1. This form of evaluation may provoke venomous debate and conflict may have a negative impact on the outcome of the evaluation. 2. The focus of the evaluation may shift to assigning blame or guilt, rather than optimizing policy. 3. As adversary-advocate models are conflict-based, possibilities for reaching an agreeable outcome are

THE LIMITATIONS OF ADVERSARY-ORIENTED EVALUATION 4. Key stakeholders are not always equally skilled, and articulate individuals are placed at an advantage. 5. This method can be time-consuming and expensive (Owens, 1973) 6. It is sometimes difficult for hearing members to develop specific, operational recommendations (Wolf, 1979. 7. Time-limitations may only allow for a narrow focus.

35

THE APPLICATION OF THE ADVERSARYORIENTED EVALUATION APPROACH

THE APPLICATION OF ADVERSARY-ORIENTED EVALUATION  Although currently out of favour, this approach has been used quite extensively in the field of educational evaluation (Owens, 1973). It has also been applied to ethnographic research (Schensul, 1985) and the evaluation of state employment agencies (Braithwaite & Thompson, 1981). Crabbe and Leroy contend that an adversary approach to evaluation should be beneficial when: 1. The program being evaluated may affect a large group of people; 2. when the issue in question is one of controversy and public attention;