ARM Lecture 6

LOGO

Construction of  Construction of  Questionnaire Lecture # 6

Contents

1.

What is questionnaire?

2.

Framework for questions for questions

3.

Types of questionnaires

4.

Guidelines

LOGO

What is questionnaire?

What is questionnaire? A

questionnaire is a research instrument consisting of a series of  questions and other prompts for the purpose of gathering information from respondents.  Although they are often designed for statistical analysis of the responses, this is not always the case.  Remember, good questionnaires take time to develop. They are not just questions on a page.

 Questionnaires

are used in sample surveys or censuses to elicit reports of facts, attitudes, and other subjective states.  The developments are beginning to transform survey questionnaire construction from an art to a science.

What is not questionnaire?

A questionnaire is not just a list of  questions. It is tempting to begin questionnaire design by drafting possible qns on the survey topics. That is obviously important, but not the right place to start. We must first think about the following topics.

Objectives of questionnaire construction?  To

maximize the proportion of  subjects answering our questionnaire - that is, the response rate.  To obtain accurate relevant information for our survey

History  In

1086 William the Conqueror surveyed the wealth and landholdings of England using a standard set of inquiries and compiled the results in the “Domesday Book.”  The questionnaire was invented by Sir Francis Galton: 1874

Advantages of Written Questionnaires  Questionnaires

are very cost effective when compared to face-to-face interviews. This is especially true for studies involving large sample sizes and large geographic areas. Written questionnaires become even more cost effective as the number of  research questions increases.  Questionnaires are easy to analyze. Data entry and tabulation for nearly all surveys can be easily done with many computer software packages.

 Questionnaires

are familiar to most people. Nearly everyone has had some experience completing questionnaires and they generally do not make people apprehensive.  Questionnaires reduce bias. There is uniform question presentation and no middle-man bias. The researcher's own opinions will not influence the respondent to answer questions in a certain manner. There are no verbal or visual clues to influence the respondent.

 Questionnaires

are less intrusive than telephone or face-to-face surveys. When a respondent receives a questionnaire in the mail, he is free to complete the questionnaire on his own time-table. Unlike other research methods, the respondent is not interrupted by the research instrument.

Disadvantages Of Written Questionnaires  Possibility

of low response rates: rates : Low response is the curse of  statistical analysis. It can dramatically lower our confidence in the results. Response rates vary widely from one questionnaire to another (10% - 90%), however, well-designed studies consistently produce high response rates.

 Inability

to probe responses. Questionnaires are structured instruments. They allow little flexibility to the respondent with respect to response format. In essence, they often lose the "flavor of the response" (i.e., respondents often want to qualify their answers). 

By allowing frequent space for comments, the researcher r esearcher can partially overcome this disadvantage. Comments are among the most helpful of all the information on the questionnaire, and they usually provide insightful information that would have otherwise been lost.

 Nearly

ninety percent of all communication is visual. Gestures and other visual cues are not available with written questionnaires. The lack of personal contact will have different effects depending on the type of information being requested. A questionnaire requesting factual information will probably not be affected by the lack of personal contact. A questionnaire probing sensitive issues or attitudes may be severely affected.

 When

returned questionnaires arrive in the mail, it's natural to assume that the respondent is the same person you sent the questionnaire to. This may not actually be the case. Many times business questionnaires get handed to other employees for completion. Housewives sometimes respond for their husbands. Kids respond as a prank. For a variety of  reasons, the respondent may not be who you think it is. It is a confounding confo unding error inherent in questionnaires.

 Finally,

questionnaires are simply not suited for some people. people. For example, a written survey to a group of poorly educated people might not work because of reading skill problems. More frequently, people are turned off by written questionnaires because of misuse.

LOGO

Framework for construction of  questionnaire

Clarify goals

Conversations with several of the stakeholders

Focus your goals

Invite people to take survey

Analyze responses

Translating data into meaningful information

Present Results

Celebrate your success

Steps for Developing Questionnaires  Develop

the Purpose



Create the Conceptual Framework



Write Questions



Design the Questionnaire

Proper construction of the questionnaire is essential to its success Powell, Ronald R.

Framework: Questionnaire Construction Specify What Information Will be Sought Identify Correct Respondent Determine Method of  Administration Question Structure

Question Wording PRETEST and Revise

Question Sequencing

Steps to design a questionnaire: 1. Write out the primary and secondary aims of your study. 2. Write out concepts/information to be collected that relates to these aims. 3. Review the current literature to identify already validated questionnaires that measure your specific area of interest. 4. Compose a draft of your questionnaire. 5. Revise the draft. 6. Assemble the final questionnaire.

Questionnaire Design - General Considerations  Most

problems with questionnaire analysis can be traced back to the design phase of  the project. Well-defined goals are the best way to assure a good questionnaire design.  When the goals of a study can be expressed in a few clear and concise sentences, the design of the questionnaire becomes considerably easier.  The questionnaire is developed to directly address the goals of the study.

 One

of the best ways to clarify your study goals is to decide how you intend to use the information. Do this before you begin designing the study. This sounds obvious, but many researchers neglect this task. Why do research if the results will not be used?

 Be

sure to commit the study goals to writing. Whenever you are unsure of  a question, refer to the study goals and a solution will become clear. Ask only questions that directly address the study goals. Avoid the temptation to ask questions because it would be "interesting to know".

 As

a general rule, with only a few exceptions, long questionnaires get less response than short questionnaires. Keep your questionnaire short. In fact, the shorter the better.  Response rate is the single most important indicator of how much confidence you can place in the results. A low response rate can be devastating to a study.  One of the most effective methods of  maximizing response is to shorten the questionnaire.

 If

your survey is over a few pages, try to eliminate questions.  Many people have difficulty knowing which questions could be eliminated.  For the elimination round, read each question and ask, "How am I going to use this information?" If the information will be used in a decision-making process, then keep the question... it's important. If not, throw it out.

 One

important way to assure a successful survey is to include other experts and relevant decision-makers in the questionnaire design process.  Their suggestions will improve the questionnaire and they will subsequently have more confidence in the results.

 Formulate

a plan for doing the statistical analysis during the design stage of the project. Know how every question will be analyzed and be prepared to handle missing data. If you cannot specify how you intend to analyze a question or use the information, do not use it in the survey.

 Give

your questionnaire a title that is short and meaningful to the respondent. A questionnaire with a title is generally perceived to be more credible than one without  Include clear and concise instructions on how to complete the questionnaire. questionnaire. These must be very easy to understand, so use short sentences and basic vocabulary. Be sure to print the return address on the questionnaire itself (since questionnaires often get separated from the reply envelopes).

 Begin

with a few non-threatening and interesting items. If the first items are too threatening or "boring", there is little chance that the person will complete the questionnair questionnaire. e.  People generally look at the first few questions before deciding whether or not to complete the questionnai questionnaire. re. Make them want to continue by putting interesting questions first.

 Use

simple and direct language. The questions must be clearly understood by the respondent.  The wording of a question should be simple and to the point.  Do not use uncommon words or long sentences. Make items as brief as possible. This will reduce misunderstandings misunderstan dings and make the questionnaire appear easier to complete.  One way to eliminate misunders misunderstandings tandings is to emphasize crucial words in each item by using bold, italics or underlining.

 Leave

adequate space for respondents to make comments.  One criticism of questionnaires is their inability to retain the "flavor" of a response.  Leaving space for comments will provide valuable information not captured by the response categories.  Leaving white space also makes the questionnaire look easier and this increases response.

 Place

the most important items in the first half of the questionnaire.  Respondents often send back partially completed questionnaires.  By putting the most important items near the beginning, the partially completed questionnaires will still contain important information.

 Use

professional production methods for the questionnaire--either desktop publishing or typesetting and keylining.  Be creative.  Try different colored inks and paper. The object is to make your questionnaire stand out from all the others the respondent receives.

 The

final test of a questionnaire is to try it on representatives of the target audience. audience.  If there are problems with the questionnaire, they almost always show up here. If possible, be present while a respondent is completing the questionnaire and tell her that it is okay to ask you for clarification of any item.  The questions she asks are indicative of problems in the questionnaire (i.e., the questions on the questionnaire must be without any ambiguity because there will be no chance to clarify a question when the survey is mailed).

LOGO

Types of questions

Classification # 1 1. Factual questions: age, gender… 2. Opinion and attitude questions: intended to determine person’s ideas, inclinations, prejudices, … with attitude scales, and indexes 3. Information questions: designed to measure respondent’s knowledge 4. Self-perception questions: questions: Similar to attitude questions but are restricted to one’s opinions about himself or herself  5. Standards of action questions: questions: used to determine how respondents would act in certain circumstances 6. Questions about actual past or present behaviour: behaviour: 7. Projective questions: questions:

Classification # 2  Four

types of response scales for closed-ended questions are distinguished: 

Dichotomous, where the respondent has two options



Nominal-polytomous, where the respondent has more Nominal-polytomous, than two unordered options



Ordinal-polytomous, where the respondent has more than two ordered options



(bounded)Continuous, where the respondent is (bounded)Continuous, presented with a continuous scale

1. Contingency questions - A question that is answered only if the respondent gives a particular response to a previous question. This avoids asking questions of people that do not apply to them 2. Matrix questions - Identical response categories are assigned to multiple questions. The questions are placed one under the other, forming a matrix with response categories along the top and a list of questions down the side. This is an efficient use of page space and respondents’ time.

3. Closed ended questions - Respondents’ answers are limited to a fixed set of responses.   Yes/no questions - The respondent answers with a “yes” or a “no”.  Multiple choice - The respondent has several option from which to choose.  Scaled questions - Responses are graded on a continuum (example : rate the appearance of the product on a scale from 1 to 10, with 10 being the most preferred appearance). Examples of types of scales include the Likert scale, semantic differential scale, and rank-order scale (See scale for a complete list of  scaling techniques.).

 Open

ended questions - No options or predefined categories are suggested. Examples of types of open ended questions include: 

Completely unstructured - For example, “What is your opinion of  questionnaires?”



Word association - Words are presented and the respondent mentions the first word that comes to mind.



Sentence completion - Respondents complete an incomplete sentence. For example, “The most important consideration in my decision to buy a new house is . . .”



Story completion - Respondents complete an incomplete story.



Picture completion - Respondents fill in an empty conversation balloon.



Thematic apperception test - Respondents explain a picture or make up a story about what they think is happening in the picture

Question Structure Open-end questions If you were to purchase a new automobile during 2001, what brand of automobile would you most likely purchase? ___________ brand

Closed-ended with ordered categories If you were to purchase a new automobile during 2001, how likely is it that you would purchase the following brands of automobiles? Very Likely

Somewhat Likely

Somewhat Unlikely

Very Unlikely

Toyota Camry

1

2

3

4

Honda Accord

1

2

3

4

Nissan Maxima

1

2

3

4

Closed-ended with unordered categories If you were to purchase a new automobile during 2001, which one of the following automobiles would you most likely purchase? Toyota Camry Nissan Maxima

t t

Honda Accord

t

Partially closed-ended If you were to purchase a new automobile during 2001, which one of the following would you most likely purchase? Toyota Camry Nissan Maxima specify)______________ 

t

Honda Accord

t

Other (please

t

Closed Vs. Open-ended Questions Advantages Closed-ended

Open-ended

•Easy and quick to answer •Answers across resps easy to compare •Answers easier to analyze on computer •Response choices make question clearer •Easy to replicate study

Disadvantages

•Can put ideas in resp's head •Resps w/ no opinion answer anyway •Resps can feel constrained/frustrated •Many choices can be confusing •Can't tell if resp. misinterpreted the question •Fine distinctions may be lost •Clerical mistakes easy to make •Force respondents into simple responses •Permit unlimited number of  •Resps give answers w/ diff. level of  answers detail •Resps can qualify and clarify •Answers can be irrelevant •Inarticulate or forgetful resps are at responses •Can find the unanticipated disadvantage •Reveal resps thinking processes •Coding responses is subjective and tedious •Requires more resp. time and effort •Intimidates respondents •When resp omits a response, can't tell if its because of belief or just forgetfulness

Closed Vs. Open-ended Questions Evaluation Criteria

Advantages

Disadvantages

Time

Less time to respond & record

Longer time to develop

Quick transfer to computer form

Evaluation Criteria

Advantages

Disadvantages

Cost

Cheaper due to lower   recording & interpreting time

Evaluation Criteria

Advantages

Disadvantages

Accuracy

Less interviewer   recording error 

Forcing answers into categories

Respondent convenience

More convenient to answer 

Structured Questions: Effect on Respondent Answers  Response

order effects  Visual presentation of response alternatives  Primacy effects 

initial items establish a cognitive framework to guide processing



initial items get deeper cognitive processing & less interference than later items



satisficing — minimize psychological costs by choosing first acceptable response alternative

SOURCE: Krosnick and Alwin, POQ

Structured Questions: Effect on Respondent Answers Response order effects

Oral presentation of response alternatives Recency effects externally based processing each read alternative terminates processing of previous alternative increased processing of later response alternatives

Source: Source: An evaluatio evaluation n of a cognitive cognitive theory theory of respons response-orde e-orderr effects in survey survey measurement,” J. Krosnick and D. Alwin, Public Opinion Quarterly , 1987, 51, 1201-1219.

Question Sequence and Order Effects

Question

Once Committed Hard to be Unfair 

Do you you think a communist communist country lik Question sequence 2 Yes reporters come in and send back to t 37% Difference !

Do you think the United States should let communist newspaper reporters from other countries come in and send back to their papers the news as they see it?

Do you think th ink a communist communist country count ry like Russia should shou ld let American American newspaper  newspaper  reporters come in and send back to their pape rs the news as they see it?

36%

Do you you think the United States shoul other countries come inSurand send ba Source: Source: “The Norm Norm of Even-ha Even-hande ndedness dness in Surveys veys as as in Life,” Life,” H. 66%

Questionnaire Questionna ire Format  Matrix

questions are often useful.



They use space efficiently



Respondents can answer quickly



Respondents can compare across questions for consistency.

 However,

matrix questions have some

problems. 

You might be tempted to put questions into a matrix when they belong in some other format.



Respondents may pattern their answers inappropriately.



Order effects may be quite pronounced, with strong potential for abuse of the survey.

 Some

special comments on order effects.



Early answers tend to influence later answers.



Some respondents (such as less educated respondents) are more subject to order effects than other respondents.



Randomizing questions tends not to help - it just makes the Randomizing survey more confusing.



Fun or interesting questions in the early part of a questionnaire questionnai re can maintain interest, but general, demographic sorts of questions can establish rapport. • You might want to adjust the order of questions according to to the type of survey – interview or questionnaire.

Other Types  Leading

Questions: A leading question is one that forces or implies a certain type of answer. E.g. 

Superb



Excellent



Great



Good



Fair 



Not so Great

 Embarrassing

Questions: Embarrassing questions dealing with personal or private matters should be avoided. Your data is only as good as the trust and care that your respondents give you. If  you make them feel uncomfortable, you will lose their trust. Do not ask embarrassing questions.

 Hypothetical

Questions: Hypothetical are based, at best, on conjecture and, at worst, on fantasy. I simple question such as: 

If you were governor, what would you do to combat terrorism?

LOGO

Most frequent errors in questionnaires

Issues to Consider To Neutral or Not to Neutral.

Strongly Disagree

Disagree

Neither Agree nor   Disagree

Agree

Strongly Agree

1

2

3

4

5

Think about how a respondent enters a response scale and how many decisions the person must make.

Pre-coding can save time with data entry. Has this presentation increased your understanding of questionnaire development?

1

Yes

2

No

Pre-code

There is more than one way to ask a question. Use a variety of question formats.

What do you think?

Were you satisfied with the quality of our services and programs? A. Yes B. No Double Barrel: Ask only one question at a time

most frequent errors in questionnaires  questions

which don't quite mean what the researcher intended  questions which don't probe to find out what the respondent really meant long and complicated questions  questions which repeat what has already been asked  questions which don't allow the respondent to answer in a way which is relevant  questions which are inappropriate to the research method

 bad

routing which leaves the interviewer wondering which question to ask next, or worse, routing to the wrong one too many questions  poorly laid out questions which are badly grouped in the questionnaire  questions which have been missed out completely  pages of the questionnaire which are missing or out of order.

 Research

showing that small changes in question wording or order can substantially affect responses has reinforced the assumption that questions must be asked exactly as worded, and in the same order, to produce comparable data.

LOGO

Guidelines

Quality aims in survey research Goal is to collect information that is:  Valid: measures the quantity or concept that is supposed to be measured  Reliable: measures the quantity or concept in a consistent or reproducible manner  Unbiased: measures the quantity or concept in a way that does not systematically underor overestimate the true value  Discriminating: can distinguish adequately between respondents for whom the underlying level of the quantity or concept is different

Qualities of a Good Question  Evokes

the truth. Questions must be nonthreatening. When a respondent is concerned about the consequences of  answering a question in a particular manner, there is a good possibility that the answer will not be truthful. Anonymous questionnaires that contain no identifying information are more likely to produce honest responses than those identifying the respondent. If your questionnaire does contain sensitive items, be sure to clearly state your policy on confidentiality.

 Asks

for an answer on only one dimension. dimension. The purpose of a survey is to find out information. A question that asks for a response on more than one dimension will not provide the information you are seeking. For example, a researcher investigating a new food snack asks "Do you like the texture and flavor of the snack?" If a respondent answers "no", then the researcher will not know if the respondent dislikes the texture or the flavor, or both

 Can

accommodate all possible answers. Multiple choice items are the most popular type of survey questions because they are generally the easiest for a respondent to answer and the easiest to analyze. Asking a question that does not accommodate all possible responses can confuse and frustrate the respondent. For example, consider the question:   

What brand of computer do you own? __  own?  __   A. IBM PC  B. Apple

 Has

mutually exclusive options. A good question leaves no ambiguity in the mind of  the respondent. There should be only one correct or appropriate choice for the respondent to make. An obvious example is:  Where did you grow up? __  up? __    A. country   B. farm  C. city   A person who grew up on a farm in the country would not know whether to select choice A or B.

 Produces

variability of responses. When a question produces no variability in responses, we are left with considerable uncertainty about why we asked the question and what we learned from the information. If a question does not produce variability in responses, it will not be possible to perform any statistical analyses on the item. For example:  

What do you think about this report?  __   __   A. It's the worst report I've read 



B. It's somewhere between the worst and best 



C. It's the best report I've read 

 Almost

all responses would be choice B,

 Order

of questions:



Go from general to particular.



Go from easy to difficult.



Go from factual to abstract.



Start with closed format questions.



Start with questions relevant to the main subject.



Do not start with demographic and personal questions.

 Follows

comfortably from the previous question. Writing a questionnaire is similar to writing anything else. Transitions between questions should be smooth. Grouping questions that are similar will make the questionnaire easier to complete, and the respondent will feel more comfortable. Questionnaires that  jump from one unrelated topic to another feel disjointed and are not likely to produce high response rates.

 Does

not presuppose a certain state of  affairs. Among the most subtle mistakes in questionnaire design are questions that make an unwarranted assumption. An example of this type of mistake is: 

 Are you satisfied with your current auto insurance? (Yes or No)

 This

question will present a problem for someone who does not currently have auto insurance. Better: 

 Are you satisfied with your current auto insurance? 

 ___ Yes  ___  Yes  __   __ No No

insurance

___ Don't ___  Don't have auto

 Does

not imply a desired answer. The wording of a question is extremely important. We are striving for objectivity in our surveys and, therefore, must be careful not to lead the respondent into giving the answer we would like to receive. Leading questions are usually easily spotted because they use negative phraseology. As examples: 

Wouldn't you like to receive our free brochure? 



Don't you think the PPP Govt is spending too much money? 

 Does

not use emotionally loaded or vaguely defined words. This is one of the areas overlooked by both beginners and experienced researchers. Quantifying adjectives (e.g., most, least, majority) are frequently used in questions. It is important to understand that these adjectives mean different things to different people.

 Does

not use unfamiliar words or abbreviations. Remember who your audience is and write your questionnaire for them. Do not use uncommon words or compound sentences. Write short sentences. Abbreviations are okay if you are absolutely certain that every single respondent will understand their meanings. If there is any doubt at all, do not use the abbreviation. The following question might be okay if all the respondents are accountants: 

What was your company’s liquidity ratio last year?  year? ______  ______ 

. How much did you spend last year for life insurance ? ______ 

 Is

not dependent on responses to previous questions. Branching in written questionnaires should be avoided. While branching can be used as an effective probing technique in telephone and face-to-face interviews, it should not be used in written questionnaires questionnaires because it sometimes confuses respondents. An example of branchin branching g is:  1. Do you currently have a life insurance policy ? (Yes or No) If  no, go to question 3 

2. How much is your annual life insurance premium ? ______

 Alternate 

Option:

How much did you spend last year for life insurance ? ______

 Does

not ask the respondent to order or rank ran k a series of more than five items. This becomes increasingly difficult as the number of items increases, and the answers become less reliable. This becomes especially problematic when asking respondents to assign a percentage to a series of  items. In order to successfully complete this task, the respondent must mentally continue to rer eadjust his answers until they total one hundred percent. Limiting the number of items to five will make it easier for the respondent to answer.

Questionnaire Construction  Keep

it short, simple, and specific (KISSS)  Logical question progression  Flow more important than grammatical correctness  Key questions first, sensitive last  Keep allied questions together  Be reasonable when requesting forecasts fully-structured tructured and open-ended questions  Mix fully-s  Clear routing instructions, use colour

General Questionnaire Sequence  Start

with simple more general questions 

nonthreatening



interesting



general respondent capability

 Respondents

will be looking for an Excuse  to Terminate or Stop Interview

Comment on this question?

 How

often did you talk to your dental rep from XYZ during the past three months? 

Never 



Rarely



Occasionally



Regularly

Revision  How

often did you talk to your dental rep from XYZ during the past three months? 

Not at all



Once per month



About 2-3 times per month



About once per week



More than once per week

Don’t Use Double Questions This

cement will be sold in premixed tubes and does not require agitation Do

you believe this product is different enough from the product you currently use and would you purchase it? • Yes • No • Don’t know

Checklist for Question Wording vague - questions and answers q bias or leading q objectionable/embarrassing q too demanding q check for double questions q do they have the ability to answer q give frame of reference q consider ability to compare responses across past studies q

Questionnaire Pretesting  PRETESTING

 Qualitative

•

METHODS

assessment

debriefing

•  Quantitative

•

assessments

pilot study (method of  administration)

Resources  Moser

CA, Kalton G. Survey methods in social investigation. 2nd ed. Aldershot: Gower; 1971.  Sheatsley, P.B. (1983) Questionnaire construction and item writing. In Ross, P.H., Wright, J.D. and Anderson, A.B.(Eds.), Handbook  of Survey Research. New York: Academic Press.  Sudman, S. and Bradburn, N.M. (1989) Asking (1989) Asking questions: A practical guide to questionnaire design. San Francisco: Jossey Bass.

 Oppenheim,

A.N. (1992) Questionnaire design, interviewing and attitude measurement. London: Pinter 

 http://www.marketresearchworld.net  www.statpac.com/surveys  “Design

and use of questionnaires: a review of best practice applicable to surveys of health service staff  and patients”, Health Technology Assessment, 2001. Vol.5, No. 31.

LOGO

Thank you for your kind attention Go forth and research…. ….but be careful out there.

Hot Tip

How

do I incorporate my logo to a slide that will apply to all the other slides? 

On the [View] menu, point to [Master], and then click [Slide Master] or  or [Notes [Notes Master]. Change images to the one you like, then it will apply to all the other slides.

LOGO

www.themegallery.com