Reid, J. (2010) The second in our series to take the mystery out of critical appraisal looks at articles based on qua...
JOURNAL CLUB
Journal club 2: Jo qualitative research Jennifer Reid’s series aims to help you access the speech and language therapy literature, assess its credibility and decide how to act on your ndings. Each instalment takes the mystery out of critically appraising a dierent type of journal article. Here, she looks at qualitative research.
k oons.co.u to rancar t fr w.. f w w w
I
rather like numbers. It seems obvious to me to count things to see if there is a pattern, evidence of progress or a useful comparison to be made. At this point, you will not be alone if you are thinking ‘anorak’! Many of you may feel that as soon as you convert something to numbers it loses any kind of real meaning. However, in research design, it’s all about horses for courses; using numbers (quantitative methods) methods) works for studying some kinds of phenomena and sticking with words (qualitative (qualitative methods) methods) works for others. Here is a story to illustrate the dierent yet potentially complementary approaches of quantitative and qualitative methods. Amy leads a small team of speech and language therapists working in the community in quite a sparsely populated area, so she and her colleagues spend a lot of their time travelling, and timetabling can be quite a juggling act. She notices that referrals are rising quite rapidly in one part of their patch while elsewhere they remain pretty constant. As she may need to change how the service is deployed to make best use of the team’s resources, she discusses this with her manager, and they agree they need to know a bit more about what might be going on. What background research would help them? Some sources of quantitative of quantitative data that might be available are:
18
1. The local authority publishes regular bulletins on population trends within age bands in NHS areas, parliamentary constituencies, council wards, secondary school catchments areas and so on. 2. The speech and language therapy service has an electronic database of referrals and their sources going back over 12 years. 3. Most of the team’s referrals come from fro m practitioners in either primary/community care (40 per cent) or education (45 per cent). 4. Amy’s not sure but feels there may have been more inappropriate referrals over the past year, and she is certain they are seeing a lot more 2-3 year-olds than they used to. 5. Four of the th e biggest schools they cover have new headteachers and there have been quite a few meetings with them to discuss roles and responsibilities and to go through the procedures for referral. 6. Several health visitors in the area retired recently and Amy feels that the team has not yet had the opportunity to develop strong working relationships with their successors. Two community paediatrician posts have also been vacant for more than a year now, and the health board has been unable either to ll the posts or to secure locum cover.
SPEECH & LANGUAGE THERAPY IN PRACTICE WINTER 2010
The numerical sources 1 and 2 should give Amy and her manager an indication of whether the trends in referrals reect (a) changes in the population demographics of the area, or (b) changes in the referral rate (the proportion of the population being referred). Items 3 and 4 provide gures for investigating whether increasing referral numbers appear to be associated with particular sources of referral rather than across the board. Items 5 and 6 probably reect some of Amy’s hypotheses about potential sources of temporary uctuation in referral rates. They could use their database to check numbers of referrals over time from these schools and healthcare practitioners to see if the pattern of uctuations over time in referral numbers is associated with personnel change. However, knowing who is referring more, or fewer, clients than is typical for their practitioner group won’t necessarily help Amy and her manager understand why why this this is happening. They need to elicit information on the perspectives of these referrers. Here is where the qualitative techniques of document study, observation and interview come into their own. In this case, exploring the content of referral forms (document study) and / or exploring referrers’ decisionmaking (using individual or group interview
JOURNAL CLUB
Criticial appraisal for speech and language therapists (CASLT) (CASLT) Download the ‘qualitative research’ framework document from www.speechmag.com/Members/CASLT. Use it yourself or with colleagues in a journal club, and let us know how you get on (email
[email protected]).
techniques) should help Amy and her manager understand, and subsequently manage, the reasons underlying the changes in their referral numbers. Qualitative methods are the ‘gold standard’ (the method of choice) for exploring and creating meaning from participants’ subjective experiences, and for gaining insight into phenomena that are ill-dened or poorly understood. The strength of these methods rests in their ability to reveal the underlying ‘truth’ of a phenomenon – in research terms, its validity . It may be true only for a single individual at one point in time and in one particular context, but listening to or observing individuals is much more likely to elucidate their subjective experience as they perceive it than any amount of counting. As Greenhalgh puts it: “The strength of qualitative research lies in validity (closeness to the truth) – that is, good qualitative research, using a selection of data collection methods, really should touch the core of what is going on rather than just skimming the surface. The validity of qualitative methods is greatly improved by the use of more than one method in combination (… triangulation triangulation), ), by the researcher thinking carefully about what is going on and how their own perspective might be inuencing the data (… reexivity ), ), … “ “ (Greenhalgh, (Greenhalgh, 2006, pp.168-9). However, you do need to bear in mind that the strength of qualitative methods may also be the source of a potential weakness. Generalisation of the ndings of qualitative research to other contexts (people, setting) may not be justied, or at least not without appeal to a wider evidence base. It is the reliability of quantitative methods which give us condence that the same study would produce roughly the same ndings if it were to be repeated elsewhere with dierent participants. This reliability allows us to generalise with a degree of condence beyond the immediate context in which a study has been conducted. Authors of qualitative studies may not help you much with this, so you do need to think carefully about the context of the research study and how this may dier from your own. From a service development point of view there is no point in searching for meaning in studies whose context or setting will not illuminate your own service issues. A fantastic research team and top-notch methods won’t do you much good if you go looking in the wrong place. As the story goes:
READ THIS SERIES IF YOU WANT TO y BE MORE EVIDENCE-BASED IN YOUR PRACTICE y FEEL MOTIVATED TO READ JOURNAL ARTICLES y INFLUENCE THE DEVELOPMENT OF YOUR SERVICE
Late one evening, a passer-by notices someone searching about under a lamppost. “Have you lost something,” says the publicspirited passer-by, “Can I help you?” The searcher replies that he has dropped his car key and accepts the oer of assistance. Some time elapses while both search the ground all round the lamppost. “Are you sure you dropped it here?” queries the passerby, “I can’t see it anywhere.” “Eh, no,” says the searcher, “I dropped it down the road but it’s so dark down there I couldn’t see properly so I came up here where it’s brighter.”
methods are expected to have their research question clearly formulated at the outset. In qualitative designs, however, it is acceptable for the aims to be inuenced by the data analysis (the so-called ‘iterative’ approach, in which results inuence questions, which then inuence results, and so on in a cyclic fashion). The aut authors hors sho should uld the theref refore ore expl explain ain if and how their aims have been shaped during the study. Question 2: Was the choice of qualitative approach appropriate?
There is a plethora of literature on qualitative methods should you wish to read about this more widely (Barbour, 2008; Immy, 2010). I also recommend a useful short summary article that is a free download from the British Medical Journal (Kuper et al .,., 2008).
QUALITATIVE RESEARCH FRAMEWORK Use the following questions adapted from Greenhalgh (2006) and CASP (PHRU, 2006) to assess the rigour, credibility and relevance of the study and whether there are implications for your service. This is set up at www/speechmag. com/Members/CASLT as a document for you to download, print o and use as an individual or with colleagues in a journal club. Question 1: Did the paper describe an important clinical problem addressed via a clearly formulated question?
Was a qualitative method appropriate in the rst place? Look for evidence that the study sought to interpret or illuminate the actions and / or subjective experiences of the research participants. Was the research design appropriate to address its aims? Consider whether the authors discussed selection and justication justi cation of their their methods. methods. Question 3: How were the participants selected?
This may be bro This broken ken down int into o whe whethe therr the aims of the research were clearly stated, what the point of the study was, why the problem is important and how the issue is relevant for clinical practice. Studies that use quantitative
Have the authors explained how the participants were selected? You may be less familiar with qualitative sampling methods – who you select to interview or observe, or which texts you choose to analyse – as they are dierent from the ‘representative ‘representative sampling’
SPEECH & LANGUAGE THERAPY IN PRACTICE WINTER 2010
19
JOURNAL CLUB (and variations thereof) used in quantitative designs. Kuper et al . (2008) have a helpful list of qualitative sampling methods for interviews or focus groups in healthcare settings. These include self-evident ones like: • typical or deviant case sampling, sampling, to less familiar ones like: • maximum-variation sampling – sampling from a range of perspectives that is wide enough to include all the factors that might inuence the nature and quality of people’s experiences, or • snowball sampling – each recruit to the study generates more participants through personal contact, often used for research with ‘hard to reach’ or stigmatised groups. Have the authors explained why the participants they selected were the most appropriate to provide access to the type of knowledge sought by the study? Were there any discussions around recruitment, such as why some people chose not to take part? Question 4: Were the data generated in an appropriate way?
Was the setting justied – the right place to look rather than the easiest ? (Not just looking under lampposts!) People’s experiences of your service may be fresh in their mind on the way out of the door from their rst appointment but perhaps they might provide more meaningful information once they have experienced some intervention intervention.. In my local setting, we recently conducted a series of focus groups with people with aphasia. SamanThe thaparticipants Pauwere la all folk whose stroke had occurred relatively recently but who were now discharged from speech and language therapy. We could be reasonably condent that: a. the group’s views provided a unique perspective on their whole care pathway, which reected our (and other healthcare practitioners’) practitioners ’) current practice, b. the impact of communication diculties had been minimised, and c. there was no undue inuence from a desire to maintain current therapeutic relationships. The ndings are being used as an evidence base for our current service strategy - for example, that we invest eort and resources in raising awareness of aphasia and health promotion for people living with aphasia in our local community community..
20
Qualitative design methods involve exploration and interpretation via data generation They are stronger on validity (closeness to the truth). They are the preferred methods for poorly understood or relatively unexplored phenomena. .
Documents
Study of of do documents produced by by re real pe people in real situations (for example, casenotes).
Passive observation
Systematic recording of behaviour and talk in naturally occurring settings.
Participantobservation
The researcher takes part in the setting as well as observing.
Semistructured interview
Face-to-face (or telephone) conversation with the purpose of exploring issues or topics in detail. Uses a pre-set list of questions or topics but is not restricted to these.
Narrative interview
Interview undertaken in a less structured fashion, with the purpose of getting a long story from the interviewee (typically a life story or the story of how a condition has unfolded over time). The interviewer uses only general prompts to “tell me more”.
Foc ocus us Gr Grou oups ps
Method Meth od of gr grou oup p int inter ervi view ew wh whic ich h exp expli lici citl tly y includes and uses the group interactions to generate data.
Table 1 Qualitative methods
Is it clear how data were generated in the study, and have the researchers justied the methods chosen? Table 1 is extracted from the rst article in this series to remind you of the most common sorts of qualitative methods. methods. Authors should make their methods very explicit so that you can judge whether bias might have crept in. For example, for an interview method, is there an indication of how interviews were conducted? If dierent interviewers were involved, how did they minimise dierences in personal interviewing style? Did they use a topic guide or a standard set of questions and / or prompts? A lot of qualitative research is exploratory and so researchers are expected to reect on their data as it is being generated. This means that methods may be modied during the study. Check whether this happened in this study, and, if so, do the authors provide enough explanation of how and why? Is the form of data clear? A good test is whether you can you visualise what the data looks like (tape recordings, video material, notes)? Illustrations of eld diaries, transcription frameworks and so on are helpful. Now we come to a bit of qualitative technospeak. Did they discuss data saturation?? This is the point at which your saturation method is generating no new information. Question 5: Was the study conducted within an appropriate ethical framework?
SPEECH & LANGUAGE THERAPY IN PRACTICE WINTER 2010
Participants in a research study need to understand what they are letting themselves in for, why the study is important and how their contribution will be used once the study is completed and disseminated. This is true of all research. However, in qualitative studies the ethical issues can be trickier, since the personal experiences of the participants will not necessarily be reduced to a set of anonymous numbers. There are potential consequences when people’s personal experiences are exposed both to themselves and to others. Have the authors discussed the ethical issues raised by the study, such as informed consent (and how they achieved this where children or participants with communication diculties are concerned), condentiality and how they have handled the eects of the study on the participants during and after the study? It’s not enough just to know that approval was gained from a recognised ethics committee co mmittee (but they should mention this as well). Question 6: What perspective is the researcher coming from, and how has this been built into the study? Another bit of qualitative technospeak is refexivity – the extent to which the researchers critically examined their own role, potential bias and inuence during the course of the study. You need to look at reexivity in the formulation of the research questions, in the data generation, including how they recruited their sample and in their choice of location. It is also critical that they explain how they responded to events during the study and considered the implications of any changes in the research design.
JOURNAL CLUB Question 7: How have the data been analysed?
it one or two dangers of its own. Ignoring data that doesn’t t into any of your categories or themes is one of them. To what extent have contradictory data been taken into account? Question 8: How credible are the ndings?
Did the authors critically examine their own role, potential bias and inuence during analysis and selection of data for presentation? Remember that qualitative research often uses an ‘iterative’ method where data generation and analysis proceed hand-inhand. Each may inuence the other in a cyclic fashion during the course of the study. This means that it is essential that the report includes an in-depth description of the analysis process. Was thematic analysis used? If so, is it clear how the categories or themes were derived from the data? ‘Bottom-up’ categories are derived from the raw data itself whereas some studies will start with some of their categories pre-set (top-down), perhaps derived from previous research. Horses for courses, but the authors should have described how they arrived at their categories. Beware of bland statements about use of, for example, ‘grounded theory’, without any kind of explanation.. Do not take on the responsibility explanation for not knowing exactly what they mean! In a journal club earlier this year we were reviewing an article on factors inuencing indirect interventions for adults with learning disability (Graves, 2007), which we found helpful for supporting our current practice. The term ‘grounded ‘grounded theory’ was new to most people in the group so we found her brief explanation invaluable – it’s a ‘bottom-up’ approach to building categories – but I also found another article (Skeat & Perry, 2008) which actually explained the grounded theory approach in a speech and language therapy context. We could do with a few more accessible reviews of methods, illustrated from the speech and language therapy eld, to help us with our appraisal of methods. It’s common to include in the text of the article examples from the data set to illustrate categories and themes. Do they explain how the data presented were selected from the original sample to demonstrate the analysis process? Are sucient data presented to support the ndings? In qualitative research, the researcher is included as one of the study components, rather than being seen as some sort of objective, reliable, impartial and unbiased observer from the planet Zog. This brings with
First of all, consider whether the ndings are explicit enough, then whether there is adequate discussion of the evidence both for and against the researcher’s researcher ’s arguments. Do they discuss how they have attempted to increase the credibility of their ndings using methods such as triangulation (using dierent sources to elicit and compare information), respondent validation (using participants that are qualied to speak about the issue in question), more than one analyst ? Do you believe the authors’ narrative? Finally, are the ndings discussed in relation to the original research questions? It’s important to know if their results did not in the end really illuminate the issue they set out to explore. After all, you’re probably only appraising the article because you think the study aims were related to some aspect of your own clinical work. Question 9: How valuable are the ndings? Do the authors discuss the contribution the study makes to existing knowledge or understanding? We have already established that it’s more dicult to generalise from qualitative studies so it’s important that the ndings of this one study are plugged into the bigger picture, however fuzzy it might be. For example, do they consider the ndings in relation to current practice or policy, to other relevant researchbased literature or to theory? A theoretical motivation is considered particularly appropriate for exploration of issues with no substantive research base. Qualitative ndings sometimes provide at best an incomplete or inconclusive story. Do the authors identify new areas where research is necessary? Do they discuss whether or how the ndings can be transferred to other populations or consider other ways the research may be used?
Question 10: What impact does this study have? Do the study ndings resonate with my own experiences (and / or those of my colleagues) and therefore have a potential impact: • for my practice practice • for my colleagues colleagues or care group • for the service as a whole? Is there a further question to be asked, or a local study to be carried out, that would allow me to generalise the ndings to my own context? And nally... In another recent journal club we used this framework to appraise an article on the perceptions of clinicians and parents on the outcomes of speech and language therapy with children aged 2-6 years (Thomas-Stonell et al., al., 2009). The study had been conducted in Canada but we looked very carefully at how they selected their participants and at the sorts of ‘provider organisations’ included. We concluded that there seemed no reason to think that the participants were signicantly dierent from the sorts of families we see in our own paediatric service. Their ndings on the disparity between parents’ and therapists’ perceptions and perspectives of change also rang true for us, providing support for our current service-wide focus on outcome measures that are clientcentred, and related to impact rather than being impairment-based. Parents, apparently, are more aware than therapists of how speech, language and communication disorders are aecting their children’s social participation. SLTP Now there’s a surprise! Jennifer Reid is a consultant speech and language therapist with NHS Fife, email
[email protected]. jenniferreid@ nhs.net. References Barbour, R.S. (2008) Introducing qualitative research: a student’s guide to the craft of qualitative research. Los Angeles: Sage. Graves, J. (2007) ‘Factors inuencing indirect speech and language therapy interventions for adults with learning disabilities: the perceptions of carers and therapists’, Int J Lang Comm Comm Dis,42(S1), Dis, 42(S1), pp.103-121. Greenhalgh, T. (2006) How to read a paper: the basics of evidence-based medicine (3rd edn). Oxford: Blackwell. Holloway, I. & Wheeler, S. (2010) Qualitative Research in Nursing and Healthcare. Chichester: Wiley-Blackwell. Kuper, A., Lingard, L. & Levinson, W. (2008) ‘Critically appraising qualitative data’, BMJ 337, pp. 687-90. Available at: http://www.bmj.com/content/337/bmj. a1035.full (Accessed: 8 November 2010.) Public Health Research Unit (2006) Critical Appraisal Skills Programme. Available at: www.phru.nhs.uk/Pages/PHD/ CASP.htm (Accessed: (Accessed : 8 November 2010.) Skeat, J. & Perry, A. (2008) ‘Grounded theory as a method for research in speech and language therapy’, Int J Lang Comm Dis, 43(2), pp.95-109. Thomas-Stone Thomas -Stonell, ll, N., Oddson, B., Roberts Robertson, on, B. & Rosenbaum, P. (2009) ‘Predicted and observed outcomes in preschool children following speech and language treatment: Parent and clinician perspectives’ , perspectives’ , J Comm Disorder Disorderss 42, pp.29-42.
SPEECH & LANGUAGE THERAPY IN PRACTICE WINTER 2010
21
