Journal club 5: observational studies

JOURNAL CLUB 5

 Journal club 5:  Jo observational studies Jennier Reid’s series aims to help you access the speech and language therapy literature, assess its credibility and decide how to act on your ndings. Each instalment takes the mystery out o critically appraising a diferent type o journal article. Here, she looks at observational studies.

READ THIS SERIES IF YOU WANT TO y

y

y

BE MORE EVIDENCEBASED IN YOUR PRACTICE FEEL MOTIVATED  TO READ READ JOURNAL ARTICLES INFLUENCE DEVELOPMENT OF YOUR SERVICE

R

esearch is undamentally a quest or explanations. Explanations go beyond simple description in order to provide an account o causal relationships, or example, between events, human belies, behaviour, experiences and ill-health. It is really important to keep the notion o  causality in mind as we try to get our heads round the observational designs used in health-related research. Causal reasoning, in a nutshell, requires us to: • know what conditions conditions preceded the phenomenon o interest, • assess which o these antecedent conditions are candidates as causal agents, and then • organise this knowledge into a plausible, plausible, causal chain o events.  There  Ther e are are a number number o observa observationa tionall resear research ch designs, and not all provide robust evidence o  causality. It is not enough to demonstrate that there is an association between two actors. I you nd that children living in high ats have poorer health, this is not good evidence that living in a at causes ill-health. The two actors may be related, with economic or social circumstances perhaps much better candidates or an underlying cause.

Experimental or observational? When an article talks about an intervention, how do I know i this is an experimental study or an observational one? Group intervention studies, especially those using randomisation to groups, are considered superior to observational designs or answering causal questions about healthcare interventions, since there is better control o  the efects o any unoreseen (conounding ( conounding)) actors. However, such an experimental design is not always practical or possible, especially where the participants’ common characteristic cannot be manipulated (such as having a genetic condition) or would not be ethical because o likely negative efects (or example, not talking to your baby). Observational studies examine aspects o people’s past and/or present lie in order to identiy relevant inormation through observation rather than experimental man-

18

NHS Fie’s Fie’s Dunermline cluster Journal Club. Author Jen Reid has her back to the camera.

ipulation. They do not ofer a particular intervention and measure directly  its efect. However, inormation may be collected about intervention(s) intervention (s) the participants have received.  The goal o observational studies is usually to identiy actors which may  be causally related, and thus may be incorporated into interventions which then produce better outcomes in the uture. Sometimes people who have received a particular intervention are ollowed up and their outcomes compared to the outcomes o  others who did not receive the intervention. An observational study appraisal tool is likely to be the one to choose or such a study. The exception is i the intervention was ofered as a core part o the research design and participants were allocated or selected to receive one intervention or another according to some preset criteria.

Observational designs  There are a numb  There number er o observa observationa tionall desig designs ns and some jargon to deal with. 1. Cohort study A cohort study ollows over time o a group o people who have something in common. (The term cohort was also used to reer to a group o Roman soldiers, which could be a

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2011

useul aide-mémoire.) The group may have a common characteristic, such as where the participants were born, how they were educated or an aspect o their health or wellbeing. Alternatively they may have all been exposed to a risk or challenging circumstance o some kind, or have received a particular health intervention.  The comp comparis arison on grou group p or coh cohort ort stu studies dies may be the general population rom which the cohort is drawn, or another cohort o persons thought to be similar except or the common characteristic under investigation. Alternatively, subgroups within the cohort may be compared with each other. This is commonly the case in birth cohort studies, where all children born in particular years in one geographical area are studied. Results are analysed to detect a cohort efect. efect. This means nding out whether membership o the cohort, and thereore having the common characteristic, appears to make a diference to the outcome. In research designed to investigate risks o adverse health outcomes, a cohort is identied before the appearance o the condition(s) under investigation. For example, Conti-Ramsden & Botting (2007), in a study o emotional health in adolescents, describe their young people with specic language impairment as, “originally recruited at 7 years o age as part o a wider study … The original

JOURNAL CLUB 5

Figure 1 Criteria or Causation, rom Bradord Hill (1965)

Critical appraisal for speech and language therapists (CASL (CASLT) T) Download the observational study and survey questionnairee frameworks questionnair from www.speechmag w ww.speechmag.. com/Members/CASLT for your own use or with colleagues in a journal club.

Temporal relationship: Cause always precedes the outcome. Strength: The stronger the association, the more likely it is that the relationship is causal. (NB Look at the signicance o those correlation coecients!) Consistency: The association is consistent across diferent studies. Dose-response relationship: An increasing amount o the proposed cause increases the outcome’s severity or risk o occurrence. Sense: The causal explanation is theoretically plausible and compatible with current knowledge. Alternate explanations: Other plausible explanations have been ruled out. Experiment: The outcome can be inuenced by an appropriate intervention. Specicity: A single putative cause produces a specic efect. This one is probably less important or our purposes, especially given the multi-actorial nature o most o the behaviours speech and language therapists are dealing with.

cohort o 242 children represented a random 50% sample o all children attending year 2 (age 7) in language units across acr oss England.” The emotional mental health o this cohort is the health outcome o interest. It is compared with a matched group o young people without a history o specic language impairment to explore whether there may be a greater risk o negative mental health outcomes or young people who have the condition. Cohort designs are particularly useul or studying developmental changes across the liespan, such as to identiy the inuence o  early circumstances, or the negative longterm efects o a condition, on lie outcomes. An example might be language and literacy outcomes or children born very prematurely. However, they are expensive to do: outcomes may take a long time to occur so you need to ollow up the same group over a long period o time, it is hard to prevent loss o participants (attrition attrition)) which is bad or the integrity o  your results and, unless your cohort is very large indeed, it may be impossible to pick up enough people with a rare outcome to gain evidence or prognosis. 2. Case-controlled study In a case-controlled (or case-control) study, on the other hand, people who have the outcome o interest (cases (cases)) are identied and matched with people who do not (controls (controls).). For example, a case-controlled design or ContiRamsden & Botting’s outcome o interest – emotional mental health in adolescence – might be to recruit participants with poor emotional mental health. They would then investigate their current language skills and  / or their developmental developmental language history in comparison with a matched group with good emotional mental health. Case-controlled studies may be the only practical design or researching rare conditions or outcomes, but on their own they provide much weaker evidence o a causal relationship because there is much more risk o  systematic bias afecting the results. You need to ensure every participant is allocated correctly as a case or not, as any misallocation can prooundly inuence the results. The measures used to

determine who is a ‘case’ thereore need to be pretty bullet-proo. This can be particularly tricky with complex human behaviour such as communication or emotional mental health. At a recent conerence I attended, speech and language therapists debated whether they would identiy the same children as languagedelayed as the team studying a large Australian preschool birth cohort. The study was using a cut-of o 1.25 standard deviations below the mean or their their age on language testing. We concluded its ‘cases’ might include quite a ew o our ‘non-cases’. 3. Cross-sectional survey  The third main observational design is the cross-sectional survey. A representative sample o the population o interest (clients, practitioners, relatives) is interviewed, examined or otherwise studied to gain inormation on a question, such as, “How many children entering primary school have poor vocabulary?” or, “What inuences speech and language therapy intervention or adults with autism and learning disability?“ or, “What do care staf in residential homes know about aphasia?” The data or crosssectional studies are collected at a single point in time. However, the study may include retrospective inormation. An example would be, in a survey o knowledge and skills or making inormation accessible or people with learning disability, asking support staf whether they had ever received any ormal training on making inormation accessible. Surveys can be relatively cheap and easy to do, but there are even more potential challenges to the integrity o the data, so it is not an appropriate design or answering causal questions. I have prepared a separate ramework tool or or surveys that use use questionnaires, which is available at www. speechmag.com/Members/CASLT. For observational study results, there are ways to evaluate how robust the evidence is or inerring causality. causali ty. Remember, just because you have established an association between two actors, this does not allow you to assume a causal relationship. In terms o the numbers, a correlation coecient such as Pearson’s r or

Spearman’s rho only indicates the presence or absence and direction o any association between the variables being measured. Hill’s (1965) criteria or causation (gure 1) were originally designed or epidemiological studies but have been widely quoted and so may pop up in authors’ discussion o the results o their observational studies. Here is an appropriat appropriately ely cautious conclusion about causality rom the Conti-Ramsden & Botting (2007) study: “Our data show a clear increased risk or this population as they near adulthood compared to peers, even when concurrent language and cognition are accounted or. This nding replicates other studies that have shown raised prevalence o psychiatric diculties in those with communication communicat ion impairments … or increased language impairment in children reerred psychiatrically … However, the association has oten been assumed to be causal in that either long-term language impairment may lead to (or exacerbate) wider diculties or psychiatric impairment may constrain communication skill. Nonetheless, it needs to be noted that the majority o adolescents with SLI in our study did not appear to sufer rom emotional problems” (p. 522).

Appraisal  The reporting o observational observational studies in peerpeerreviewed journals has been inuenced by the STROBE statement (von Elm et al., 2008). Like the Bradord Hill criteria (gure 1), this was originally devised to improve reporting o epidemiological research but it has been extended to other areas. Although designed or authors, it may provide some guidance or readers too. Observational studies are perhaps less common in speech and language therapy literature, so I have ound it helpul to have a tool that encapsulates all the main observational designs rather than trying to match separate tools to cohort, case-controlled and cross-sectional studies. I developed the ollowing appraisal ramework  or speech and language therapists rom relevant CASP tools (PHRU, 2006) and the STROBE statement (combined checklist).

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2011

19

JOURNAL CLUB 5 Question 1: Did the study address a clearly ocused issue?

Which population was studied? Which risk  actors or outcomes were investigated? Did the study try to detect a benecial or harmul efect? Is the underlying issue one o causation? Try ormulating the reviewers’ stated aims into a research question i they have not done so explicitly in the article. Is this question important or your clinical practice? Question 2: Was the choice o design appropriate? Is an observational design an appropriate way o answering the research question under the circumstances? Remember that a group intervention study is a more powerul design or demonstrating causality. Try to work out whether this is a cohort, case-controlled or cross-sectional study. For a cohort study, the participants should have been recruited beore the outcome o interest has occurred, and the cohort should have something in common (though this can be a very general characteristic, such as being born in Scotland in 2005, or a more specic one like being a sibling o a child with autism spectrum disorder). For a case-controlled study, ‘cases’ and ‘controls’ are identied at the outset o the study, criteria or ‘caseness’ are crucial or quality control, and the outcome o interest should be rare or harmul. Crosssectional surveys are probably the easiest to spot, since we meet them regularly in everyday lie. For surveys, sampling methods which ensure that the participants represent adequately the population o interest are very important or quality control.

In general, you are looking or selection bias which might compromise the extent to which the ndings can be generalised (external validity). Were participants representative o a clearly dened and clinically relevant population? Appraise the eligibility (inclusion and exclusion) criteria, the sources and methods o selection o participants and how cohorts were ollowed up. The selection method should be systematic – explicit, reliable and replicable - especially or a casecontrolled study, where it is crucial that there is no misallocation o cases. Scrutinise also the way that controls have been selected. Are they matched, populationpopulationbased or randomly selected, and is the rationale justied? I controls were matched, were the matching criteria appropriate? Authors should also provide inormation to allow you to assess whether those who were invited to participate but declined could be diferent in any important way rom the study participants. Potential controls are perhaps more likely to decline or ignore invitations to participate, so this may be even more important or this group. How many participants were there, was there a rationale or this and were the numbers sucient to support generalisation o the ndings? I the study is asking, “How many people have…” you need to think whether the sampling is o newly identied (incidence) or o cases across the whole population (prevalence) (see gure 2), and which would provide a more appropriate answer to the research question.

Question 5: Has there been adequate attention to conounding?

Figure 2 Incidence or Prevalence? iNC NCidence idence is the number o  New Cases in a given time span Prevalence is the Proportion o cases in the Population

Question 4: Were phenomena measured enough to minimise bias?

accurately

Question 3: Were participants recruited in an acceptable way?

You should be given enough inormation to assess how well all the phenomena involved have been assessed or otherwise measured, both actors (cohort characteristics or caseness criteria) and outcomes. Are denitions clear enough? Were measurements subjective or

20

objective, and do they measure what they are supposed to measure? Externally validated measures, like ormal tests, need less supporting inormation than measures developed or the purposes o the study. Some outcomes may take a long l ong time to occur, so was the timerame o the study long enough to assess this accurately or all participants? Moreover, the participants who are lost to ollow-up may have diferent outcomes rom those who were available, so attrition rates need to be given and their potential impact discussed. A owchart o recruitment and ollow-up schedule, indicating attrition numbers, can be really helpul or long-term studies. As in intervention trials, the study method should minimise the possibility o perormance bias by employing similar measurement methods or both cases and controls, and by blinding those undertaking the assessments to participants’ status wherever easible.

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2011

Conounding is the inuence o unoreseen actors. Check which actors have been considered and list any you think might be important that the authors seem to have overlooked. How, i at all, have the researchers taken account o the conounding actors in the design and/or analysis? Have a look in the data analysis section or evidence they have used statistical techniques such as modelling, regression or sensitivity analysis to make adjustments or conounding actors. Here is some relevant wording rom the Conti-Ramsden & Botting (2007) article: “… “… all the analyses above comparing those with SLI and those with NLD [no language disorder] remained unchanged ater controlling or gender…” Question 6: What are the results o this study? As or intervention studies, it helps to try to sum up the bottom-line result o the study in one sentence – this also helps to

JOURNAL CLUB 5 ensure you’ve got it straight enough in your own head to be able to communicate the gist o your appraisal to others. Consider whether the analysis appears appropriate to the design. Bear in mind that an observational study can only demonstrate associations: the presence or absence o  an association, its strength (weak/strong correlation) and direction (positive = one goes up, so does the other; negative = one goes up, the other goes down). How are the results expressed? Begin by having a look at their descriptive statistics such as the numbers or proportion o  participants with a given outcome, or tables o  average diferences. Inerential statistics draw conclusions rather than simply describing.  These may includ include e analy analysis sis o table tabless o  correlations or o measures o diference (between groups, as in a randomised controlled trial), such as analysis o variance (ANOVA), modelling, regression or sensitivity analysis. It might help to think o the study ‘variance’ as all the measured diferences amongst the participants. You then use your statistical analysis to try to make sense o this variance. The more o the variance accounted or in the end, the better the evidence is that the study has captured the strongest - and thereore potentially causal - actors. You have already had a look or evidence that conounding actors were considered. Are the numerical results adjusted or conounding? Take into account the list you made o overlooked conounding actors and consider i conounding could still explain an important part o the results? How did they evaluate the efect o individuals reusing to participate and did they adjust the overall results accordingly? Did this make much diference? For the phenomena o interest, how large and how meaningul is this size o result? (Continue your practice in NOT glossing over the sections with the p-values and condence intervals!) And, o course, do the results answer the study’s questions?

bias, conounding, or even chance (especially i the numbers o participants was small). For studies drawing causal conclusions, run them through the causality criteria in gure 1 (p.19).

think that the study participants and setting may be very diferent rom your own caseload? Can you quantiy the potential local benets and / or harms?

Question 8: Do the results o this study t with other available evidence?

Question 10: Should policy or practice change as a result o  the evidence contained in this study?

Consider evidence rom other studies, o  all types, or consistency o ndings. A wellconducted systematic review would be particularly helpul. You You may need to conduct (or even commission) a literature review and appraise the quantity and quality o the available evidence beore you are able to assess this. Bear in mind costs and benets; the issue in question will have to be particularly important (specic, relevant, timely, and with resource implications) implications) or your service beore you decide to invest resources in a more comprehensive review o the evidence base in this area. Question 9: Can the results be applied to the local population?

Question 7: Are the ndings plausible?  To what exten  To extentt can we gener generalise alise these ndings? As in other appraisal rameworks you need to examine the detail o the study to determine whether there are important diferences between the context o the research study and your own context. Do you

Do you believe the ndings? As with any research, a big efect is hard to ignore, but this is only true i the study design and methods were o a high enough standard, so any aws you have identied need to be borne in mind.  The sorts sorts o things things to consider consider are whether whether the results could have been unduly inuenced by

As with the other appraisal tools, we should evaluate the study’s contribution to the evidence base or local service provision. Does this study have implications or my practice, or that o my colleagues or more widely? Is there a urther question to be asked, or more evidence needed, beore I can answer this question? Always bear in mind that an individual observational study rarely provides suciently robust evidence to recommend changes to clinical practice or decisionmaking. However, or certain questions, observational studies provide the only evidence we can access. Recommendations rom observational studies are always stronger when supported by other evidence. In a local journal club report on the ContiRamsden & Botting study (2007), the group concluded that the study added weight to a growing body o evidence showing a raised risk o negative mental health in young people with specic language impairment. However, since the majority o the group with specic language impairment did not have negative mental health symptoms, there is no simple, causal relationship between specic language impairment and mental health. There must be other actors at play. So, speech and language therapists need to be alert to the raised risk in young people with specic language impairment o anxiety and depression, either o which would have an impact on our clinical decision-making or on SLTP the success o our intervention interventions. s.  Jennifer  Jennif er Reid Reid is a consult consultant ant speec speech h and langu language age therapist with NHS Fife, email [email protected]. Cartoons are by Fran, www.francartoons.co.uk.

Reerences Bradord Hill, A. (1965) ‘The Environment and Disease: Association or Causation?’, Proceedings of  the Royal Society of Medicine 58, pp.295-300. Conti-Ramsden, G. & Botting, N. (2008) ‘Emotional health in adolescents with and without a  Journall of Child Psychology Psychology and Psychiatry  Psychiatry 49(5), history o specic language impairment (SLI)’,  Journa pp.516–525. Public Health Research Unit (2006) Critical Appraisal Skills Programme . Available at: http://www. phru.nhs.uk/Pages/PHD/CASP.htm (Accessed 29 July 2011). von Elm, E., Altman, D.G., Egger, M., Pocock, S.J., Gøtzsche, P.C., Vandenbroucke, J.P. (2008) ‘The Strengthening the Reporting o Observational Studies in Epidemiology (STROBE) statement:  Journal nal of of Clini Clinical cal Epi Epidem demiolo iology  gy 61(4), pp.344-349. guidelines or reporting observational studies’, Jour

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2011

21