From Volume 5, Issue 2, April 2012 | Pages 57-60
There is an increasing volume of research undertaken within orthodontics and with this comes a need to evaluate what is available. This short series of three articles aims to help the orthodontist revise basic concepts of critical appraisal and pertinent statistics.
Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.1 Critical appraisal skills can be applied to almost any published research study. There are many resources available to assist with critical appraisal including a variety of checklists. However, for a checklist to be effective, one has to have a good understanding of the points to be considered. Therefore, an understanding of the research process, the rationale, strengths and weaknesses of different study designs and statistical and clinical inference are all essential to enable the reader to appraise effectively.
Scientific research usually involves undertaking a methodological study in order to answer a specific question. The introduction of a study should ideally summarize the literature, highlighting any shortcomings and clearly defining the questions that the study is intending to answer (its aims). These may include a hypothesis that will be tested, such as whether a treatment is effective (Box 1).
Assessment of a piece of research should always consider its validity, which can be split into two types:
Bias may be defined as a systematic (regular) difference between the results from the study and the true state of affairs. This is a very important concept to consider when critically appraising a paper. There are many different types of bias, which include selection bias, measurement bias and publication bias. It can invalidate the study results if it is not taken into account properly when drawing conclusions.
The Hawthorne effect is a type of bias in which subjects in a study change their behaviour, usually positively, purely in response to being observed. Some types of subjects may be more likely to be lost to follow-up and this can also bias the results.
The word ‘confounding’ originates from the latin ‘confundere’ which means ‘to mix together’. Confounding factors are those which make it appear as though there is a direct relationship between the exposure and the outcome (positive confounding factors) or which mask an association which is truly present (negative confounding factors): that is, they ‘get in the way’ of the comparison between groups that are being investigated. Confounding occurs when there are differences in factors that we are not interested in directly between the groups being compared that also happen to affect the outcome we are interested in assessing (Box 3).
It is helpful to be aware of ways that researchers have tried to reduce bias and confounding at the design stage or in the analyses by:
There are many different types of study design and the appropriate one to use depends on the type of the research question being posed. Some designs yield more valid results than others. Sometimes the ideal design cannot be undertaken because of a lack of resources, ethical or time restrictions. Even if a study is not optimal for a stated research question it can often still provide valuable information which will help to answer the question posed.
Published checklists to aid critical appraisal are generally categorized according to study type. Hence, since the paper may not explicitly state the study type (or it may even be wrongly labelled), it is important to be able to distinguish different types of study design to select the correct checklist to use.
Although there are several ways to classify studies, it is often helpful to consider how the study is designed with respect to time. A study may be prospective (often considered advantageous as it can be planned, looking forward in time) or retrospective (looking at past events). A cross-sectional study looks at a single timepoint, whilst a longitudinal one extends over a period of time.
Although prospective studies are generally given more credence, it is worth noting that retrospective studies can still be valuable, although we do need to be aware of the potential for recall bias. Data that has been collected specifically for the research study and not based on retrospective recall will generally be more reliable.
It is also worth noting that any study may have both prospective and retrospective elements.
An observational study reports what already exists, but in contrast an interventional one can be considered as experimental. It is possible to ascribe causality directly from a well-designed experimental study but not from even the most perfect observational study.
A survey describes a snapshot in time and so shows how things are at that timepoint.
Cohort studies are observational, longitudinal, prospective studies which follow groups over time. Note that case reports and case series may also be regarded as uncontrolled cohort studies. Their prospective nature means that they can be used to assess the true sequence of events and outcomes. Other applications of cohort studies are to measure incidence or risk of disease and assess prognosis or infer (but not prove) causation. Since short durations of follow-up may not be long enough to identify meaningful changes, cohort studies may be costly, requiring individuals to be followed over long time spans. Furthermore, very large sample sizes are required for rare outcomes to be observed. Bias can be introduced in the form of the healthy entrant effect since the design requires individuals to be disease-free at the start and hence they may be unrepresentative of the target population. Cohort studies may also be susceptible to the Hawthorne effect and bias introduced by losses to follow-up.
Case-control studies are observational, longitudinal, retrospective studies that can help reveal differences between groups. They are relatively quick, cheap and easy to perform and may be used to study rare conditions. A wide range of risk factors can be investigated in the same study. They can also be used to infer (but not prove) causation. As they are retrospective, there is no loss to follow-up. However, case-control studies are not suitable if the risk factor is rare and are also susceptible to recall bias. Furthermore, the onset of the disease may have preceded exposure to a potential risk factor and, since recording is retrospective, this may not be apparent.
Clinical trials are prospective (therefore can be planned) and they generally evaluate an intervention or therapy. They allow a single variable to be evaluated by comparing groups, usually using an untreated control group or a group given standard therapy. They may investigate either efficacy (the impact of interventions under optimal conditions, showing that internal validity is present) or effectiveness (the impact of interventions under ordinary conditions, showing that external validity is present). Randomization is the preferred means of allocating individuals to treatment or control groups as the chances of bias and confounding are reduced. Such trials are ethically constrained if potentially damaging factors are being investigated. They may also be complicated to co-ordinate and therefore relatively costly. However, the gain in validity usually warrants the additional cost and effort where feasible (Box 4).
The following terms are in common parlance in the literature and their relevance is explained below:
The advantages of a systematic review include increased power and precision in addressing the research question. A comprehensive objective search strategy is used to find all relevant studies addressing a specified research question. The source studies are methodically located, appraised and synthesized to provide as reliable an overview as possible. Ideally, there should be at least two reviewers who are blind to study authorship (Box 6).
The Cochrane Collaboration6 is an independent, international, not-for-profit organization, dedicated to making up-to-date, accurate information about the effects of healthcare readily available worldwide. The major product of the collaboration is the Cochrane database of systematic reviews, published quarterly as part of the Cochrane library. They are prepared by volunteer healthcare professionals and overseen by editorial teams. Ideally, Cochrane reviews are based on the inclusion of randomized controlled trials and, if there is a lack of such data, it is difficult to draw any meaningful conclusions.
A meta-analysis6 uses statistical methods to combine the results of different studies with the aim of integrating their findings, pooling data and identifying the overall trend of results. In this way, the results of a number of small but similar studies can be combined to achieve a large enough sample size to detect an effect and the power is increased. They are often associated with a systematic review.
The studies are weighted according to how much information is provided. Those with more participants, more events and lower variance carry more weight. Subgroup analyses can be undertaken post hoc if it is suspected that certain features may alter the effect of an intervention (such as gender or disease subtype). Sensitivity analyses can also be carried out post hoc whereby the meta-analysis is repeated without including lower quality trials. Ultimately, a single summary statistic is calculated to represent the treatment effect found in each study and these are then combined to give an overall measure. Forest plots are usually used to display the results (Table 1, Box 7).
