A very brief guide on how to read a report of an orthodontic clinical trial

In this short article, we hope to outline some tips to assist with efficient and effective interpretation of a randomized controlled trial. We have based this on our own experience of interpreting the orthodontic literature. However, we do not intend this to be an extensive analysis of the critical appraisal literature. In-depth papers and texts documenting both the interpretation and appraisal of clinical trials have already been published.^1,2

It is widely acknowledged that the randomized controlled trial provides us with high levels of scientific evidence. Trials are designed to minimize bias and the effects of confounding. It is important to note that non-randomized studies may be considered complementary, being better suited to answer certain questions, while also potentially informing trial design and feasibility.^3,4 However, a trial is only as good as a paper that reports on its methods and findings, with research waste pervasive across the biomedical literature.⁵ If the paper is not written clearly, and does not include all the information we need, its value may be limited. In this respect, research methodologists and statisticians have developed an accepted formula for presenting clinical trial methods, which has been outlined in the CONSORT Statement.⁶ This is an excellent reporting guide, but it is also a great learning resource. An orthodontic version of this has been published to help with orthodontic aspects of trial reporting and interpretation.⁷ Authors are encouraged to include all of these points in a trial report to ensure that the methodology is transparent, while also allowing detailed appraisal. We have selected several of these points as a framework for ‘how to read an orthodontic clinical trial.’

The abstract and how to use it

Limited interaction with research papers has prompted a perhaps inappropriate reliance on research abstracts. Moreover, this has also spawned the development of structured abstracts that are designed to address key areas of study design, results and interpretation.^8,9 When read in isolation, the truncated nature of the abstract may risk the misinterpretation of research findings.

It is accepted that the abstract is not always accurate, and may be inadequately reported.¹⁰ This may happen because it is frequently the last part of a paper to be written and can be something of an afterthought. Furthermore, investigators tend to write abstracts to attract the readers' attention and may ‘spin’ or overstate their findings.¹¹ We are as guilty as anyone of doing this.

The following simple suggestions may assist you in dealing with research abstracts:

• First, it is sensible to avoid reading the abstract in isolation. You already know that you are interested in the paper from your initial search for information. So why not go straight into reading the paper?
• If you read an abstract, look for structured abstracts because these are laid out with required headings. These make reading the abstract easier, while also potentially leading to a more complete summary of the research study.
• Ascertain whether the methods are clear and are related to the title.
• If you expect a randomized controlled trial and see that the abstract does not contain information on randomization, this may not be a clinical trial.¹²
• Are the results clear? Do not assume that all the results are included in the abstract.
• Are the conclusions clear? Do not assume that they necessarily reflect the findings at the end of the paper.

In summary, the abstract gives only the briefest information about the study. If you want to critically appraise the paper and use it to inform your practice, reading the paper is important. Just reading abstracts is not good practice and opens you up to misinterpretation.

The introduction

The introduction should be brief and clear. It is important that the rationale for the study is obvious and supports the aims and the hypothesis.

What's the aim?

This is a statement about the primary focus of the paper. It outlines what the authors hope to achieve at the end of the research project. They are usually broad. So, let's look at a poor example:

‘We aim to assess whether magic brackets are better than conventional ones.’

This is very vague, and a better aim would be:

‘To assess the effects of magic and conventional brackets on the overall duration of orthodontic treatment.’

The objectives

The objectives are more specific and include the actions required to meet the aims. They are clear statements that define how the aims will be assessed. It is usual to have one or two primary objectives. A poor example for our study is:

‘To find out if self-ligating brackets reduce the length of orthodontic treatment.’

A better one is:

‘To compare the effects of self-ligating brackets, relative to conventional brackets on the duration of orthodontic treatment in months and on the number of required visits for orthodontic patients aged between 12 and 16 years at the start of treatment.’

This may seem a little pedantic. But, clear aims and objectives are essential in our reading of a paper.

In search of the lost hypothesis

You should look for the hypothesis just after the aims and objectives. The hypothesis should be unambiguous and describe a relationship between two or more of the variables that the authors are testing in the paper. You should look to see if it has the following qualities:

• Is clear and specific
• Is relevant to the research question
• Is testable. This concept is crucial to the relevance of the research paper.

In clinical research, authors often state the null hypothesis. This is because statistical tests enable us to accept or reject the null hypothesis. It is helpful to look at some good and poor examples of hypotheses. Let's go back to the ‘self-ligating bracket’ study. An inadequate null hypothesis would be:

‘Self-ligating brackets are not better than conventional brackets.’

This statement does not satisfy the criteria above. A better one would be:

‘There is no difference in treatment duration in months between self-ligating and conventional brackets when treating 11–16-year-old children.’

As with the aims, this information is important for the interpretation of the paper. If it is not clear, there is a danger that the report will simply drift, and we will achieve nothing by reading it.

The method

Participants

It is essential that the authors clearly describe the participants. Ultimately, we need to know whether the findings from the study are relevant to our practice. For example, studies carried out in university settings involving non-paying patients treated by students may be less applicable to seasoned practitioners in private offices. Similarly, one would have to question the validity (and generalizability) of undertaking a clinical trial relating to growth modification in adults!

Sample size calculation

Often, this is seen as the preserve of statisticians with mathematical minds that intimidate us; however, no randomized controlled trial is complete without one. Calculations should be undertaken before commencement, and be based on realistic estimates of what we feel are clinically significant between-group differences. Adequate numbers are essential to ensure that the study has sufficient power to identify meaningful differences and to ensure that the investigation is credible.

Statistical methods

The key message is that statistics should be focused and finite. Specifically, we should design our statistical approach to answer our key question (eg ‘Does the use of bracket A result in more efficient alignment of the lower anteriors than Bracket B?’) rather than undertaking a battery of analyses. The latter risks spurious false positive findings, whereby increasing the number of analyses increases the risks of obtaining positive (statistically significant) results. Indeed, it has been suggested that ‘if you torture the data for long enough, it will confess to anything.’

Randomization and concealment

Investigators use these methods to minimize selection bias in a trial. In effect, they minimize the risk that operators may influence the treatment allocation to participants. We will not go into the theory behind the methods, but we shall try to simply explain these concepts. The most crucial point is that the authors should clearly state the randomization and allocation concealment methods. If they do not do this, you may conclude that the study is at risk of bias. Let's look at randomization first.

Investigators randomize treatments to produce an unpredictable sequence of allocations. As a result, they allocate treatment by chance. Significantly, it is not influenced by the operator. There are several methods of randomization.

• Simple: This is equivalent to fair coin tossing. However, the main drawback is that the group sizes may be unbalanced. Furthermore, there may be differences in gender between groups.

There are two solutions to this problem, which are:

• Permuted block randomization: Investigators do this to keep the sizes of the treatment groups similar.
• Stratified randomization: This is done to make sure that the covariates are balanced between the group. For example, gender.

Again, the authors should explain the method of randomization. If they fail to make this clear, the study may be subject to bias.

Investigators use allocation concealment to hide a participants' allocation from other people. In other words, it prevents investigators from choosing or being suspected of selecting interventions. They may do this because they have a biased preference to treat an individual participant in a certain way.

Investigators commonly use two methods of allocation concealment in orthodontic trials. The first method involves sequentially numbered, opaque, sealed envelopes. The investigator discovers the allocation when they open the envelope. Unfortunately, sealed envelopes are not totally secure. Investigators can, for example, open the envelopes or hold them up to the light. As a result, it has been suggested that sealed envelopes are not acceptable.¹³

An alternative is a central allocation system. In fact, most well-run clinical trials use a method of central allocation. In this method, the investigators send the participants' details to an external study team, and this team randomly allocates the patient to treatment. The allocation is then fed back to the investigator. There are many web-based methods of doing this. This is much more secure and minimizes the risk of selection bias.

Appropriate outcomes

The use of appropriate outcomes is crucial to a research study with a minimum recommended set of orthodontic trial outcomes recently developed.^14,15 If the investigators choose to measure outcomes that are not relevant, this will detract from the applicability of the findings. This is a problem on many levels. Not least because clinical research is often expensive and arduous. The selection of inappropriate outcomes is a frequent cause of waste in research.

Irrelevant outcomes

The use of irrelevant and obscure outcomes is sadly true for orthodontic research. This is typified by the isolated use of some traditional orthodontic measures; for example, cephalometric analysis. These measurements are often misused, and investigators may generate many related measurements and run simple univariate tests across these. There is a risk that the significant results will be focused upon distorting the interpretation of the results. Furthermore, investigators can fixate on minor differences of less than 1 degree, which may be clinically meaningless. Unfortunately, this practice continues to abound with the development of CBCT, which enables investigators to measure ad infinitum to the most minor degree.

What should you look for?

So, when you are reading a paper, we think you should consider the following questions in relation to outcomes:

• Is the outcome relevant to my clinical practice? For example, in a study looking at the effectiveness of a new bracket, do the authors investigate the overall duration of treatment? Other clinically relevant outcomes are occlusal index scores, harm due to treatment, socio-psychological measures and appearance of the teeth and face.
• Does the outcome have meaning to our patients? For example, discomfort, the appearance of appliances and effects on self-esteem. Ideally, all outcomes should have value to patients. If they do not, we may be missing some of the critical impacts of treatment.
• Is the effect size that the authors report clinically relevant? This is a significant problem in orthodontics. It does not matter what outcome the authors select. If there is no indication of the clinical relevance of the effect size, the paper may be of little added value.

In summary, if the outcomes lack relevance to your practice or to your patients, or we are measuring minute differences, you should start to question the value of the paper.

The results

So, you have worked your way through the introduction and complexities of the method. Now you are ready for the results. We always hope that the results are clear and understandable. As a result, we look for explicit statements of the findings and simple tables. Things to look for:

• Is the presentation of the results clear, and do the authors stress the essential findings?
• Is the primary outcome presented first? Or have the authors led with one of the secondary outcomes? Authors sometimes do this when the secondary outcomes reveal differences that are either interesting or of statistical significance.¹⁶ When you see this, you should concentrate on the primary outcome results.
• Do they present data on only a few outcomes? Again, this is good practice because this suggests that the study is focused. On the other hand, you should be deeply suspicious of massive data tables reporting many cephalometric variables with multiple statistical tests. This may point to data dredging and again suggests a lack of focus and the risk of superfluous or misleading findings.
• Do the authors present between-group differences including 95% confidence intervals? This helps you to appreciate the size of any effect related to one intervention rather than focusing on a binary indication of statistical significance (or otherwise).
• Do the tables and text include P values and not merely state ‘non-significant’ or ‘significant’?
• Do the authors state whether their findings are clinically significant? We can decide if this is the case for most orthodontic outcomes. However, this may be more complicated for some studies focusing on airway dimensions and sleep-disordered breathing.

By considering these six questions, you may be able to judge whether the results are important or relevant to your clinical decision making.

What can we conclude?

This is a simple step. If you have read and interpreted the paper carefully, the authors' conclusions may be of little added value. Instead, you will have reached your own opinion concerning the value of the research, whether these apply to your clinical practice and whether your practice should be amended accordingly.

Authors are known to base their conclusions on their own interpretation of the findings, which can be misleading for readers. Therefore, as per the abstract, it may be unwise to read the conclusions without reading the paper in more depth.¹⁷

We appreciate that this summary constitutes merely a brief overview of the pitfalls to be wary of when reading and interpreting clinical trial reports. However, we hope that you have found it helpful, that it stimulates further reading and equips you to read clinical trials with more confidence and clarity. Enjoy reading orthodontic trials!