Index of orthodontic treatment need applied to plaster models and to their three-dimensional digital equivalents

Materials and methods

The sample consisted of 30 sets of study casts with a range of malocclusions (IOTN categories 1–5) based on the Index of Orthodontic Treatment Need data and, in particular, the Dental Health Component. It was estimated that, at a 95% confidence level, 19 casts would provide the standard statistical power of 80% and 30 casts a power of 95%, which is desirable. The study models were all of good quality, accurately and uniformly trimmed with their bases parallel to the occlusal surface. The inclusion criteria were good quality models, all teeth having normal morphology, no visible attrition, no caries and no restorations affecting the mesio-distal or buccolingual crown diameter.

The plaster casts were professionally scanned by a digital model company using a 3Shape's R250^TM laser scanner with an overall scanning cycle time of approximately 7 minutes.

The three-dimensional equivalents were stored on a laptop computer and the principal investigator (PS) was instructed in the use of the 3Shape's OrthoAnalyzer^TM software and how to view the models on screen. 3Shape's OrthoAnalyzer^TM is a dedicated software package used for the analysis and planning of orthodontic treatments, using the data of patient's cases scanned with a 3Shape R250^TM 3D scanner.

The casts were initially scored on two occasions at least four weeks apart by one experienced and calibrated consultant orthodontist (NM) to set the ‘gold standard’.

This exercise provided two scores for each cast on two occasions (ie four scores for each model – two DHC and two AC.) The scores derived from this exercise were used for comparison with those of the four other examiners. These four specialists were chosen on the basis that they were experienced in the use of computers and had previously been calibrated in the use of IOTN by attending the calibration course in occlusal indices at Cardiff University (UK). The four examiners were each asked to examine the study casts on two separate occasions, at least four weeks apart. On a third and fourth occasion, they were asked to examine the digital model equivalents on the computer.

One of the strengths of the current study design is having four examiners, assuming that all four have good calibration (weighted kappa scores ≥0.6) for both inter- and intra-reliability. An additional feature of the study was the addition of two pairs of identical models, of which the examiners were not informed, bringing the total number of models actually examined to 32. This allowed for some descriptive analysis of the extent to which these duplicates were scored differently. The duplicate pairs were not presented consecutively, although they were examined only minutes apart. This aspect of the design meant that the true sample size was 30 distinct models.

For the Dental Health and Aesthetic Components on the IOTN, the weighted kappa statistic¹⁶ was used, with scores >0.6 regarded as good agreement and scores >0.8 regarded as very high (or almost perfect) concordance.¹⁷ Data were initially entered into an Excel file before being analysed in SAS v9.1 (SAS Institute, NC, USA) for cross-tabulation and calculation of kappa statistics.

Results

Discussion

This study has shown that good inter-examiner reproducibility of the Dental Health Component of IOTN scores for the study casts was found between the four examiners and the gold standard (weighted kappa range 0.66–0.90) with good reproducibility for the Aesthetic Component of IOTN scores (weighted kappa range 0.64–0.72). When intra-examiner reproducibility for the study casts was examined, this was found to be good to almost perfect for the DHC (weighted kappa range 0.75–0.91) and good for the AC (weighted kappa range 0.72–0.80). Intra-examiner reproducibility for the digital models was found to be very good to almost perfect for the DHC (0.87–0.94) and good to almost perfect for the AC (0.77–0.92). When reproducibility of the scores between the plaster casts and their digital equivalents were examined, the weighted kappa scores for the DHC indicated a good to almost perfect level of agreement (0.71–0.91); for the AC scores, the range of agreement was moderate to very good (0.54–0.86).

There is no evidence to show that any of the examiners performed significantly better than any of the other examiners and inferences can be made in addition to each kappa score. For example, from Examiner C's initial DHC examinations of plaster versus digital casts, their estimate of reliability obtained from the kappa score for the study sample was 0.74, but we can be confident that, if asked to examine different sets of 30 casts repeatedly, then the interval 0.59–0.89 would include a kappa score equal to Examiner C's true measure of reliability on 95% of occasions.

There are essentially two factors that influenced the standard error:

In most cases, the sample size was more than adequate to show clear results, although a minority of the results may have benefited from having a larger sample size. The sample size was calculated purely on the variability of the gold standard's intra-reliability (because these were the only data available when designing the study). As it turned out, the gold standard had higher reliability than any of the other examiners and the variability of this reliability was greater than Examiner A, about the same as Examiner B and less than Examiner C and D. In other words, NM was the rater with the highest reproducibility, but Examiner A is the least variable. However, NM's variability was still relatively low, so the overall variability of the group of examiners may have been underestimated.

Aesthetic Component reliability scores in general are liable to be slightly lower than DHC scores because it uses a 10-point scale rather than a 5-point scale. However, it would be expected that the difference in the size of the scale would account for only a slight difference in the reliability scores, given that weighted kappa scores were used. The lower reliability scores for AC compared with DHC generally, together with the higher AC reliability in digital models, suggests that reliable AC measurements may be harder to achieve with plaster models compared with digital models.

Both previous investigations into the application of IOTN to study models and alternative media found similar results. Buchanan et al¹⁸ found a high level of examiner reliability when the Dental Health Component of the IOTN was applied clinically to a group of patients and later to study models and photographs taken at that same visit. Aesthetic Component scores, however, showed poor reproducibility. Two-dimensional pictures of study models on a computer screen were investigated by Mok et al¹⁹ who found a high level of agreement for the DHC (kappa = 0.79) and a lower agreement for the AC (kappa = 0.56) between measurements obtained from orthodontic casts and their images. A study by Veenema et al²⁰ concluded that differences in ICON score between plaster and digital models resulted in mostly statistically insignificant values (P values ranging from 0.07–0.19), except for one observer in one sample.

Conclusions

This study provides no evidence to reject the null hypothesis. There does not appear to be any difference between expert rating of plaster and digital models in either component of the IOTN, although less agreement has been observed between the methods for the Aesthetic Component.