JDE
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

J Dent Educ. 68(12): 1228-1234 2004
© 2004 American Dental Education Association
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Kournetas, N.
Right arrow Articles by Geis-Gerstorfer, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Kournetas, N.
Right arrow Articles by Geis-Gerstorfer, J.

Educational Methodologies

Assessing the Reliability of a Digital Preparation Assistant System Used in Dental Education

Nikos Kournetas, D.D.S.; Berthold Jaeger, D.D.S., Dr.Med.Dent.; Detlef Axmann, Dr.Rer.Nat.; Martin Groten, D.D.S., Dr.Med.Dent.; Stefan Lachmann, D.D.S., Dr.Med.Dent.; Heiner Weber, D.D.S., Ph.D.; Juergen Geis-Gerstorfer, Dr.Rer.Nat., Dipl.Ing.

Key words: preclinical dental education, PREPassist, accuracy, repeatability, reproducibility, measurement error, limits of agreement

Submitted for publication 05/17/04; accepted 09/23/04


   Abstract
Top
 Abstract
Materials and Methods
 Results
 Discussion
 Conclusions
 References
 
Use of a digital preparation assistant system may improve considerably the quality of preclinical dental education, provided the system works reliably. Thus, the purpose of this pilot study was to quantitatively assess the reliability of a new preclinical digital preparation assistant system (PREPassist, KaVo, Germany). The system was used to repeatedly scan four different unprepared and four different prepared teeth both with and without repositioning. Corresponding measurements were made to quantify accuracy, repeatability, and reproducibility. This was done by estimating the measurement error. Based on this estimation, respective limits of agreement were calculated. We used these ranges, along with assessments of the measurement’s accuracy, to judge whether the results satisfy our expectations for clinically acceptable measurements. For preclinical laboratory instruction, the results indicate an acceptable accuracy (mean accuracy of 89 µm) of the measurements. This assessment applies as well to repeatability, given by the range of the respective limits of agreement (range <200 µm). However, in the case of reproducibility, the limits revealed discrepancies of practical importance (range >200 µm). Reproducibility of tooth repositioning in the available mounting device is unacceptable for preclinical laboratory instruction because of the observed range >200 µm. Thus, there is a need for the manufacture of new and more reliable mounting devices because reproducibility procedures are mostly encountered in preclinical instruction in restorative techniques. In contrast to reproducibility, accuracy and repeatability are acceptable for practical purposes. Balancing advantages and disadvantages, we conclude that, in general, the PREPassist system delivers reliable results.

The use of dental simulation units is intended to support life-like clinical training conditions in restorative dentistry.1 However, their use does not necessarily result in effective development of clinical preparation skills in the trainee. The concerns of dental preclinical instruction in restorative techniques, in particular, focus on the expected enhancement of the self-evaluation and self-assessment capacity of the trainee.

Additionally, we have observed that infrequent and inconsistent preparation training during the pre-clinical curriculum may sometimes result in gradual degradation of both understanding and application of the principles of preparation. When feedback is based only on unassisted visualization, the student has to attempt to produce an optimal preparation based solely on the instructor’s guidelines.

Digital preparation assistant systems were introduced during the last few years to enhance the student’s learning process. With such systems, as for example the PREPassist system (Kavo, Germany), students can judge the quality of their own preparations, using a computer (Figure 1Go). The system creates visualizations of different preparations of resin teeth by use of a CCD camera. The software consists of four windows. The upper part shows the original tooth and the reference preparation (instructor), while the lower is designated for the trainee. The results of the scanning process are shown in computer images while results from measurements of points, distances, and angles are presented as cross-section slices along with associated numeric values. Individual criteria can be provided by the instructor.2 Use of this system could lead to a more effective, more objective, and eventually more efficient learning of operative skills.



View larger version (106K):
[in this window]
[in a new window]
  		Figure 1. The user interface of the PREPassist system
Difference column indicates the deviations between the trainee preparation and the original tooth. Trainee: the student’s preparation; instructor: the "ideal" preparation according to the Tuebingen criteria.

 
Despite increasing reports that describe digital preparation assistant systems, there is only anecdotal evidence concerning the reliability of such systems. Clearly, the quality of the digital preparation training depends directly on the reliability of the system, assessed by suitable measurements. An assessment of the reliability of such measurements is a necessary condition of scientific practice. The value of such an assessment for any system depends on the practical consequences of its usage. Thus, an advanced technological instrument such as the 3-D optical scanner (PREPassist), along with supporting software, should be thoroughly tested before its implementation in preclinical education.

As a generic term, "reliability of measurement" is comprised of accuracy, repeatability, and reproducibility. Following the International Vocabulary of Basic and General Terms in Metrology,3 accuracy of the scan measurement expresses the closeness between the result of the subject’s measurement by the scan system and the true value of it. The term "true value" or "value" reflects the idealized concept of the perfect (error-free) assessment of a measurand, which is, by nature, indeterminable (pp. 32, 41).4 Further, repeatability expresses the closeness of the results of two consecutive scans of the same tooth without removing it. Finally, reproducibility expresses the closeness of the results of two consecutive scans of the same tooth after removal and repositioning.

The purpose of this pilot study was to assess quantitatively the reliability of measurements of the PREPassist system by assessing the accuracy and repeatability of the scanning process and its reproducibility when repositioning the teeth to be scanned—a critical component of preclinical instruction in restorative techniques.

Criteria for reliability were deviations in the order of the magnitude that might be still perceived by a dental student or a dental clinician during a preparation process. These deviations are supposed to lie between 100 and 200 µm. We consider that distinguishing smaller deviations simply by eye is unrealistic.


   Materials and Methods
Top
 Abstract
 Materials and Methods
Results
 Discussion
 Conclusions
 References
 
The qualitative concept of reliability was operationalized with a pilot study. For this purpose, four different unprepared and four different prepared teeth (FDI codes 13, 14, 25, 26) were scanned repeatedly, both with and without being repositioned. The preparations were solely extracoronal (preparation for PFM crown for 13, 14, 25 and for cast crown for 26). Under these conditions, each tooth was scanned three times. Thus, a scan was repeated two times. Measurements were generated by comparing the second scan S2 with the first scan S1 and the third scan S3 with the first scan S1. These comparisons were quantified by measuring corresponding deviations D1 = S2 – S1 and D2 = S3 – S1 in µm on each tooth at twenty randomly selected points in three two-dimensional cuts mesio-distal and three two-dimensional cuts lingual-buccal. D1 and D2 were measured by the PREPassist software. In particular, the system is programmed to superimpose each repeated scan over the first scan and then measure the corresponding deviations. Because of this system-specific property we analyzed D1 and D2. The scheme of the measurement procedure is shown in Figure 2Go.



View larger version (20K):
[in this window]
[in a new window]
  		Figure 2. Schematic chart of the scans and measurement procedure

 
The measured deviations, both with and without repositioning the teeth, were used to quantify reproducibility and repeatability, respectively. For this purpose, the extent of these deviations was determined by condensing them into the so-called measurement error sw, estimated by the common within-subject standard deviation.5

In the case of accuracy, the deviation between a single measurement and its true value would be expected to be less than 1.96 x sw for 95 percent of measurements.5 In the case of repeatability/reproducibility, the difference between two repeated measurements of the same tooth would be expected to be less than 2.77 x sw for 95 percent of the measurements.5 Such frequency interpretations of sw are valid under normal distribution assumptions, which will be tacitly given; otherwise, 95 percent has to be substituted by 75 percent.

Moreover, as to repeatability/reproducibility, the given frequency interpretation assumes that the mean value &dmacr; of the differences is zero. Otherwise, some bias exists that questions assertions on repeatability and reproducibility, respectively. The so-called Bland and Altman plot can check bias and distribution of these differences.6 Here, D1 – D2 = S2 – S3 is plotted against 1/2 (D1 + D2) = 1/2 (S2 + S3) – S1 (see Figure 3Go). In this plot the bias estimated by &dmacr; is visualized by the line next to zero. The lines &dmacr; –2.77 x sw and &dmacr; + 2.77 x sw define the limits of agreement.6



View larger version (32K):
[in this window]
[in a new window]
  		Figure 3. Bland and Altman plots for repeatability/reproducibility for unprepared and prepared teeth

 
An estimate for the measurement error sw resulted from a one-way analysis of variance. The dependent variable was the measured deviation; the different measurement points formed the independent variable. The square root of the common within-subject variance, called the common within-subject standard deviation, provides an estimate for sw.5

Estimations were calculated separately considering repositioning and not repositioning the teeth and considering unprepared and prepared teeth. The statistical analysis was performed with the JMP statistical software system (JMP 5.0.1a 1989-2002 SAS Institute Inc.).


   Results
Top
 Abstract
 Materials and Methods
 Results
Discussion
 Conclusions
 References
 
Table 1Go lists estimates of measurement error sw, accuracy and limits of agreement with repositioning (reproducibility) and without repositioning (repeatability) the teeth for the unprepared and for the prepared teeth, respectively.


View this table:
[in this window]
[in a new window]
  		Table 1. Estimated measurement error, accuracy and reproducibility/repeatability, and bias given in µm for unprepared and prepared teeth
 
The magnitudes of sw for repeatability and reproducibility differ, considering the unprepared and prepared teeth. For example, sw of 68 µm was calculated for unprepared teeth after repositioning them, whereas sw was about 30 µm in the case of prepared teeth without repositioning them. These facts are visualized in Bland and Altman plots (Figure 3Go) which, moreover, exhibit all data.

Repeatability of about 100 µm for unprepared teeth results in 95 percent limits of agreement of –22 µm – 100 µm = –122 µm for the lower limit and of –22 µm + 100 µm = 78 µm for the upper limit. This means that deviations between the third scan and the first scan may be 122 µm below or 78 µm above deviations between the second scan and the first scan. In case of prepared teeth, corresponding deviations are bounded by –89 µm and 77 µm.

Considering reproducibility, the corresponding deviations are bounded by –158 µm and 218 µm for unprepared teeth and by –100 µm and 160 µm when teeth are prepared. Estimations of the bias in the reproducibility situation are at least twice as much as compared to those in the repeatability situation, as can be seen in Table 1Go.


   Discussion
Top
 Abstract
 Materials and Methods
 Results
 Discussion
Conclusions
 References
 
Generally, two methods are available to estimate accuracy of a measuring device. One comprises the measurement of the deviation of the measured results from a calibrated reference,7 whereas the other estimates accuracy through estimating the measurement error sw of repeated measurements of the same object.5 Because of the absence of a calibrated reference, we used the second method, which allowed us to calculate estimates for repeatability and reproducibility.

The mean accuracy for the system was estimated to be 89 µm, by averaging 133 µm, 92 µm, 71 µm, and 59 µm from Table 1Go. Thus, the system’s mean accuracy lies below the range of 100 µm to 200 µm, which we defined as the limit of deviations that can be detected by human vision. Clearly, on average the deviation between a single measurement and its true value would be expected to be less than 100 µm for 95 percent of measurements.5 Simply put: preparation amount measured by the PREPassist system is better than clinical assessment by eye, with a deviation of less than 100 µm from the true preparation amount.

Accuracy as well as repeatability/reproducibility is to be interpreted statistically in this context. The word "expected" expresses the long-run behavior of repeated measurements.

The equation repeatability/reproducibility = x accuracy5 relates the estimated quantities, leading to 95 percent limits of agreement. The clinical relevance of such limits has to be assessed by the user for each individual application,8 according to their feasibility in preparation practice. The criterion for such a decision must be a realistic difference in preparation amount that can be distinguished by a person who prepares the tooth.

Repositioning of teeth accounts for differences in the estimated measurement errors between repeatability and reproducibility. Actually, the main problem with repositioning was that the mounting device allowed for more than one position of the teeth. The measurement error for repeatability is lower than that of reproducibility measurements, which indicates that the software and the communication between the camera and the computer work quite accurately.

While the range of 95 percent limits of agreement of repeatability may reach values up to 200 µm [78 µm – (–122 µm)] for unprepared teeth, the corresponding reproducibility range may reach values up to 376 µm [218 µm – (–158 µm)] for unprepared teeth. The rationale of scanning unprepared teeth was to examine whether the system was more sensitive regarding prepared compared to unprepared teeth.

According to the limits of clinical perceptibility by eye, the system’s reproducibility was found to be poor, at least for instructional purposes in a pre-clinical lab. However, it is important to note that two experienced teaching assistants evaluated the same preparations and poor reproducibility was also documented for this human assessment.9

All preparation types tested were solely extracoronal. It is likely that the performance of the system could be influenced from the preparation type (i.e., intracoronal vs. extracoronal).

The reproducibility range is critical for instructional purposes because reproducibility procedures are mostly encountered in preclinical laboratory courses. Because both of these ranges lie in the area of clinical perception, a contradictory false (i.e., either positive or negative or irregularly both of them) feedback for the students could not be excluded. Therefore, we decided, based on the results of this study, that students should draw the planned preparation corrections on the tooth (with a drawing pen) before using the PREPassist system. In this way the students could guide themselves, even though the system suffers from reproducibility deviations.

The calculated bias of the measurements was considered zero. Even if in Figure 3Go the bias of the reproducibility measurements is not zero ({approx}30 µm), it was considered zero for preclinical laboratory instruction since it is located beyond the range of clinical perception. Decisions like these are always individually made, based on subjective criteria.

There seems to be no explanation available for the resulted differences of bias comparing reproducibility and repeatability (reproducibility bias was at least twice as much as compared to the repeatability situation).

The method used here to calculate reliability of the PREPassist system has been applied in other studies in medical and dental research.4–6,10,11 The terms "accuracy," "repeatability," and "reproducibility" are very often used in a confounding manner. While accuracy has been defined as the degree to which a measurement actually represents what it is intended to represent12 or the closeness of the agreement between the result of a measurement and a true value of the measurand,3 very often (in the name of accuracy) either repeatability or reproducibility are measured and stated.

It is essential to assess the reliability of a measuring device that is used to assist students’ learning in the preclinical phase of their dental education. The accuracy of a 3-D optical scanner system for casts and impressions was reported to be 24 µm and 13 µm, respectively.7 A relationship of 1:10 between repeatability and reproducibility of results of tooth surfaces for a 3-D laser scanner was also reported.13 The reliability of 3-D scanners is judged based on individual, subjective criteria.


   Conclusions
Top
 Abstract
 Materials and Methods
 Results
 Discussion
 Conclusions
References
 
Accuracy as well as repeatability of the PREPassist system was found to be generally sufficient for its use in student education, according to this pilot study. Thus, we suggest its implementation into preclinical education as a supportive educational strategy. Reproducibility of the system still suffers from the imprecision of the mounting device. The adjustment of this problem is expected to improve the reproducibility of the system.


   Footnotes
 
Dr. Kournetas, Dr. Jaeger, and Dr. Lachmann are Assistant Professors, Dr. Axmann is Biostatistician, Dr. Groten is Associate Professor, and Dr. Weber is Medical Director of the Department of Prosthodontics, Medical Materials, and Technology, and Dr. Geis-Gerstorfer is Head of the Section of Medical Materials and Technology, Department of Prosthodontics, Clinic for Dental, Oral, and Maxillary Medicine—all at University Hospital, Tuebingen, Germany. Direct correspondence and requests for reprints to Dr. Detlef Axmann, Department of Prosthodontics, Medical Materials, and Technology, Clinic for Dental, Oral, and Maxillary Medicine, University Hospital, Osianderstrasse 2-8, 72076 Tuebingen, Germany; 49-7071-2987433 phone; 49-7071-293982 fax; detlef.axmann{at}med.uni-tuebingen.de.

This study was partly funded by the Ministry of Science, Research, and the Arts of the State of Baden-Wuerttemberg, Germany, under the program "Innovative Projects in Education."


   REFERENCES
Top
 Abstract
 Materials and Methods
 Results
 Discussion
 Conclusions
 References
 

  1. Lehmann KM, Stachniss V. Aspekte einer zukunftsorientierten propaedeutischen Ausbildung in den restaurativen Faechern der Zahnheilkunde (Aspects of a future-oriented propaedeutic education in restorative dentistry). Dtsch Zahnarztl Z 2000;55:517–17.
  2. Jaeger B, Kournetas N, Groten M, Lachmann S, Weber H, Geis-Gerstorfer J. PREPassist as digital assistance for dental education. Int Poster J Dent Oral Med 2003;5:169.
  3. International vocabulary of basic and general terms in metrology. Berlin: Beuth Verlag GmbH, 1994.
  4. Guide to the expression of uncertainty in measurement. Geneva, Switzerland: International Organization for Standardization, 1995.
  5. Bland JM, Altman DG. Measurement error. BMJ 1996;313:744[Free Full Text]
  6. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307–10.[Medline]
  7. DeLong R, Heinzen M, Hodges JS, Ko CC, Douglas WH. Accuracy of a system for creating 3D computer models of dental arches. J Dent Res 2003;82:438–42.[Abstract/Free Full Text]
  8. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8: 135–60.[Abstract/Free Full Text]
  9. Quinn F, Keogh P, McDonald A, Hussey D. A pilot study comparing the effectiveness of conventional training and virtual reality simulation in the skills acquisition of junior dental students. Eur J Dent Educ 2003;7:13–9.[Medline]
  10. Axmann D, Gomez G, Groten M. Der Bland und Altman Plot-eine einfache graphische Methode zur Einschaetzung der Verlaesslichkeit von Messverfahren (The Bland and Altman plot: a simple graphical method to estimate the reliability of measuring methods). Dtsch Zahnarztl Z 2002;57:613–6.
  11. Groten M, Axmann D, Probster L, Weber H. Determination of the minimum number of marginal gap measurements required for practical in-vitro testing. J Prosthet Dent 2000;83:40–9.[Medline]
  12. Hewlett ER, Orro ME, Clark GT. Accuracy testing of three-dimensional digitizing systems. Dent Mater 1992;8:49–53.[Medline]
  13. Mehl A, Gloger W, Kunzelmann KH, Hickel R. Entwicklung eines neuen optischen Oberflaechenmeßgeraets zur präzisen dreidimensionalen Zahnvermessung (Development of a new optical device for a precise three-dimensional measurement of teeth). Dtsch Zahnarztl Z 1996;51:23–7.




This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Kournetas, N.
Right arrow Articles by Geis-Gerstorfer, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Kournetas, N.
Right arrow Articles by Geis-Gerstorfer, J.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS