J Dent Educ. 71(6): 759-766 2007
© 2007 American Dental Education Association
Educational Methodologies |
Expert Performance on a Virtual Reality Simulation System
Els R. Wierinck, Ph.D.;
Veerle Puttemans;
Stephan P. Swinnen, M.D., Ph.D.;
Daniel van Steenberghe, M.D., Ph.D.
Key words: expert performance, virtual reality, augmented feedback, retention, education
Submitted for publication 12/14/06;
accepted 03/23/07
 |
Abstract
|
|---|
The objective of this research was to determine if the essence of expert performance could be captured on a virtual reality simulation system. Six experts in operative dentistry, six experts in periodontology, and six novice dental students performed a Class II tooth preparation task on the lower left second premolar. All subjects performed a pre-test to assess the basic skill level of each group. During the (limited) training component of the study, the three groups practiced three tooth preparations and received augmented feedback. At both a one-minute and one-day interval, subjects performed a final test in the absence of augmented feedback. All preparations were graded by the simulation system. The results showed at pre-test a significantly better performance of the experts in operative dentistry as compared to the novices. During the practice (acquisition) phase, the experts in operative dentistry outperformed both the periodontologists and novices, whereas the experts in periodontology performed more accurately than the novices. After one minute and one day following practice, similar results were obtained. Retention performance was most accurate after a one-day delay. Based on these results, the simulator appears to be a valid and reliable tool to capture expert performance. It is an effective screening device for assessing the level of expert performance.
Alot of our prevailing activities in daily life involve coordination skills that evolve across daily practice; these include writing, driving a car, playing a musical instrument, and so on. Within the field of motor learning, it has been shown that intensive and/or prolonged training precedes the successful execution of these complex tasks.1 Masters in a particular topic, often referred to as experts or specialists, attain their highest performance levels only after a decade or more years of practice, the so-called "10-year rule."2 Through this practice, expert performers gradually develop specific perceptual-cognitive mechanisms for the planning, execution, monitoring, and post factum evaluation of actions.3 The intensity of practice and adaptability of the training to the specific needs of the learner are crucial factors. In 1993, Ericsson et al.4,5 introduced the term "deliberate practice" for the individualized training activities designed to improve specific aspects of an individual’s performance through repetition and successive refinement. Others have shown that several forms of feedback, often of the visual type, can influence this learning process in either a positive or negative way.6–10
In dental education, students traditionally devote several years to the acquisition of sufficient fine motor skills to prepare them for entry-level dental practice. The active training period ranges from two to five years, depending on the type of curriculum and the nation. While in Europe the use of the high speed handpiece in dental anatomy and other preclinical labs usually starts early in the first year of the dental curriculum, in North America it is often postponed until late in the first year or the start of the sophomore year. Because considerable differences in students’ performance are noticed, particularly at the start of training, students normally practice a long time in a supervised phantom head laboratory before they are allowed to start patient care. After graduation, one can elect to focus or specialize in a particular field of oral health care and receive advanced training in it. After continued training and experience of several years, recognition as an expert in one of the oral health specialties can be obtained.
Recently, virtual reality (VR) simulators have been introduced into the dental curriculum as training devices for manual dexterity acquisition in tooth preparation tasks. A computerized simulator allows practicing tooth preparations in the presence of augmented visual feedback, resulting in enhanced performance under particular conditions, at least in novice students.11–13 However, the question remains whether the simulator can also capture the essence of expert performance. Our study directly relates to construct validity of the VR simulation by exploring performance differences on a representative tooth preparation task between experts in operative dentistry and nonexperts in that particular field. For this purpose, operative dentists were compared to periodontologists and novice dental students. Both groups of dentists had a similar amount of professional background, but in a distinct field, and thus presumably acquired different manual skills unique to their area of specialization. The naïve group (novice students) did not have previous practice in any high-skill procedure. We hypothesized that if the VR simulator serves to evaluate expert tooth preparation performance appropriately, a significantly better score for the operative dentist experts as compared to the periodontologists and novice students should be observed during basic and skill retention testing.
|
Materials and Methods
|
|---|
Eighteen right-handed volunteers were enrolled in the study. The enrollment was based on willingness of staff members to participate. Considering the small size of the academic staff at both departments, it meant that nearly its entirety was enrolled. All participants were randomly recruited from a master group, and they received no incentives or rewards for participation. A completely anonymous outcome of the experiment was guaranteed. Participants were assigned to three experimental groups, each consisting of six subjects: the operative dentists (EXP), the peri-odontologists (PER), and the naïve (NAIV) group. Groups were matched in terms of gender (three male and three female subjects in each group), background (training in dentistry at K.U. Leuven), and clinical expertise. The study was performed at the beginning of the academic year 2005–06. The EXP group, aged forty to sixty-three (mean: 48), had professional experience ranging between seventeen and thirty-three years (mean: 24). They were very experienced in preparing cavities for carious lesions in teeth. The PER group, aged thirty-seven to fifty-eight (mean: 46), consisted of periodontologists who had not performed any tooth preparations since their graduation at least fifteen years ago. Their professional experience, which consisted of soft tissue and bone surgery including drilling in bone, ranged between fourteen and thirty-three years (mean: 23). As such, the EXP and PER groups were matched in age and years of expertise in their particular field. The NAIV group included six first-year dental students aged seventeen to forty (mean: 23) who had no previous experience with tooth preparation tasks or any related manual dexterity training. For purposes of the study design, it is important to note that manual dexterity, in general, does not decline prior to the age of sixty-five years, so the age of the subjects should not influence their performance capacity.14,15
The task assigned to the three groups consisted of a Class II amalgam preparation on the lower left second premolar. The preparation’s dimensions as well as the outline shape were clearly defined, including an occlusal lock and distal box (Figure 1
). All tooth preparations were completed, evaluated, and graded on the daily calibrated DentSimTM computerized training system (DenX, Jerusalem, Israel). Reliability of measure was limited to 300 micrometers.16 The latter unit includes a simulated patient or manikin with head and dentoform, dental handpiece, light source, infrared camera, and two computers.17 The optic tracking device, consisting of an infrared camera and light emitting diodes on the head and handpiece, allows spatial orientation of the manikin’s head and full registration of the handpiece’s movements. These registration data are analyzed and converted into a virtual three-dimensional (3-D) image of the preparation in progress, which is presented on a screen to the subject. As such, the software provides virtual representation of the preparation in progress and enhanced feedback, comparing the subject’s performance with an ideal preparation at any moment during tooth preparation as well as after its completion. This additional information will be referred to as augmented feedback. All preparations were cut in an ivorine tooth with a diamond cylindrical bur used at ultra-high speed and with continuous water spray. Participants were supplied with a graduated periodontal probe and mouth mirror.
All subjects were briefly instructed on the use of the simulation system and the requirements of the Class II preparation. To start, the three groups performed a pre-test preparation (P) to assess their basic skill level. During this test, the simulator provided no augmented feedback. Subsequently, three training preparations were performed, each within a time limit of thirty minutes, with augmented feedback provided. These three training trials (ACQ1-ACQ2-ACQ3) will be referred to as the acquisition phase. During this acquisition phase, participants had access to the same instructor but only for technical advice. Finally, two retention tests were administered: at one minute (R1) and one day (R2) following completion of the acquisition phase and in absence of augmented feedback. As such, all participants performed five consecutive tooth preparations on the first day (P, ACQ1, ACQ2, ACQ3, and R1) and one final preparation (R2) on the second day of the study.
All preparations were evaluated and graded by the VR system, assigning to each tooth preparation an overall preparation score up to 100. This score was the primary variable of interest, reflecting the amount of proper versus erratic preparation, i.e., the extent of correctness. As such the maximum score of 100 was decreased with the sum of all error scores. The passing grade for this task is routinely defined at 60 by the DentSim grading system itself. Three evaluation parameters contributed to the calculation of this overall preparation score, namely, outline shape (OUT), cavity floor (FLO), and cavity walls (WAL). The error scores of these parameters were determined to reveal the execution strategy across performance. No additional error points were imposed for accidental damage of the adjacent tooth or for reduction of the tooth cusp that was too extensive. Finally, the preparation times were recorded. The number of times and the duration of evaluation by referring to the augmented feedback sources were also analyzed.
A one-way 3 x 1 (Group x Test) analysis of variance (ANOVA) was conducted on all dependent variables (performance and error scores, performance times) obtained at the pre-test (P) to identify initial differences among the three groups. The data were tested for departure from normality and, if necessary, transformation of the dependent variable was performed. Additional Tukey HSD post hoc tests were performed when significant effects were encountered (p<0.05). Similarly, 3 x 3 (Group x Acquisition Trial) and 3 x 2 (Group x Retention Test) repeated measures ANOVAs were conducted to compare performance of all groups across acquisition (ACQ1-3) and retention tests (R1-2), respectively.
|
Results
|
|---|
At pre-test (Figure 2), a significant group effect (F2,15=5.65, P<0.05) revealed performance differences among the three groups. The EXP group had a higher preparation score than the NAIV group (Table 1). This was due to a better accuracy on cavity floor assessment (F2,15=8.58, P<0.01). The EXP group’s tendency to perform with greater accuracy than the PER group just failed to reach significance (P=0.094).
Across acquisition (Table 2), the only significant finding was a main effect for group (F2,15=9.28, P<0.01). Post hoc tests and additional analyses revealed a more accurate performance for the EXP group than both the PER and NAIV groups, due to better performance on both outline shape (F2,15=10.59, P<0.001) and cavity floor (F2,15=8.04, P<0.01). The PER group performed more accurately than the NAIV group, due to lower error scores on the outline shape criterion (F2,15=10.59, P<0.001). Analysis of the cavity wall criterion revealed no significant differences (P>0.05). Additionally, analyses of the number and durations of evaluations performed across acquisition revealed two trial main effects. All groups consulted the evaluation information more frequently during the first trial (ACQ1), relative to the third trial (ACQ3) (F2,34=6.73, P<0.01). Evaluation time of the preparations in progress was also the longest during the first acquisition trial (ACQ1) (F2,34=8.40, P<0.001).
At the retention tests (Table 3), in the absence of augmented feedback, preparation scores improved from R1 to R2, yielding a significant main effect for test (F1,17=4.49, P<0.05). The main effect for group was significant (F2,15=6.72, P<0.01). Post hoc tests revealed a significantly better performance for the EXP group as compared to both other groups and for the PER group as compared to the NAIV group. Additional analyses of the error scores revealed a performance pattern similar to that obtained across acquisition. The EXP group performed with greater accuracy than both other groups on outline shape (F2,15=4.51, P<0.05) and cavity floor assessment (F2,15=15.03, P<0.001), while the PER group outperformed the NAIV group on outline shape (F2,15=4.51, P<0.05). Analysis of the cavity walls criterion revealed no significant main effects (P>0.05).
The overall maximum preparation scores throughout the study were 89, 75, and 64 for the EXP, the PER, and the NAIV groups, respectively
Figure 3 depicts the overall performance times for all groups at the pre-test, acquisition, and retention. Statistics revealed a similar initial execution speed for all groups (P>0.05). During acquisition, only a significant main effect for trial was noticed (F2,34=17.53, P<0.001). Post hoc analysis showed an increased preparation speed across training. The first acquisition trial (ACQ1) was performed significantly slower than both subsequent trials. At retention, there was a test (F1,17=5.38, P<0.05) and a group (F2,15=3.76, P<0.05) main effect. Post hoc tests revealed that preparation speed decreased significantly from R1 to R2, while the PER group performed at the lowest rate during retention, relative to both other groups.
View larger version (28K):
[in this window]
[in a new window]
|
Figure 3. Preparation times (s) of all groups at pre-test (P), acquisition (ACQ1-3), and retention (R1-2)
|
|
|
Discussion
|
|---|
The goal of our study was to determine if a VR navigation system could be used as a screening device for expert performance. Therefore, two groups of experts (EXP and PER) were compared with a group of novice dental students (NAIV). All subjects performed a preliminary brief test, three training trials, and two retention tests. It was expected that preparation scores between groups would differ if the simulator is able to discriminate differences in expert level.
At pre-test, the experts in operative dentistry revealed a better performance than the naïve group. This superior initial performance was expected and in agreement with the findings of Ericcson,4,5 Starkes et al.,18 and Charness et al.,19 reporting that better performance is a consequence of the amount of practice and marks expert performance.
During acquisition, performance accuracy within each group remained similar, as the time allocated to training was too limited to reach a higher level of skill.20 The process, during which more effective movement patterns are developed, often takes several years. As such, the short-term acquisition period allowed the three groups only to get acquainted with the task and the feedback system, without improving performance. Preparation speed increased across acquisition, due to decreased evaluation of the augmented feedback. Especially, the operative dentist experts evaluated the first training preparation frequently, resulting in a relatively high error score on the cavity wall criterion (WAL).
Finally, two retention tests were administered to assess whether the obtained performance differences across acquisition referred to permanent or rather transitory performance effects. As such, these tests provided insight into the immediate and delayed recall capacities of each group. Although a delay of one minute or one day may seem short, it is recognized as a sufficient time lapse to assess the permanent features of improved skill.21 Since the limited training rather influences the familiarity with the system, longer time intervals were not considered. These tests, moreover, evaluate the transfer capacities when performance is accomplished in the absence of augmented feedback. Carefully selected modes of additional artificial input commonly serve as an important tool to aid skill learning but may have some disadvantages, such as creating dependency on the system.9,22 Indeed, continuous provision of concurrent feedback guides performance but may not be beneficial for learning. Therefore, the real test for evaluating skill achievement is to test trainees when they are refrained from any augmented feedback. Again, the operative experts’ high performance level was evident at retention test occasions. They were able to adapt their performance to the task constraints. Probably, this was due to better cognitive activities, such as a highly structured knowledge base of operative dentistry principles and well-developed pattern-recognition abilities. These superior skills enabled them to detect and correct their errors effectively in the absence of previously presented augmented feedback. As such, the experts had acquired over the years mechanisms to increase their control and ability to monitor performance in representative situations from the domain of expertise.4 Furthermore, retention performance improved after one day as compared to the immediate retention performance at one minute. This can be explained by a possible fatigue phenomenon at day one, where five preparations had to be performed. The improvement across retention, moreover, ensured permanent rather than transitory skill retention.9
Across all six tooth preparations, the operative expert group scored significantly better than the periodontologists and novices, indicating that both expert and nonexpert performance on a tooth preparation task can be captured and rated on a simulator. Each time, the accuracy on assessment of cavity floor (FLO), including the depth, inclination, and smoothness of the cavity floor, captured the essence of the expert-level performance. Superior performance of the operative experts on a cavity floor’s depth reflects mastery in 3-D insight of both cavity configuration and tooth morphology. A better floor inclination suggests a solid knowledge of the principles of operative dentistry, while superior performance on floor smoothness reveals control of minute finger movements needed for tooth preparation. The latter implies flexion and extension of the fingers within mostly the axial plane. Although the periodontologists are used to a counter-angle handpiece to drill in jaw bone, they are not familiar with these minute movements. In contrast, drilling in bone tissue, e.g., while inserting endosseous implants, uses movements within the wrist, while the counter-angle handpiece is maintained with a constant finger grasp. Tooth preparation tasks, however, are performed in harder tooth structure, frequently using a high-speed handpiece. Furthermore, the periodontologists outperformed the novices across acquisition and retention, indicating that the simulator has a potential to discriminate among different levels of expertise. Their performance was superior on the outline shape criterion (OUT), indicative of a better visual representation of the criterion task. Eventually, the tooth preparation skill, acquired on phantom teeth in a preclinical setting many years ago, was still stored in and recalled from the periodontologists’ motor memory. This is in agreement with the encoding specificity principle,23 according to which increasing similarity between the practice contexts increases recall performance.
Finally, we want to stress an important issue. The stringent criteria applied by the system to score a cavity preparation led to few top scores, even in the group of operative experts. The scoring system, provided with the VR system, seems arbitrary to a certain extent. One should point out that the training period was limited in our experimental set-up. Thus the experts who reached close to 90 percent would probably reach more than 90 percent if ample training time was provided. They also felt that the set-up of the navigation system being different from the clinical situation meant change of body positioning. The need to keep the space between the infrared camera and the LEDs on the handpiece free implies a proper ergonomic position, which is not always maintained even by experts. Although experts admitted they experienced the experimental set-up as an unfamiliar environment, the results illustrate the benefits derived from further professional training.
|
Conclusions
|
|---|
The VR simulator is a valid and reliable screening device to capture expert performance even after brief training to familiarize the subject with the new environment. The experts in operative dentistry revealed higher levels of excellence in performance than the novice students. Furthermore, it was possible to distinguish experts in operative dentistry from experts in periodontology, which suggests that expertise is specific to the domain of oral health care within which the skill has been developed and has been maintained. The stringent criteria used by the VR system renders reaching top scores improbable even for experts in the field after such a brief training.
|
Footnotes
|
|---|
Dr. Wierinck is Junior Academic Staff Member, Skills Lab, School of Dentistry, Oral Pathology and Maxillofacial Surgery; Dr. Puttemans is Associate Academic Staff Member, Motor Control Laboratory, Biomedical Kinesiology; Dr. Swinnen is Professor, Motor Control Laboratory, Biomedical Kinesiology; and Dr. van Steenberghe is Professor, Department of Periodontology—all at Katholieke Universiteit Leuven, Leuven, Belgium. Direct correspondence to Dr. Els Wierinck, K.U. Leuven, School of Dentistry, Oral Pathology & Maxillofacial Surgery, 7 Kapucijnenvoer, 3000 Leuven, Belgium; 32-16-332308 phone; 32-16-332435 fax; Els.Wierinck{at}med.kuleuven.be.
|
REFERENCES
|
|---|
- Schmidt RA, Lee TD. Motor learning concepts and research methods. In: Schmidt RA, Lee TD, eds. Motor control and learning. Champaign, IL: Human Kinetics, 1999:263–84.
- Simon HA, Chase WG. Skill in chess. Am Sci 1973; 61:394–403.
- Ericsson KA, Lehmann AC. Expert and exceptional performance: evidence of maximal adaptation to task constraints. Annu Rev Psychol 1996; 47:273–305.[Medline]
- Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004; 79(Suppl):S70–S81.[Medline]
- Ericsson KA, Lehmann AC, Tesch-Römer C. The role of deliberate practice in the acquisition of expert performance. Psychol Bull 1993; 100:363–406.
- Annett J. Feedback and human behaviour. Middlesex: Penguin, 1969.
- Salmoni AW, Schmidt RA, Walter CB. Knowledge of results and motor learning: a review and critical reappraisal. Psychol Bull 1984; 95:355–86.[Medline]
- Lee TD, Swinnen SP, Verschueren S. Relative phase alternations during bimanual skill acquisition. J Mot Behav 1995; 27:263–74.[Medline]
- Swinnen SP. Information feedback for motor skill learning: a review. In: Zelaznik HN, ed. Advances in motor learning and control. Champaign, IL: Human Kinetics, 1996:37–66.
- Swinnen SP, Lee TD, Verschueren S, Serrien DJ, Bogaerds H. Interlimb coordination: learning and transfer under feedback conditions. Hum Mov Sci 1997; 16:749–85.
- Wierinck E, Puttemans V, Swinnen S, van Steenberghe D. Effect of augmented visual feedback from a virtual reality simulation system on manual dexterity training. Eur J Dent Educ 2005; 9:10–6.[Medline]
- Wierinck E, Puttemans V, van Steenberghe D. Effect of tutorial input in addition to augmented feedback on manual dexterity training and its retention. Eur J Dent Educ 2006; 10:24–31.[Medline]
- Wierinck E, Puttemans V, van Steenberghe D. Effect of reducing frequency of augmented feedback on manual dexterity training and its retention. J Dent 2006; 34:641–7.[Medline]
- Krampe RT. Aging, expertise and fine motor movement. Neurosci Biobehav Rev 2002; 26:769–76.[Medline]
- Carmeli E, Patish H, Coleman R. The aging hand. J Gerontol 2003; 58A:146–52.
- Wierinck E. The use of navigation system-based feedback on the learning process of motor skills during tooth preparation. Leuven, Belgium: K.U. Leuven, 2006.
- Rose JT, Buchanan JA, Sarrett DC. The DentSim system. J Dent Educ 1999; 63(5):421–3.
- Starkes JL, Deakin JM, Allard F, Hodges NJ, Hayes A. Deliberate practice in sports: what is it anyway? In: Ericcson KA, ed. The road to excellence: the acquisition of expert performance in the arts and sciences, sports, and games. Mahwah, NJ: Erlbaum, 1996:81–106.
- Charness N, Krampe R, Mayr U. The role of practice and coaching in entrepreneurial skill domains: an international comparison of life-span chess skill acquisition. In: Ericcson KA, ed. The road to excellence: the acquisition of expert performance in the arts and sciences, sports, and games. Mahwah, NJ: Erlbaum, 1996:51–80.
- Schmidt RA, Lee TD. The learning process. In: Schmidt RA, Lee TD, eds. Motor control and learning: a behavioral emphasis. 3rd ed. Champaign, IL: Human Kinetics, 1999:357–83.
- Horn RR, Williams AM, Scott MA, Hodges NJ. Visual search and coordination changes in response to video and point-light demonstrations without KR. J Mot Behav 2005; 37:265–74.[Medline]
- Schmidt RA, Young DE, Swinnen SP, Shapiro DC. Summary knowledge of results for skill acquisition: support for the guidance hypothesis. J Exp Psychol Learn Mem Cogn 1989; 15:352–9.[Medline]
- Tulving E, Thomson DM. Encoding specificity and retrieval processes in episodic memory. Psychol Rev 1973; 80:352–73.
Top
Abstract
Materials and Methods
