J Dent Educ. 72(11): 1296-1303 2008
© 2008 American Dental Education Association
Educational Methodologies |
Evaluation of an Audience Response System in a Preclinical Operative Dentistry Course
Ana Elashvili, D.D.S., M.S.;
Gerald E. Denehy, D.D.S., M.S.;
Deborah V. Dawson, Sc.M., Ph.D.;
Marsha A. Cunningham, B.S., M.S.
Key words: audience response system, interactive lecture, operative dentistry
Submitted for publication 04/23/08;
accepted 07/30/08
 |
Abstract
|
|---|
Student performance was compared on written and psychomotor skill tests of freshman dental students receiving conventional lectures versus the same lectures containing interactive components using TurningPoint, a wireless audience response system (ARS). The research design was a controlled crossover study with seventy-seven freshman dental students conducted in a preclinical operative dentistry course. Two randomized groups alternated the two study lectures, one with ARS and the other without ARS. Student knowledge retention was measured through written examination using immediate posttest, as well as questions on the unit and final examinations. Psychomotor skill tests were given on both lecture topics. Statistically significant differences indicating superiority of ARS were identified for performance on the immediate posttest and psychomotor skill test only for the lecture "Principles of Dental Bonding." The other examinations/skill testing showed no significant difference. These results indicate that ARS is a promising teaching tool for dental education.
Increased population and greater public awareness of the importance of quality dental care, along with rapid progression of advanced technology and knowledge in dentistry, have brought about a high demand for qualified dental clinicians, educators, and scientists. There is a great challenge in providing good dental education to meet these demands.1,2 An article published in the Journal of Dental Education3 states that "the curriculum at most dental schools is based on a model of educational delivery that is at least fifty years old, while emerging science, technology, and disease patterns promise to change oral health care significantly." Dental education is in need of active learning strategies such as dental students’ active engagement during the lecture and immediate feedback, factors that are associated with the development of the mental capacities for the expert practice of dentistry.4 This study was conducted to assess an alternative method of classroom dental education. Specifically, a traditional lecture style and an interactive lecture style using a wireless audience response system (ARS) were compared.
Audience response systems, such as Turning-Point (Turning Technologies, LLC), have been recommended for use in classroom settings to provide instant feedback to and from the students. Various devices and software used for this purpose can provide the instructor the ability to graph and display responses to any question. Two research groups prominent in the rapid development and implementation of ARS are Mazur’s group at Harvard5 and Dufresene’s at the University of Massachusetts,6 both in physics courses. These groups have described the audience response systems as successful tools to make classrooms more interactive and students more active participants in their own learning process. Advantages of ARS are that they help focus the audience’s attention, emphasize the key points of the lecture, engage the audience with the speaker, identify deficiencies in knowledge, and motivate the students. Two main features distinguishing an ARS from other feedback methods are anonymity and the ability to be used effectively in large classrooms.7–9 Thus far, there has been little documented use in the health care field and to an even lesser extent in dentistry.
Because of the lack of published research using wireless interactive technology in the dental field, little is known about the educational impact of ARS on dental students. A student opinion questionnaire was the main data collection method in the only studies done in the dental field to validate ARS.10–12 Only two studies done in the medical field13,14 have validated knowledge retention of the ARS by using a comparison of pre- and posttests on written performance.
In our study, performance was compared on both written and psychomotor skill tests of freshman dental students receiving conventional lectures versus the same lectures containing interactive components using TurningPoint, a wireless ARS. Since dental student knowledge is translated into clinical performance, it is important to determine if ARS learning also transfers to psychomotor performance. To our knowledge no studies have been conducted to measure the impact of ARS use on transfer of knowledge through psychomotor skill performance. A survey questionnaire was also given to the students to determine attitudes towards the ARS style of lecture.
|
Methods
|
|---|
This study was a randomized controlled crossover clinical trial. A wireless audience response system, TurningPoint, was used to convert a conventional/traditional lecture into an interactive lecture. The study was conducted with approval of the University of Iowa Human Subjects Office. The subjects in the study were freshman dental students at the University of Iowa College of Dentistry enrolled in the Operative Dentistry Techniques course, an introductory course for dental students conducted in the simulation clinic during the second semester of the first year. Two lecture topics were used in the study: "Principles of Dental Bonding" and "Class IV and V Composite Resin Restorations." The class was randomized into two groups; each group received one of the two selected lectures in the interactive format and the other lecture in the conventional format (Figure 1
). Both received the same lecture in either interactive or conventional styles on the same day by the same lecturer. The format was reversed for the groups for the second lecture. The conventional and the interactive lectures were identical PowerPoint lectures, except for the addition of the TurningPoint questions and response items to the interactive lecture.
TurningPoint is a combination of hardware and software designed to support interaction and communication in classes. After an interactive question appears on the screen, the student presses a clicker button corresponding to the answer considered to be correct. The responses are electronically collected, summarized, and simultaneously displayed graphically in the auditorium or classroom (Figure 2).
Eleven to twelve ARS questions were asked during the interactive lecture. Most questions were asked before introducing new topic areas, while a few questions were asked immediately after covering part of the material of a particular topic. The TurningPoint keypad units were assigned to the students randomly and anonymously, and only overall class answer results were documented. When the majority of the students answered incorrectly on any TurningPoint question, the lecturer explained that key concept of the lecture again. Lectures were approximately one hour long. During both interactive and non-interactive lectures, students had the opportunity to ask questions of the lecturer.
Student knowledge retention was measured through written and practical examinations. The written examinations included an immediate post-test (Examination 1, E1), a unit examination (E2), and a final examination (E3). Ten questions on each topic were asked on each written examination. The E1 questions did not count for the students’ grade, and the students were not provided with the answers. The questions on the individual exams were identical for the two test groups. However, to avoid rote memorization of the original questions, four parallel questions covering the same topic area were asked on the ARS lecture E1, E2, and E3 examinations. One practical examination (E4) was given involving the preparation and restoration of a Class IV restoration with composite resin material, and the second practical examination consisted of a bonding exercise in which a composite resin stub was bonded to a tooth and the measured outcome of performance for the exercise was the shear strength determined using a shear bond tester, T-63010 (Bisco, Schaumburg, IL). Student performance scores for the two groups were compared for both practical examinations.
Students’ attitudes toward the ARS style of lecture were assessed by a survey questionnaire. There were two questionnaires: Questionnaire I, to evaluate the general quality of each class, and Questionnaire II, to evaluate the ARS (Figure 3). Questionnaire I was administered to both groups after each of the two classes. It had only four questions and addressed the clarity and effectiveness of the entire lecture. Questionnaire II provided students with the opportunity to express their feelings about the ARS. This questionnaire consisted of ten questions and was given after each interactive lecture using ARS. Both questionnaires were constructed based on a summary of the literature on ARS.
Students were introduced to the research and were asked to become volunteers for the study. The alternative for those students not participating in the research was to attend the conventional lectures and take practical, unit, and final examinations as determined by the course requirements. They would not attend the lecture using ARS or take the post-test or questionnaires, and their scores on unit and final examinations would not be used for research purposes. All dental students agreed to become the study participants, so that the alternatives were not necessary.
Wilcoxon rank sum tests were used to compare student knowledge and performance for E1, E2, E3, E4, and Questionnaire I. Ridit analysis, a variant of the Cochran-Mantel-Haenszel mean score test, was used to consider whether the distribution of Likert responses for each item in Questionnaire II differed between the two study groups15–17—that is, whether the distribution of ordinal ratings of agreement differed for the group of students receiving ARS for the "Class IV and V Composite Resin Restorations" lecture vs. the group of students receiving ARS for the "Principles of Dental Bonding" lecture.
|
Results
|
|---|
Study participants were freshman dental students (2006–07 academic year) of the University of Iowa College of Dentistry, taking the Operative Dentistry Techniques course. A total of seventy-seven students between the ages of twenty-one and forty participated in the study. Students were divided randomly into two groups, with forty students in Group I and thirty-seven students in Group II (when one student from Group II accidentally attended the other group’s first lecture, he was moved from Group II to Group I). All forty students from Group II attended the lecture with ARS, and only thirty-four out of thirty-seven students attended the conventional lecture on dental bonding. Thirty-nine students out of forty from Group I attended the conventional lecture, and all thirty-seven students from Group II attended the lecture with ARS on the Class IV and V composite resin restorations.
The data provided no evidence of a difference in the distribution of age between the two study groups (p=0.19, Wilcoxon rank sum test) and no evidence of a difference in the distribution of gender between the two study groups (p=0.50, Fisher’s exact test). Median age was twenty-three for the entire sample and for each of the study groups. Females comprised 44.2 percent of the entire sample of seventy-seven participants.
Statistically significant differences were found in favor of the ARS in scores on the immediate written posttest (mean scores of 8.7 vs. 7.6, p=0.002) and in performance bond strength testing (means of 26.7 vs 23.3 MPa, p=0.039) for the lecture "Principles of Dental Bonding." The other examinations/skill testing showed no significant difference between the two research groups. The results of E1, E2, E3, and E4 on both lectures are provided in Tables 1 and 2.
View this table:
[in this window]
[in a new window]
|
Table 1. Results for Wilcoxon rank sum test comparing ARS vs. conventional lecture and group means for the lecture "Principles of Dental Bonding"
|
|
View this table:
[in this window]
[in a new window]
|
Table 2. Results for Wilcoxon rank sum test comparing ARS vs. conventional lecture and group means for the lecture "Class IV and V Composite Resin Restorations"
|
|
In assessing student evaluations of the lectures (Questionnaire I), the data provided strong evidence that the responses to Question 4 ("I can easily transfer my knowledge gained from the class to the practical examination") were significantly higher in the group receiving the "Principles of Dental Bonding" lecture with ARS than in the group receiving this lecture via the conventional lecture format (p=0.0039, Wilcoxon rank sum test). The data provided no evidence that the two study groups differed in the distribution of other questions on this instrument for any of lectures. Table 3 summarizes the results for Questionnaire I.
View this table:
[in this window]
[in a new window]
|
Table 3. Results of Questionnaire I for Wilcoxon rank sum test for the lectures "Principles of Dental Bonding" and "Class IV and V Composite Resin Restorations"
|
|
In assessing student reactions to the ARS (Questionnaire II), no Strongly Disagree responses were recorded. Question numbers 1, 6, and 8 showed no negative responses; 10.68 percent of all answers were Neutral, and 2.45 percent of all answers were Disagree. Ninety-nine percent of the participants stated that they enjoyed attending the lecture using TurningPoint/ARS (on question N:1 answers Strongly Agree and Agree combined); 84.4 percent would like to have ARS in their future classes (question N:2); 88.3 percent considered that ARS helped them to understand the main concept of the lecture (question N:3); 84.4 percent considered that ARS helped them to identify their weaknesses and strengths in understanding the given material (question N:4); 88.3 percent considered that using ARS helped keep them fully engaged in the class (question N:5); 97.4 percent considered that the anonymity of ARS helped them to more freely and honestly answer the questions asked during the interactive lecture than during a conventional lecture (question N:6); 58.4 percent considered that ARS motivated them to really study the material covered during the lecture (question N:7); 98.7 percent thought that the material covered by ARS was clear and understandable (question N:8); and 92.2 percent considered a lecture using ARS more informative than a conventional lecture (question N:10).
The responses of Group I receiving ARS for the lecture "Principles of Dental Bonding" tended to show greater agreement and enthusiasm than the responses of Group II receiving ARS for the lecture "Class IV and V Composite Resin Restorations." However, both groups gave responses that were enthusiastic about ARS. Table 4 shows the significance probabilities (p-values) from the ridit analysis.
View this table:
[in this window]
[in a new window]
|
Table 4. Significance probabilities (p-values) associated with the Cochran-Mantel-Haenszel test of group differences using ridit scores for Questionnaire II questions with statistically significant results
|
|
|
Discussion
|
|---|
Previous studies validating ARS in the dental field10,11 used student questionnaires as the main data collection method. None of these studies were controlled crossover studies, and they did not have the ability to validate the same lecture with and without ARS as was done in our research project. The controlled crossover study design gave strength to this study by distinguishing the effect of ARS on the lecture through comparing a similar lecture given by the same lecturer with and without ARS.
This research study attempted to eliminate any possible bias in lecture delivery and evaluation. The challenge was to incorporate ARS questions into the lecture by providing appropriate feedback to both the students and the lecturer without introducing any new concepts that would give support to the interactive lecture delivery over the conventional lecture delivery.
A possible complication of the study was the inadvertent scheduling of the second lecture on a stress-filled day. The lectures on the topic "Class IV and V Composite Resin Restorations" were given in the afternoon of the same day that the freshman dental students had an important examination in gross anatomy in the morning. The impact of the level of stress in the learning environment as a result of an overwhelming dental curriculum has been shown to influence the success of dental students.18,19
In this research project, the results received on the two different lectures were not similar. For the lecture "Class IV and V Composite Resin Restorations," the data provided no evidence that the two research groups differed in responses to Questionnaire I (questionnaire given to evaluate the overall lecture quality), the immediate posttest, the related question on the unit examination, the related question on the final examinations, or the practical examination. For the lecture "Principles of Dental Bonding," student response data provided statistically significant results supporting the superiority of ARS in question #4 of Questionnaire I, as well as the results of the immediate posttest and the performance skill test. The different results for the two lecture topics may possibly be explained either by the stress of dental students in taking an important examination before the lectures on "Class IV and V Composite Resin Restorations" or by the comprehension dissimilarity of the two lecture topics.
Different results were found when comparing group performance in the practical examinations for the two research lectures. A statistically significant difference favoring ARS was found in favor of the practical examination for the "Principles of Dental Bonding" lecture but not for the practical examination on the "Class IV and V Composite Resin Restorations" lecture. A possible explanation for this difference may be the nature of the dental bonding exercise and Class IV composite resin preparation and restoration exercise. The performance of dental bonding requires knowledge and repetition of various sequential steps. Although specific timing and sequence are crucial to successful bonding, they do not require highly developed psychomotor skills. Understanding the technique can be achieved and performed correctly by a student without skilled physical dexterity. In contrast, refined psychomotor skills play an important role in the outcome of Class IV composite resin restorations. Although the student may have the knowledge of proper preparation and restoration of the Class IV composite resin, without the dexterity to perform the results may be affected.
Results from student assessments suggest that ARS helps students to understand the main concept of the lecture and to identify their weaknesses and strengths in understanding the given material. Students indicated that ARS keeps them fully engaged in the class. The anonymity of the ARS makes it possible for the lecturer to interact with each student in a large classroom and, at the same time, to receive an honest answer. Teacher-student and student-student interactions make a lecture more productive and enjoyable to attend, which makes ARS a productive tool for the stressful environment that dental students are facing. The results of this study agree with other reports in that the majority of students would like to have ARS in their future classes.
In conclusion, these results also indicate that ARS is a promising teaching tool for dental education. ARS can turn the classroom into an active, enjoyable, and informative environment. This technology has the potential to increase student knowledge retention and the ability to transfer the student’s knowledge through psychomotor skill performance when used carefully in the context of the lecture. Further research is needed to evaluate its most effective application in the curriculum.
|
Acknowledgments
|
|---|
Gratitude is expressed to Dr. Murray R. Bouschlicher, director of the operative dentistry techniques course; Mr. Sean Kelley, manager, Dental Educational Media Services; Dr. Deborah S. Cobb, graduate program director, Department of Operative Dentistry; and students of the Class of 2010 at the University of Iowa College of Dentistry.
|
Author Information
|
|---|
Dr. Elashvili is Clinical Assistant Professor, Department of Restorative Dentistry, University of Colorado, School of Dental Medicine; Dr. Denehy is Professor and Head, Department of Operative Dentistry, University of Iowa, College of Dentistry; Dr. Dawson is Professor and Director of Biostatistics, University of Iowa, College of Dentistry; and Ms. Cunningham is Associate Professor, Department of Preventive and Community Dentistry, University of Iowa, College of Dentistry. Direct correspondence and requests for reprints to Dr. Ana Elashvili, University of Colorado, School of Dental Medicine, P.O. Box 6508 F845, Aurora, CO 80045; 303-724-7070 phone; 303-724-7079 fax; ana.elashvili{at}ucdenver.edu. The study was conducted during the residency of Dr. Ana Elashvili at the University of Iowa.
|
REFERENCES
|
|---|
- Field MJ, ed. Dental education at the crossroads: challenges and change. An Institute of Medicine report. Washington, DC: National Academy Press, 1995.
- Kalkwarf KL, Haden NK, Valachovic RW. ADEA Commission on Change and Innovation in Dental Education. J Dent Educ 2005; 69(10):1085–7.[Free Full Text]
- Pyle M, Andrieu SC, Chadwick DG, Chmar JE, Cole JR, George MC, et al. The case for change in dental education. J Dent Educ 2006; 70(9):921–4.[Abstract/Free Full Text]
- Hendricson WD, Andrieu SC, Chadwick DG, Chmar JE, Cole JR, George MC, et al. Educational strategies associated with development of problem-solving, critical thinking, and self-directed learning. J Dent Educ 2006; 70(9):925–36.[Abstract/Free Full Text]
- Mazur E. Peer instruction: a user’s manual. Upper Saddle River, NJ: Simon & Schuster, 1997.
- Dufresene RJ, Wenk L, Mestre JP, Gerace WJ, Leonard WJ. Classtalk: a classroom communication system for active learning. J Comput Higher Educ 1996; 7:3–47.
- Hake RR. Interactive engagement versus traditional methods: a six-thousand student survey of mechanics test data for introductory physics courses. Am J Physics 1998; 66:64–74.
- Abrahamson AL. Teaching with a classroom communication system: what it involves and why it works. Puebla, Mexico: Autonoma de Puebla, 1999.
- Fies C, Marshall J. Classroom response systems: a review of the literature. J Sci Educ Tech 2006; 15(1):101–9.
- Johnson JT. Creating learner-centered classrooms: use of an audience response system in pediatric dentistry education. J Dent Educ 2005; 69(3):378–81.[Abstract/Free Full Text]
- Turpin DL. Enhance learning with an audience response system. Am J Orthod Dentofacial Orthop 2003; 124(6):607.[Medline]
- Homme J, Asay G, Morgenstern B. Utilization of an audience response system. Med Educ 2004; 38(5):575.[Medline]
- Schackow TE, Chavez M, Loya L, Friedman M. Audience response system: effect on learning in family medicine residents. Fam Med 2004; 36(7):496–504.[Medline]
- Pradhan A, Sparano D, Ananth CV. The influence of an audience response system on knowledge retention: an application to resident education. Am J Obstet Gynecol 2005; 193(5):1827–30.[Medline]
- Agresti A. Analysis of ordonal categorical data. New York: Wiley, 1984.
- Fleiss J. Statistical methods for rates and proportions. 2nd ed. New York: Wiley, 1981.
- Bross I. How to use ridit analysis. Biometrics 1958; 14:18–38.[Medline]
- Barberia E, Fernandez-Frias C, Suarez-Clua C, Saavedra D. Analysis of anxiety variables in dental students. Int Dent J 2004; 54(6):445–9.[Medline]
- Henzi D, Davis E, Jasinevicius R, Hendricson W, Cintron L, Isaacs M. Appraisal of the dental school learning environment: the students’ view. J Dent Educ 2005; 69(10):1137–47.[Abstract/Free Full Text]
Top
Abstract
Methods
