J Dent Educ. 70(12): 1355-1361 2006
© 2006 American Dental Education Association
Educational Methodologies |
Student Accuracy and Evaluation of a Computer-Based Audience Response System
Robert G. Holmes, D.M.D., M.S.;
John S. Blalock, D.M.D.;
Merle H. Parker, D.D.S., M.S.;
Van B. Haywood, D.M.D.
Key words: Socratic method, inquiry-based learning, cooperative learning
Submitted for publication 04/23/06;
accepted 08/29/06
 |
Abstract
|
|---|
We have incorporated an audience response system into our curriculum to increase student interaction in the teaching process. Classroom Performance System (CPS) is a computer-based audience response system that allows students to answer questions posed to the entire class by entering responses on a keypad. The responses are tallied and displayed on the classroom screen for all students to see. The purpose of our study was to determine student accuracy using the system with three different methods of administering questions. A secondary purpose was to assess students’ perceptions about using the system. Our hypothesis for the study was that there should be no difference in volunteer accuracy or questionnaire responses to the three methods of gathering responses. Sixty-two dental students volunteered to participate. Using three methods (projected on a screen, verbal, and written), volunteers were given "responses" to enter into the system using CPS wireless remote answering devices. In the projected and verbal formats, the teacher managed the assessment by controlling the pace of input. In the written format, students were given responses on paper to input into the system at their own pace. At the end of the sessions, volunteers completed an anonymous questionnaire regarding their experiences with the system. The accuracy of responses was similar in the teacher-managed assessments (projected and verbal format). There was a statistical difference in the accuracy of responses in the student-managed assessment (p=<0.000001). Questionnaire responses also showed that students preferred teacher-managed assessments. The hypothesis was disproved. The overall response to this audience response system and its methods of gathering information was very positive.
In the fourth century B.C.E., Plato wrote the "Socratic Dialogues" describing Socrates’ method of teaching in which questions were periodically posed to students and the students were forced to use their information to reach a conclusion on their own.1,2 Since then, lecturers have used the "Socratic Method" to engage their students one on one in class in the same instructional dialog.
Recently, educators have tried to improve on the Socratic Method by using classrooms wired with devices that allow students to provide answers anonymously to questions posed in class. This method of posing the same question to everyone at once eliminates potential embarrassment by not requiring the individual student to answer aloud. Instead, the entire audience participates simultaneously: all students enter a response on a keypad, and the responses are tallied and displayed for the entire class to see.
In previous studies, it has been concluded that traditional didactic lecturing methods are the least effective method of imparting information to students.3 In spite of the lecture topic, these traditional "one-way" methods are especially not effective at stimulating the audience.3 In a previous survey, students felt that lectures in which questions were asked were more stimulating than those without questions.4 Audiences have also commented in previous studies that an audience response system enhances the educational experience.5 This system of teaching has a desirable side effect of increasing the level of attention of the participants.6,7
For three years, we have utilized Classroom Performance System (CPS) in the dental school curriculum. CPS is a computer-based audience response system that allows students to answer questions by entering responses on a keypad. The responses are tabulated and displayed on the classroom screen. Originally, the system was used only for review during lectures to ensure students were grasping lecture topics in Dental Anatomy and Occlusion. Soon the system was used to administer short quizzes in Statistics. Then, as faculty were introduced to the system, it was used in Operative Dentistry and Fixed Prosthodontics. Most recently, it has been used to prepare students for the National Boards in the above areas as well as Pharmacology.
CPS allows the lecturer to administer questions in several formats. Questions can be preplanned, inserted into the lecture, and projected through a computer at designated spots in the lecture. In another classroom questioning strategy, the lecturer spontaneously stops during the discussion of a difficult concept and poses a question verbally without projecting it on the screen to determine if students understand the material or to ask students to apply the information to a problem. These verbal questions can be tabulated with an audience response system such as CPS. In either method, all students are polled instantly to gauge their understanding.
However, there has been concern expressed about the students’ capacity to accurately enter responses using an electronic key pad. Therefore, this study was conducted to compare the accuracy of student responses when answering projected questions, verbally stated questions, and written questions.
|
Methods
|
|---|
Classroom Performance System (eInstruction, Denton, Texas, 1-888-707-6819, www.einstruction.com) is a system designed for ease of use in the classroom setting. The system consists of a set of handheld response pads, an infrared sensor, and the software for analysis of results (see Figure 1
). In this instance, we used sixty-two response pads. However, the system can be used with up to 512 response pads. An additional sensor is recommended for every ninety response pads.
Sixty-two first-year dental students volunteered to assist in the study of this teaching method. The first-year class was selected because of their lack of experience with this teaching device. Students ranged in age from twenty-one to forty. The class was 66 percent male and 34 percent female.
This study was administered in a classroom in the dental school. An LCD projector and computer running the CPS software (version 3.4) were used. One professor administered the exercise while the others monitored the volunteers for difficulties.
In this study, there were no questions. The volunteers were given prearranged "responses" (using letters "A" through "E") to input into the system using the three methods: projected images, verbally stated responses, and written responses. For example, in the teacher-managed (projected) methods, the letter "A" was projected on screen, and the volunteer was to enter that response. In the teacher-managed (verbal) method, the instructor would say the letter "A." In the student-managed (written) assessment, the volunteers were given a sheet of paper with the responses to enter.
The first method was teacher-managed assessment (projected using CPS), which consisted of displaying (using Microsoft PowerPoint 2003) the responses ("A," "B," "C," "D," or "E") that the volunteers were to enter. The CPS software can perform this same function independently of PowerPoint, but the overwhelming majority of the lecturers in our school utilize PowerPoint. We decided to simulate classroom conditions as closely as possible. The answer was displayed, and after all volunteers entered a response, the teacher moved forward to begin the procedure again. Twenty simulated questions were administered in this format.
The teacher-managed assessment (i.e., verbal questioning) consisted of the lecturer announcing the "answers" for the students to input. This simulated the students being asked ad hoc and impromptu questions during a lecture. As before, all volunteers answered, and the teacher moved forward. Twenty simulated questions were also administered in this format. After these two sessions (thirty minutes total), the volunteers were dismissed for the day and asked to return the second day to complete the study.
The next day, the same sixty-two volunteers reported for the student-managed assessment. The same instructor administered this session. This assessment consisted of the twenty "answers" being distributed on paper to each volunteer at the beginning of the session. Once the CPS program was started, the volunteers entered the responses using the keypads as before, but they were allowed to proceed at their own pace. When all volunteers had completed entering the twenty responses, the administrator stopped the assessment.
The sessions were designed to last as long as a normal lecture. The student-managed assessment was designed to simulate a quiz or short test. Therefore, volunteers were allowed all the time they needed to complete the task to their satisfaction.
At the end of the last session, the volunteers were asked to complete a questionnaire regarding the three methods and their response to the system as a whole and its applications to the dental school classroom. Volunteer accuracy in entering the preplanned response was tabulated using the CPS software (version 3.4). Statistics were computed using NCSS (Number Crunching Statistical System, 2004 version, Kaysville, Utah, 800-898-6109, www.ncss.com).
|
Results
|
|---|
The student volunteers’ accuracy levels were high using all methods. Their accuracy in entering a preplanned response during the teacher-managed assessment (projected) ranged from 97 to 100 percent; the mean accuracy was 98.95 percent. The accuracy of their response entry on the CPS pad during the teacher-managed assessment (spontaneous verbal questioning) also ranged from 97 to 100 percent; the mean accuracy was 99.4 percent. The accuracy of response entry during the student-managed assessment (students answer questions presented to them on paper) ranged from 90 to 97 percent; the mean accuracy was 93.6 percent.
Statistical analysis was performed using a one-way analysis of variance (ANOVA) with a Tukey-Kramer post hoc test to compare groups of results. Differences were considered significant at alpha=0.05.
The ANOVA results are shown in Table 1. The probability value was significant (p=<0.000001). As can be seen in Table 2 and Figure 2, the teacher-managed assessments (displayed and verbal) were statistically different from the student-managed assessment (written responses).
View this table:
[in this window]
[in a new window]
|
Table 2. Tukey-Kramer multiple comparison test comparing the three groups of results and showing teacher-managed assessments statistically different from student-managed assessment
|
|
View larger version (10K):
[in this window]
[in a new window]
|
Figure 2. Box plot of percent correct answers for each of the methods of administering questions using CPS
|
|
Student opinion of the methods of administering questions was varied, but with a clear preference for projected questions and teacher management of the testing, with over 91 percent liking/strongly liking and only 3.3 percent disliking this method. Verbal questioning was liked/strongly liked by 75 percent and disliked by 13 percent of respondents. The student-managed assessment was liked/strongly liked by only 62 percent and disliked by one-quarter of the participants (Figure 3).
View larger version (23K):
[in this window]
[in a new window]
|
Figure 3. Results from participant survey on questioning methods: A) projected question/teacher-managed assessment; B) verbal question/teacher-managed assessment; C) written question/student-managed assessment
|
|
When asked if they would feel comfortable using an audience response system such as CPS in class, 88 percent of the students agreed/strongly agreed, with less than 2 percent disagreeing. Over 93 percent reported that they liked the ability to participate anonymously in class (Figure 4).
View larger version (17K):
[in this window]
[in a new window]
|
Figure 4. Responses to questions about audience participation systems: A) "I would feel comfortable using this system in class"; B) "I like participating anonymously in class"
|
|
|
Discussion
|
|---|
For the past three years, CPS has been a useful adjunct to the traditional one-way didactic lecture at our dental school. Lecturers have been surprised at the small amount of training necessary to begin implementing the system in their courses. Students have been overwhelmingly enthusiastic about using the system in daily lectures.
As more areas of the school become aware of the system’s capabilities, the uses increase. Students have used CPS to elect class officers, and faculty have used it to elect students to honor societies.
Students were given space on the questionnaire to offer comments on subjects not addressed in the didactic portion of the form. Ninety percent of respondents commented that they enjoyed using the system and appreciated the chance to participate in a lecture anonymously. One-third of respondents suggested that the system is better used on quizzes of ten questions or less.
|
Conclusion
|
|---|
Accuracy is the students’ ability to enter the correct answer. The students were given (in three different forms) the "correct" answer to input using the audience response system. This study determined that there was a difference in students’ accuracy from method to method with entry of a written response being the least accurate (93.6 percent) versus approximately 99 percent for projected or spoken responses.
According to the questionnaires, students prefer the teacher-managed assessment methods (projected or verbal questions) over the student-managed method of entering responses at their own pace. According to questionnaire written responses, students would prefer the system’s use in lecture for a few questions rather than for quizzes.
Our experience has been extremely positive. We recommend the implementation of an audience response system such as CPS.
|
Footnotes
|
|---|
Dr. Holmes is an Assistant Professor, Department of Oral Rehabilitation, Medical College of Georgia School of Dentistry; Dr. Blalock is an Assistant Professor, Department of Oral Rehabilitation, Medical College of Georgia School of Dentistry; Dr. Parker is the Prosthodontic Director of the Veterans Administration Hospital, Augusta, Georgia; and Dr. Haywood is a Professor, Department of Oral Rehabilitation, Medical College of Georgia School of Dentistry. Direct correspondence and requests for reprints to Dr. Robert G. Holmes, Medical College of Georgia School of Dentistry, Department of Oral Rehabilitation, AD-3231, Augusta, GA 30921-1260; 706-721-2881 phone; 706-721-8349 fax; rholmes{at}mail.mcg.edu.
|
REFERENCES
|
|---|
- Socratic method: a Socratic method research portal. At: www.socraticmethod.net. Accessed: April 20, 2006.
- Stevenson DC. The Internet classics archive. At: http://classics.mit.edu/plato/euthyfro.html. Accessed: April 20, 2006.
- Butler JA. Use of teaching methods within the lecture format. Med Teach 1992; 14(1):11–25.[Medline]
- Papp KK, Miller FB. The answer to stimulating lectures is the question. Med Teach 1996; 18(2):147–50.
- Latessa R, Mouw D. Use of an audience response system to augment interactive learning. Fam Med 2005; 37(1): 12–4.[Medline]
- Copeland HL, Longworth DL, Hewson MG, Stoller JK. Successful lecturing: a prospective study to validate attributes of the effective medical lecture. J Gen Intern Med 2000; 15:366–71.[Medline]
- Miller R, Ashar BH, Getz KJ. Evaluation of an audience response system for the continuing education of health professionals. J Contin Educ Health Prof 2003; 23: 109–15.[Medline]
Top
Abstract
Methods
