Short-Answer Examinations Improve Student Performance in an Oral and Maxillofacial Pathology Course

Methods

This study evaluated the effects of examination question format on student academic performance. Student examination scores in an oral and maxillofacial pathology course were compared among three second-year dental school classes in which different examination question formats (multiple-choice, short-answer, or both; see Table 1⇓) were used. In the 2004–05 class, course grades were determined from scores obtained on an equally weighted combination of short-answer examinations and multiple-choice examinations (SA/MC, 2004–05). In the 2006–07 class, course grades were determined using only multiple-choice examinations (MC, 2006–07) and in the 2007–08 class, course grades were determined using only short-answer examinations (SA, 2007–08).

Examination scores in the prerequisite general pathology course were used to determine whether the three classes were academically comparable. Grade categories (80–100 percent, 70–79 percent, <70 percent) in the general pathology course were used to identify subgroups of students of different academic ability. The examinations procedures (all multiple-choice questions) in the prerequisite general pathology course were the same in all three years.

In the three different academic years in this report, there were no changes in instructors in the oral and maxillofacial pathology course and only slight changes (affecting five of sixty-one contact hours) in instructors in the general pathology course. There were no changes in course content or objectives for either course. Any other changes among the three years, such as slight changes in emphasis or changes in clinical cases and images, were changes that typically occur from year to year.

To improve the validity of scores from all multiple-choice examinations (that is, comparability with short-answer examination scores), we retrospectively applied the standard correction for guessing.⁴ However, only the uncorrected scores were used in the calculation of a student’s official grade in these years. Because the correction for guessing was applied retrospectively, it had no influence on student behavior.

The oral and maxillofacial pathology course at the University of Texas Health Science Center at San Antonio, given during the spring semester to second-year dental students, consisted of fifty hours of didactic lecture and four two-hour examinations. At the outset of this course, students were told explicitly that they were expected to learn and understand each of the following characteristics for the various pathologic processes discussed in the course: etiology, age and sex predilection, most common anatomic location, distinguishing features (clinical, radiographic, microscopic), diagnostic aids and laboratory tests, treatment options, and prognosis. Moreover, students were told they would be examined on these characteristics. Both short-answer and multiple-choice examinations were constructed to test not only recall of memorized facts but also to test student knowledge and understanding of these disease characteristics. The following illustrates a test question on the same information using both short-answer and multiple-choice formats.

A fourteen-year-old female demonstrates hypertelorism, frontal bossing, mandibular prognathism, and palmar pitting. A panoramic radiograph reveals multiple radio-lucent lesions in the right and left posterior mandible and a radiolucent lesion surrounding the right maxillary canine. Based on the clinical and radiographic findings, what is the most likely pathologic process associated with these multiple radiolucent lesions?

Short-answer: Odontogenic keratocyst

Multiple-choice: A. Dentigerous cyst

B. Odontogenic keratocyst

C. Calcifying odontogenic cyst

D. Ameloblastic fibroma

E. Ameloblastoma

Each of the four examinations in the oral and maxillofacial pathology course covered between eleven and thirteen hours of lecture material and comprised 25 percent of the final grade. No comprehensive final examination was given. Students received final course grades based on averages calculated from the scores on the four examinations. The questions on both short-answer and multiple-choice examinations were equally weighted to the topics that were presented prior to each of the four examinations. This was to ensure that a given topic was not stressed more often than another topic and that a topic was not stressed more in one question format than another.

In the SA/MC (2004–05) class, each of the four examinations was divided into two one-hour examinations. The first hour of each examination was based on the presentation of twenty-five clinical cases. Each student was given a written examination containing the clinical histories corresponding to the twenty-five clinical cases that would be projected. Each case consisted of a brief written clinical history and projected gross, microscopic, and/or radiographic findings. Students were advised to respond succinctly to the two short-answer questions for each case and not to use verbose responses. Appropriate answers to short-answer questions typically consisted of one or more sentences or key words. All short-answer examinations were graded solely by the course director (coauthor ACJ). The short-answer questions were graded by identifying key words delineated at the time of construction of the examination. Points were not deducted for spelling errors as long as responses were phonetically correct. If a student gave several answers, only the first answer was evaluated; no partial credit was awarded. The second hour of each examination in the SA/MC (2004–05) class consisted of fifty multiple-choice questions, each of which had one correct answer and four plausible distractors. At the end of the second hour, the multiple-choice examination answer sheets were collected and graded electronically. Since the multiple-choice and short-answer examinations each consisted of fifty questions, they were equally weighted in the calculation of each student’s final grade.

In the MC (2006–07) class, each of the four examinations was comprised of seventy-five multiple-choice questions of construction similar to that described in the foregoing paragraph. In the SA (2007–08) class, each of the four examinations was comprised of seventy-five short-answer questions of construction similar to that described in the foregoing paragraph for the 2004–05 class.

The general pathology course at the University of Texas Health Science Center at San Antonio is given in the fall semester to second-year dental students and immediately precedes the oral and maxillofacial pathology course. The course and examination methods were the same for all academic years included in this report. The course consisted of sixty-one hours of didactic lecture, four two-hour review sessions, and four two-hour examinations. The review sessions were structured in a question and answer format. Each faculty member who had previously presented didactic information for the upcoming examination presented a brief verbal review of their topics. Students were then allowed to ask questions, the answers to which were discussed by the faculty member. This procedure was repeated until there were no further questions. Each of the examinations consisted of seventy-five multiple-choice questions with one correct answer and four distractors; test construction strategies were similar to those described for the oral and maxillofacial pathology course. The multiple-choice questions covered information presented in the lectures and reading assignments in the period immediately preceding each examination. Each two-hour examination comprised 25 percent of the final course grade. No comprehensive final examination was given. Students received a final course grade based on the averages calculated from the four two-hour examinations. Each of the four examinations covered between thirteen and nineteen hours of lecture material. When the multiple-choice questions were constructed, the questions were equally weighted to the topics that were presented prior to each of the four examinations.

We analyzed data from students who completed all four examinations in the oral and maxillofacial pathology course and all four examinations in the prerequisite general pathology course. The analyses presented in this report were based on eighty-eight students in the SA/MC (2004–05) class, eighty-eight students in the MC (2006–07) class, and eighty-six students in the SA (2007–08) class. This study was approved by the Institutional Review Board of the University of Texas Health Science Center at San Antonio.

The correction for guessing utilized was a modification to the common grading method for multiple-choice examinations (number-correct or number-right scoring) in which zero points are assigned for an incorrect answer and full credit is given for a correct answer.^4,7 Since each multiple-choice question had five possible answers, the standard correction for guessing consisted of awarding -¼ for an incorrect answer, 0 for a question not answered, and +1 for a correct answer. The retrospective application of the correction for guessing had no influence on student behavior, and students answered all multiple-choice questions because they expected to benefit by guessing. The retrospective correction for guessing therefore was equivalent to applying a straight line adjustment to the multiple-choice examination scores such that a grade of 100 percent was unchanged and a grade of 20 percent was adjusted to zero. The equation of this straight line was: Corrected Score (%) = 1.25 [Uncorrected Score (%) – 20].⁴ The grade categories used in this report (80–100 percent, 70–79 percent, <70 percent) were based on corrected multiple-choice examination scores and therefore correspond to uncorrected scores of 84–100 percent, 76–83 percent, and <76 percent.

Means were compared between classes using analysis of variance.⁸ Cumulative relative frequency distributions were used to graphically show distributions of numerical scores. Numerical course averages in the oral and maxillofacial pathology course and in the general pathology course were used to assign course grade category as 80–100 percent, 70–79 percent, and <70 percent. Frequencies in grade categories were analyzed using the chi-square test for independence;⁸ if low expected frequencies were encountered, an exact procedure was used to obtain the P-value. We classified students’ performance based on their grade category in the general pathology course (an estimate of student academic ability) and then analyzed their respective numerical scores in the oral and maxillofacial pathology course by these classifications and academic class in a two-way analysis of variance⁸ to investigate Aptitude-Treatment Interactions.⁵ All calculations were carried out using SAS 9.1.3 (SAS Institute, Cary, NC).

Results

Oral and Maxillofacial Pathology Course

Student performance assessed by multiple-choice examinations.

The class means and grade category distributions of the multiple-choice examination scores in the oral and maxillofacial pathology course are given in Table 2⇑. Utilization of half short-answer examinations was associated with enhanced student performance on the multiple-choice examinations as indicated by a significantly higher mean score for the SA/MC (2004–05) class than for the MC (2006–07) class (81.6 vs. 78.4, P=0.0101).

The grade category distributions for the multiple-choice examinations in the oral and maxillofacial pathology course are shown in Figure 1⇓ and Table 2⇑. The two pie charts in the left-hand column of Figure 1⇓ show little difference in the grade distributions for the two classes in the general pathology course (not significantly different as stated earlier). The grade distributions of the MC (2006–07) class in the general pathology and oral and maxillofacial pathology courses were not different (P=0.8211); this is not surprising because the examination formats were the same, that is, all multiple-choice questions. However, in the SA/MC (2004–05) class, student performance on the multiple-choice examinations in the oral and maxillofacial pathology course was significantly better than performance in the general pathology course (P=0.0010). The grade category distribution in the oral and maxillofacial pathology course based on the multiple-choice examinations in the SA/MC (2004–05) class reflected a higher fraction of students in the 80–100 percent grade category than in the MC (2006–07) class (56.8 percent vs. 43.2 percent, P=0.0704, Table 2⇑). There also was a lower fraction of students in the <70 percent grade category in the SA/MC (2004–05) class than in the MC (2006–07) class (5.7 percent vs. 18.2 percent, P=0.0105, Table 2⇑). Thus, by utilizing a short-answer format examination in the oral and maxillofacial pathology course in the SA/MC (2004–05) class, the students’ performances improved relative to the grade distribution anticipated from the general pathology course.

Figure 1.

Distribution (percent of students) of grade category in the general pathology course and oral and maxillofacial pathology course based on multiple-choice examination

Student performance assessed by short-answer examinations.

The mean score on the short-answer examinations for the SA/MC (2004–05) class was not significantly different from the mean score for the SA (2007–08) class (82.8 vs. 82.8, P=0.9628, Table 2⇑); and the grade category distribution was not significantly different between the SA/MC (2004–05) class and the SA (2007–08) class (Table 2⇑, P=0.2607). Thus, there was no association of student performance with utilization of half short-answer examinations relative to using all short-answer examinations.

Oral and Maxillofacial Pathology Course Performance for Students of Different Ability

The foregoing analyses demonstrated improved overall class performance in the oral and maxillofacial pathology course when short-answer format questions were used. We further investigated how individual students of differing academic ability, as estimated by their grade category in the general pathology course, performed in the oral and maxillofacial pathology course relative to examination question format.

Student performance assessed by multiple-choice examinations.

The cumulative frequency distributions of individual student averages on the multiple-choice examinations for the oral and maxillofacial pathology course by grade category in the general pathology course are shown in the left-hand column of Figure 2⇓. These cumulative distributions indicated that students in all three grade categories (80–100 percent, 70–79 percent, <70 percent) in the general pathology course performed better in the oral and maxillofacial pathology course in the SA/ MC (2004–05) class when short-answer questions comprised half of the examinations. Furthermore, the magnitudes of the improved performances were similar in all three grade categories determined from the prerequisite general pathology course.

Figure 2.

Cumulative relative frequency distributions of average student scores from four multiple-choice examinations and four short-answer examinations in the oral and maxillofacial pathology course by grade category in the prerequisite general pathology course and academic class

The foregoing impressions were confirmed quantitatively. For all general pathology grade categories, the average scores (Table 3⇓) on the multiple-choice examinations in the oral and maxillofacial pathology course were significantly higher in the SA/MC (2004–05) class than in the MC (2006–07) class (80–100 percent grade category, 88.5 vs. 84.4, P=0.0012; 70–79 percent grade category, 80.7 vs. 75.6, P=0.0001; <70 percent grade category, 72.6 vs. 69.1, P=0.0408). The differences in mean score on the multiple-choice examinations (Table 3⇓) between the SA/MC (2004–05) class and the MC (2006–07) class were similar in all three general pathology grade categories (differences calculated from mean scores in Table 3⇓ were 4.0 for the 80–100 percent grade category, 5.1 for the 70–79 percent grade category, and 3.5 for the <70 percent grade category), indicating that there was no Aptitude-Treatment Interaction (academic class by grade category interaction, P=0.7075).

View this table:

• View inline
• View popup

Table 3.

Mean scores on the multiple-choice examinations and the short-answer examinations in the oral and maxillofacial pathology course by grade category in the general pathology course (retrospectively corrected) and academic class

The grade category distributions of the multiple-choice examinations in the oral and maxillofacial pathology course are given in Figure 3⇓ and Table 4⇓. All of the students in the 80–100 percent grade category in the general pathology course achieved this target grade category in the oral and maxillofacial pathology course in the SA/MC (2004–05) class, in which short-answer examinations were included. However, in the MC (2006–07) class, there was a significant fraction of these students (80–100 percent grade category in the general pathology course) that dropped into the 70–79 percent grade category in the oral and maxillofacial pathology course (0 percent vs. 24.4 percent, P=0.0019). For the students in the 70–79 percent grade category in the general pathology course, the fraction of students moving up into the 80–100 percent grade category in the oral and maxillofacial pathology course approximately doubled in the SA/MC (2004–05) class relative to the MC (2006–07) class (51.4 percent vs. 23.3 percent, P=0.0203). Also, in the SA/MC (2004–05) class, none of these students (70–79 percent grade category in the general pathology course) moved down into the <70 percent grade category in the oral and maxillofacial pathology course, whereas in the MC (2006–07) class, 20 percent of students fell into the <70 percent grade category (P=0.0072). Among those students in the <70 percent grade category in the general pathology course, there was a significantly greater fraction of students that remained in this grade category in the oral and maxillofacial pathology course in the MC (2006–07) class than in the SA/MC (2004–05) class (58.8 percent vs. 23.8 percent, P=0.0281).

View this table:

• View inline
• View popup

Table 4.

Grade category distribution based on multiple-choice examinations and short-answer examinations in the oral and maxillofacial pathology course by grade category (retrospectively corrected) in the general pathology course and academic class

Figure 3.

Distribution of grade category in the oral and maxillofacial pathology course based on multiple-choice examination by academic class and grade category in the general pathology course

Student performance assessed by short-answer examinations.

The cumulative frequency distributions of individual student averages on the short-answer examinations in the oral and maxillofacial pathology course by grade category in the general pathology course are shown in the right-hand column of Figure 2⇑. These cumulative distributions clearly indicated that, within the three general pathology grade categories, students in the SA/MC (2004–05) class and the SA (2007–08) class performed comparably in the oral and maxillofacial pathology course. For students in all general pathology grade categories, the average scores (Table 3⇑) on the short-answer examinations were not significantly different between the SA/MC (2004–05) class and the SA (2007–08) class (80–100 percent grade category, 90.3 vs. 88.9, P=0.2935; 70–79 percent grade category, 81.6 vs. 81.4, P=0.8635; <70 percent grade category, 73.4 vs. 74.2, P=0.6613). The differences in mean scores between the SA/MC (2004–05) class and the SA (2007–08) class were similar in all three grade categories, indicating that there was no Aptitude-Treatment Interaction (academic class by grade category interaction, P=0.5974).

The grade category distribution in the oral and maxillofacial pathology course based on the short-answer examinations are given in Figure 4⇓ and Table 4⇑. There was no significant difference in performance on short-answer examinations in the oral and maxillofacial pathology course between the SA (2007–08) class and the SA/MC (2004–05) class, regardless of student ability as assessed by grade category in the general pathology course.

Figure 4.

Distribution of grade category in the oral and maxillofacial pathology course based on short-answer examination by academic class and grade category in the general pathology course

Discussion

This study has shown that utilization of more challenging short-answer questions, as opposed to cued multiple-choice questions, on examinations in a second-year dental school oral and maxillofacial pathology course was associated with significantly improved student academic performance. Moreover, this improved performance was observed in students of all academic abilities as determined by their scores (80–100 percent, 70–79 percent, <70 percent) in the prerequisite general pathology course, in which multiple-choice questions were used exclusively. Of note, utilization of all short-answer examinations was not associated with any additional improved performance compared to having only half of the examinations comprised of short-answer format questions.

The results of a comparison between the two classes with the most divergent examination question formats are worthy of note. The SA (2007–08) class (all short-answer) significantly outperformed the MC (2006–07) class (all multiple-choice) (class means 82.8 vs. 78.4, P=0.0003, Table 2⇑). It was striking to us that, even though the MC (2006–07) class (all multiple-choice) had the advantage of cueing and was also rewarded for educated guessing, they did not achieve the academic performance of the SA (2007–08) class (all short-answer). Moreover, the students in the SA (2007–08) class were not accustomed to short-answer format questions since nearly all courses in the first two years of our dental school curriculum exclusively utilize multiple-choice examinations.

Our interpretation of the improved performance associated with the utilization of short-answer questions is that the students’ study regimens likely differed from and were more effective than those they would have used if the examinations were comprised of only cued multiple-choice questions. The study regimens used to prepare for short-answer examinations also resulted in improved scores on the multiple-choice format questions when this format represented half of the examinations and short-answer format questions the other half (SA/MC, 2004–05 class) compared to all multiple-choice format questions (MC, 2006–07 class). These results are consistent with the results from Balch’s study that found that, faced with the expectation of short-answer examinations, students were motivated to study differently and also more effectively.⁹ Furthermore, Balch reported that students anticipating a short-answer examination had higher perceived anxiety levels while studying for the examination relative to the anxiety levels of students anticipating a multiple-choice examination; however, the relative perceived anxiety level between the two groups was reversed during the actual examination, which, in fact, was the same multiple-choice examination for both groups.

The results reported here strongly support the use of short-answer format questions. We believe that utilization of other types of examination questions, such as essay format questions, which require even more integration of information than short-answer questions, would further motivate students to modify their study regimens and result in an even better comprehension of the subject matter. Our results using half short-answer questions suggest that only a fraction of the examination questions would need to be essay format in order to achieve such benefits.

Previously, we documented in the same oral and maxillofacial pathology course the beneficial effects on student performance of prospective (with the students’ awareness) implementation of correction for guessing on multiple-choice questions (2005–06 class). The examination format in the 2005–06 class was identical to that in the SA/MC (2004–05) class—that is, one-half of the examination questions were multiple-choice questions and the other half short-answer questions.⁶ Prospectively implemented correction for guessing was associated with an additional improvement in academic performance that was over and above the improvement associated with the use of short-answer questions reported here. However, in marked contrast to the current study, the improvement was seen only in those students whose previous academic performance in the general pathology course placed them in the lowest part of the grade distribution;⁶ furthermore, the improved performance by these students was not only realized on the multiple-choice examinations but on the short-answer examinations as well. Because the correction for guessing only changed performance in a group of underachieving students, we interpreted this observation as representing an Aptitude-Treatment Interaction.⁵ Thus, we believe that the reasons underlying the overall improved student performance seen in the present study by inclusion of short-answer questions on examinations were different from the improved performance attained by the underachieving students after imposition of correction for guessing of multiple-choice questions.⁶

Finally, we believe it is important to draw a clear distinction between utilizing examinations solely for the purpose of evaluating students’ academic achievements and using examinations as a vehicle to challenge students to learn more effectively. Our study has shown that the utilization of short-answer examinations created a more challenging learning environment and motivated students to adopt more effective study regimens.