J Dent Educ. 68(6): 656-659 2004
© 2004 American Dental Education Association
Educational Methodologies |
Influence of Online Formative Assessment Upon Student Learning in Biomedical Science Courses
Byron L. Olson, Ph.D.;
James L. McDonald, Ph.D.
Dr. Olson is Professor, Department of Oral Biology, and Dr. McDonald is Professor, Department of Oral Biology and Associate Dean for Dental Education—both at the Indiana University School of Dentistry. Direct correspondence and requests for reprints to Dr. Byron L. Olson, Department of Oral Biology, Indiana University School of Dentistry, 1121 W. Michigan Street, Indianapolis, IN 46202; 317-274-5419 phone; 317-274-2419 fax; bolson{at}iupui.edu.
Key words: formative, summative, assessment
Submitted for publication 12/18/03;
accepted 03/31/04
 |
Abstract
|
|---|
Research suggests that high-quality formative assessment has a strongly positive effect upon student learning. Unfortunately, formative assessment does not appear to be frequently used in didactic dental curricula. Our hypothesis was that providing students with practice online exam questions would enable those who voluntarily took the exam to perform better on subsequent summative exams than did students who did not utilize this opportunity. A test bank of exam questions was written for dental students enrolled in two different biomedical science courses. Half the questions were arbitrarily assigned to an electronic test site; the other half were used as a written summative classroom exam taken later. Students who took the online formative exam in the first semester course scored 8.8 percent higher on the summative exam than did those who did not take the practice exam. This represents almost a full letter grade higher for the formative exam-takers. Students who took the formative online exam in the second semester course scored 5.2 percent higher on the summative exam than did the non-takers. Both of these differences were statistically significant. Under these experimental circumstances, providing formative online exams appeared to promote student performance as reflected by higher scores on the summative exams.
It is widely accepted that assessment has many powerful effects on student learning.1–2 These effects include not only what is learned, but also students’ approaches to learning.3 Formative assessment occurs when educators feed information back to students in a "low-stakes" manner that enables the student to learn better and engage in a self-reflective process regarding the feedback.4–6 Its purpose is to provide both feedback on performance and suggestions for improvement.7–8 Such an assessment can be provided using a wide range of methods. In contrast, summative assessment represents a "higher-stakes" evaluation of student learning at a given point in time, generally used to assign grades to students.9–10 Summative assessment requires making a judgment about the learning that has occurred and is not designed to provide the immediate feedback useful for helping students during the actual learning process. If the primary purpose of evaluation is to support high-quality learning, then formative assessment ought to be understood as the most important assessment practice.11
Paper-based formative assessments have a number of limitations, foremost of which is that students must all be gathered together at one specific site and at a specific time. Individualized feedback is time-consuming, and analysis of question reliability and validity can be tedious.12 These considerations constitute a powerful argument for moving to an online formative assessment process.12–14 Numerous faculty development workshops provided at our institution have strongly endorsed the value of formative assessment in promoting student learning and encouraged a broader use of this educational strategy. However, within the context of a dental school curriculum in general, and basic science courses specifically, little information is available to convince faculty that the benefits to be gained outweigh the additional expenditure of resources. Thus, this pilot study was conducted in two different courses in the first-year basic science curriculum of the school to evaluate the effectiveness of providing students with opportunities for formative assessment. The Molecular Cell Biology course occurs in the first semester of the first year, and the Endocrinology module of the Systems Approach to the Biomedical Sciences course occurs in the second semester of the first year.
|
Methods
|
|---|
The student population for this study was the first-year dental class at the Indiana University School of Dentistry (n=98). Formative, online exam questions were provided for students at two points in the first-year curriculum. The initial use of the practice questions occurred in the first semester as part of the Molecular Cell Biology (MCB) course. The follow-up use of the practice questions occurred in the second semester as part of the Endocrine Module in the Systems Approach to the Biomedical Sciences (SABS) course. The authors wrote a test bank of fifty-six exam questions for the MCB course and Endocrine module. Half of the questions in each of the courses were arbitrarily assigned to an electronic test site; the other half were used as a written summative classroom exam taken a few days later. The questions on both exams were of approximately equal content. The electronic test site is called OnCourse. OnCourse was conceived and designed by educators, for educators, by the WebLab group, which is a part of Indiana University’s Advanced Information Technology Laboratory. This site includes a tools section, which enables teachers to give and assess online testing as well as to record and monitor grades. The formative exam results were immediately available to the students electronically once they completed the test, as were the correct answers to each question.
In the first semester MCB course, forty-five of the ninety-eight students chose to take the online formative exam, which consisted of twelve multiple-choice questions. In the second semester, forty-five of the ninety-eight students again chose to take the online formative exam consisting of sixteen multiple-choice questions. All the students took the corresponding summative exam during the regularly scheduled written exam periods. All the online formative and summative questions were of the single-answer, multiple-choice format. In addition to answering the questions posed on the online formative exams and on the written summative exams, the students provided an online assessment of the courses/ modules at their conclusion including a question stating, "The use of the electronic formative (practice) exam questions helped me prepare more effectively for my biweekly exam," using a Likert scale ranging from 1 (strongly agree) to 5 (strongly disagree).
Data analysis was performed by one-way analysis of variance (ANOVA). Comparisons were made between the formative and summative exam scores for the students who took the formative exams. Also, comparisons were made between the summative exam scores of those who did not take the formative exam and those who did take the formative exam, and comparisons were made between the course grades for the formative exam takers and the non-formative exam takers. The effect sizes were calculated for the above comparisons. Finally, the accumulative GPAs of those students who took the formative exams were compared with the accumulative GPAs of those students who did not take the formative exams. The level of significance was p<0.05.
|
Results
|
|---|
The results of the study are summarized in Tables 1
(for the MCB course) and 2 (for the Endocrine module SABS course). Table 1 shows that the mean score of the forty-five students who took the twelve-question formative online exam for the MCB course was 8.73 (72.8 percent). The mean score on the MCB summative exam for these students was 10.16 (84.7 percent). The differences in formative exam scores and summative exam scores were significant (F [1,88] = 12.33, p<0.001). The effect size was 0.74. Those students who did not take the formative exam had a mean score of 9.34 (77.8 percent) on the summative exam. The differences in summative exam scores between students who completed the formative assessment (10.16) and students who did not (9.34) were statistically significant (F [1,96] = 6.89, p<0.05). The effect size was 0.54. Those students who took the formative exam also had a higher mean final course score than the students who did not complete the formative exam. The difference in final course scores (final grade percentages) between the formative exam-takers and the non-takers was not statistically significant. On their course assessment feedback, the students who took the formative exam had a mean score of 2.02 on the Likert scale, indicating that they agreed that the formative exam questions helped them better prepare for the summative exam questions.
Table 2
indicates the mean score of the forty-five students who took the sixteen-question formative online exam for the Endocrine Module of SABS was 10.09 (63.0 percent). When these students took the summative written exam, their mean score was 14.31 (89.4 percent). The differences in formative vs. summative exam scores were statistically significant (F [1,88] = 31.99, p<0.001). The effect size was 1.19. Those students who did not take the formative exam in the Endocrine Module had a mean score of 13.6 (85 percent) on the summative exam. The difference in performance between the formative exam takers and the non-takers was again statistically significant (F [1,96] = 5.95, p<0.05). The effect size was 0.5. Those who took the formative exam also had a higher mean final course score than did those who did not take the formative exam. The difference in performance between the formative exam takers and the non-takers barely reached significance (F [1,96] = 4.22, p=0.043). The effect size was 0.42. The students who took the formative exam had a mean score of 1.71 on the Likert scale of the course assessment instrument, indicating that they collectively fell between "agreeing" and "strongly agreeing" that the formative exam questions helped them prepare better for the summative exam questions.
|
Discussion
|
|---|
It is clear that those students choosing to use the formative exam questions as a learning tool had higher scores on their summative exams. This result was noted in both the first semester Molecular Cell Biology course and in the second semester Endocrinology module of the Systems Approach to the Biomedical Sciences course. This result is consistent with suggestions in the medical and nursing education fields regarding the potential value of this type of assessment.15–18 More specifically, our findings are consistent with those of Buchanan14 who reported that a World Wide Web-based formative assessment package used in undergraduate psychology courses enhanced learning. Similar findings were also reported by Velan et al.12 This latter group concluded that online formative assessment among undergraduate medical students taking pathology is an effective method of promoting learning. Many of these formative assessments conducted at other institutions reflect innovative educational methods in medical education and have been authored using Question-Mark Perception and the World Wide Web. With regard to the research results presented in this article, we are unaware of similar reports in which formative assessment via electronic methodologies has been evaluated in dental education and the results published.
Perhaps the most pressing issue in correctly interpreting the significance of our results is the possibility that it was primarily the "better" students who availed themselves of the educational opportunity to take the formative exams and this was why the formative exam-takers performed better in the subsequent summative exams. However, as one measure, a comparison of dental course GPAs between the two groups does not support this particular contention. There were no statistical differences between the accumulative GPAs of either group or the accumulative GPAs of either group minus science courses that utilized the formative exam questions.
|
Summary
|
|---|
This project clearly represents only a modest attempt to provide formative assessment as a method for improving student performance. However, based on the relative success of this initial effort, subsequent efforts can now justifiably be of a more ambitious nature. Since this specific report deals with only a small portion of the overall biomedical science curriculum, it is possible that if online formative assessment was extended to the entire basic science curriculum, the impact might be considerably greater. Further investigation will be necessary to determine whether the statistical differences noted in this research will be of practical value.
|
REFERENCES
|
|---|
- Croak T. The impact of classroom evaluation practices on students. Rev Educ Res 1988;55:438–81.
- Sebatanne E. Assessment and classroom learning: a response to Black and Wiliam. Assessment Educ 1998;5:123–30.
- Newble D, Jaeger K. The effect of assessment and examination on the learning of medical students. Med Educ 1983;17:165–71.[Medline]
- Ramaprasad A. On the definition of feedback. Behav Science 1983;28:4–13.
- Sadler D. Formative assessment and the design of instructional systems. Instr Science 1989;18:119–44.
- Kurosawa K, Harackiewicz J. Test anxiety, self-awareness, and cognitive interference: a process analysis. J Pers 1995;63:931–51.
- Rolfe I, McPherson J. Formative assessment: how am I doing. Lancet 1995;345:837–9.[Medline]
- Schultz P, Davis H. Emotions and self-regulation during test taking. Educ Psychol 2000;35:243–56.
- Black P. Testing: friend or foe? Theory and practice of assessment and testing. London: Falmer Press, 1998.
- Biggs J. Assessment and classroom learning: a role for summative assessment? Assessment Educ 1998;5:103–10.
- Black P, Wiliam D. Assessment and classroom learning. Assessment Educ 1998;5:7–74.
- Velan G, Kumar R, Dziegielewski M, Wakefield D. Web-based self-assessments in pathology with Question-Mark Perception. Pathology 2002;34:282–4.[Medline]
- Bull J, Stephens D. The use of Question Mark software for formative and summative assessment in two universities. Innovations Educ Training Int 1999;36:128–36.
- Buchanan T. The efficacy of a world-wide web mediated formative assessment. J Comput Assist Learning 2000;16:193–200.
- Ramey S, Hay M, Louise M. Using electronic portfolios to measure student achievement and assess curricular integrity. Nurse Educ 2003;28:31–6.[Medline]
- Maki W, Maki R. Mastery quizzes on the web: results from a web-based introductory psychology course. Behav Res Meth Instrum Comput 2001;33:212–6.
- Taras M. Using assessment for learning and learning from assessment. Assessment Eval Higher Educ 2002;27:501–10.
- Wiliam D. Integrating summative and formative functions of assessment. Keynote address to the European Association for Educational Assessment, Prague, Czech Republic, November 2000.
This article has been cited by other articles:
|
|
|
|
 
W. Hendricson, E. Eisenberg, G. Guest, P. Jones, L. Johnson, F. Panagakos, J. McDonald, and L. Cintron
What Do Dental Students Think About Mandatory Laptop Programs?
J Dent Educ.,
May 1, 2006;
70(5):
480 - 499.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Methods
