J Dent Educ. 72(5): 585-592 2008
© 2008 American Dental Education Association
Critical Issues in Dental Education |
Relationship Between Dental Students’ Performance on Standardized Multiple-Choice Examinations and OSCEs
Padraig C. Dennehy, D.M.D.;
Srinivas M. Susarla, D.M.D., M.P.H.;
Nadeem Y. Karimbux, D.M.D., M.M.Sc.
Key words: OSCE, MCQ, National Board Dental Examination, assessment, competency
Submitted for publication 06/10/07;
accepted 02/11/08
 |
Abstract
|
|---|
The purpose of this study was to examine the relationship among dental students’ performance on the National Board Dental Examination (NBDE Parts I and II), comprehensive written multiple-choice question examinations (MCQ examinations), and objective structured clinical examinations (OSCEs) administered at the Harvard School of Dental Medicine (HSDM). Outcomes were measured during the third and fourth years at HSDM for the classes of 2006 and 2007. Three separate OSCE exams and two comprehensive MCQ examinations are administered during years 3 and 4 at HSDM per class. The study group was made up of sixty-two students (thirty-two females and thirty males). The average NBDE Parts I and II scores of the study population were 93.32 ±4.02 and 84.63 ±4.25, respectively, and were associated with outcomes on all three OSCE examinations (p
0.044). However, using multiple regression models, the only statistically significant association occurred between NBDE Part II and OSCE 2 (p=0.003). Analysis showed that didactic predictors (NBDE Parts I and II and comprehensive MCQ examinations) explained 20.4 to 22.1 percent of the variability in OSCE scores. These results suggest that performance on OSCE examinations is not highly correlated with performance on NBDE Parts I and II and HSDM-administered MCQ examinations. The findings suggest that OSCE examinations are more likely to measure other qualities such as problem-solving ability, critical thinking, and communication skills.
The objective structured clinical examination (OSCE) is a clinically based examination that is administered in many disciplines including medicine, dentistry, and nursing. Harden first developed the OSCE in 1975 in a Scottish medical school curriculum;1 since then, the examination has grown in popularity among professional institutions.
Dentists and dental students need practical and interpersonal skills, as well as a sound knowledge base and technical skills in clinical dentistry. Traditional methods of evaluation (written assessments including essay and multiple-choice question [MCQ] examinations) have been criticized for focusing simply on students’ knowledge base and their abilities to memorize, while ignoring other important characteristics and cognitive capacities of a clinician. The results on written examinations such as MCQ-based tests do not typically determine or predict clinical success.2–6 Although dental school faculty naturally desire to assess students’ ability to actually apply their fund of biomedical knowledge during the process of patient care and to measure students’ competence in problem-solving, critical thinking, and communication skills, many assessment tools that have been developed to assess these more sophisticated components of professional competence tend to be subjective in nature, and many have not been validated.1,7–10
The OSCE can be used to assess comprehension of factual knowledge, but in health professions education, the OSCE is primarily intended to measure students’ acquisition of skills such as patient communication, patient education, coherent presentation of history and physical examination findings in a defined time period, patient interviewing technique, specific physical examination skills (e.g., testing visual acuity or cranial nerves), or measuring vital signs. The Harvard Medical School (HMS) and Harvard School of Dental Medicine (HSDM) introduced the OSCE as a method of assessment in a problem-based learning (PBL) curriculum. The OSCEs at HSDM were designed to assess some of the skills that are supposedly gained during a PBL curriculum (skills such as critical thinking, problem-solving, and hypothesis formation).
OSCEs can be used to assess some of the skill sets that are expected of a health care professional. OSCE formats can vary depending on their setting and the context of the curriculum. In most OSCEs, students are observed directly by faculty, videotaped for later review by faculty, or assessed by a trained "simulated" patient as they interview and examine a patient and provide patient education. In some OSCEs, students may then travel to a "report" station where they are orally quizzed on their findings from a patient examination or asked to develop a differential diagnosis. In other formats, students may be asked to read a case simulation, review radiographic images, or look at a dental laboratory procedure and reach an assessment at discipline-specific stations.11–17 The latter is the style of OSCE that was developed at the time of PBL implementation at HSDM.
At HSDM, during the OSCE, the students travel from station to station where they are presented with clinical scenarios. Faculty members evaluate the students (on a three-point scale) on their abilities to answer each question. In addition, the students are assessed (on a three-point scale) on their abilities to present their answers in a clear and logical manner and on their problem-solving abilities. The student is expected to hypothesize, think critically, and communicate clearly and logically about the clinical scenario being presented to them. Studies have shown that the OSCE format is best suited to assess diagnostic, interpretation, and treatment planning scenarios, but it has limitations in assessing students’ capacity to perform clinical procedures in large part because it is not feasible to ask students to perform invasive and nonreversible procedures on patients, real or simulated, during an OSCE and also because students have a limited amount of time at OSCE stations to perform tasks—usually less than would be available in a real clinical setting.18–22
The OSCE is set up to provide an objective evaluation of a student’s performance by giving every student the same questions from calibrated evaluators who rate student responses according to a standardized grading system, thus minimizing interexaminer error. Reliability, as defined by Davenport et al.,11 is the accuracy with which candidates are ranked and pass/fail decisions are made. Validity differs from reliability in that it requires that student competence is appropriately assessed by a method of evaluation that actually measures what it is intended to measure and also encourages good learning strategies to be used.1,7 Studies by Roberts and Norman in 1990 and Brown et al. in 1987 show that there can be a high degree of interexaminer reliability in an OSCE situation.1,7 Further studies have shown that the OSCE can be a reasonably reliable, valid, and objective method of assessment, but its main drawback is that it can be resource-intensive.8 Resources include a large body of evaluators, all of whom must be calibrated in advance; an appropriate testing location with space for multiple stations; and time for preparation for the case including duplication of models, radiographs, and charting.
Dental students in all U.S. dental schools are required to take the National Board Dental Examination (NBDE). The NBDE has two portions (Part I and Part II), both of which are computerized tests scheduled by the student at independent testing centers. The NBDE Part I is a one-day examination comprised of 400 questions, typically taken by dental students near the end of their second year or between the second and third years of dental school, although there is increasing variation in the timing of the test from school to school and some schools now have their students take the NBDE Part I at the conclusion of the first year. The purpose of the NBDE Part I is to assess students’ comprehension of the basic sciences in the disciplines of anatomy and physiology, biochemistry, microbiology, and dental anatomy. Until 2006, the NBDE Part I employed a standard one best response multiple-choice question format or multiple true-false format for virtually all of its 400 questions. Starting in 2007, the NBDE Part I now includes eighty questions (20 percent of all questions) based on case scenarios. The NBDE Part II is usually taken during the fourth year of dental school and tests primarily clinical knowledge from all of the dental specialties. The NBDE Part II differs from Part I in that it is a two-day exam, the second day consisting of case-based questions (Figure 1
).
View larger version (15K):
[in this window]
[in a new window]
|
Figure 1. Timeline of examinations at Harvard School of Dental Medicine
A shows timing of three OSCEs and two comprehensive multiple-choice examinations (Comp I and Comp II); B shows timing of NBDE Parts I and II.
|
|
Previous performance on essay/MCQ examinations has been shown to be a predictor of performance for future academic and clinical success in some studies. However, there are also several studies that show that performance on these kinds of examinations is a poor predictor of future academic and clinical performance.2–5 For example, Webb found that performance on the Dental Admission Test (DAT), an examination required for entrance into dental school, was not correlated with a student’s grade point average (GPA) in the final year of dental school. Specifically, students with high DAT scores covered the entire spectrum of GPAs from high to low.4 Park et al. examined whether a correlation existed between didactic measures, such as DAT scores, GPAs, and NBDE I scores, and clinical productivity and proficiency. Park et al. found a lack of any statistically significant associations between the predictors and the outcomes for productivity and proficiency.3 Hangorsky reported discrepancies between class rank of final-year dental students and their examination performance on dental licensing certification examinations and found there was no correlation between the two measures.5
The aim of this study was to explore the relationship between students’ performance on the NBDE Part I, NBDE Part II, and comprehensive MCQ examinations administered at HSDM and their performance on OSCEs administered during the curriculum. The study was implemented to test two hypotheses: 1) there is no association between students’ performance on standardized written examinations such as tests in the MCQ format and their performance on the OSCEs administered at HSDM; and 2) students’ performance on NBDE Part II, which has a substantial case-based component, is associated with their OSCE performance.
|
Materials and Methods
|
|---|
This was a retrospective cohort study, evaluating data from the Harvard School of Dental Medicine Classes of 2006 and 2007. Predictor variables were students’ scores on National Board Dental Examination Part I and Part II and HSDM comprehensive multiple-choice examinations that are intended to be standardized assessments of students’ comprehension of the basic science foundations for the practice of dentistry, the behavioral sciences, and knowledge of clinical principles pertinent to the various components of general dental practice. Outcome variables were students’ performance scores on the OSCE examinations. Institutional Review Board approval was sought and granted by Harvard Medical School. Data on student performance were obtained from the office of the registrar. To protect student confidentiality, all subjects’ grades were assigned random identification numbers prior to release to the investigators, thereby disconnecting student identity from grades. No identifying data such as names or social security numbers were provided.
In the HSDM curriculum, junior (year 3) and senior (year 4) students participate in a total of three OSCE examinations. These examinations are administered in February/March of year 3 (OSCE 1), October of year 4 (OSCE 2), and April of year 4 (OSCE 3) (see Figure 1
). On the same day of OSCE 2 and OSCE 3, which are administered in the morning, comprehensive MCQ examinations are also administered in the afternoon. Questions on the comprehensive MCQ examination are not based on the OSCE exam administered earlier. The content of the first comprehensive MCQ examination is similar to that of the NBDE Part II, and the second comprehensive multiple-choice examination is similar to the written portion of the North East Regional Board (NERB) licensure examination. The MCQ examinations are developed by faculty members in each specialty area and are based on MCQs from the NBDE Part II and NERB examinations. In general, the MCQ examinations are used as preparation evaluations to assess students’ "readiness" for the NBDE Part II and the NERB exam.
The OSCE at HSDM was first used when the school introduced a problem-based learning (PBL) curriculum in 1994. At HSDM, the OSCE is structured as an examination consisting of ten stations that test different disciplines. These disciplines are periodontics, oral pathology and radiology, treatment planning, prosthodontics, endodontics, operative dentistry, orthodontics, pediatric dentistry, oral surgery, and oral health and epidemiology. The format is a series of timed stations, each six minutes in duration. A case history, photographs, hard and soft tissue charting, radiographs, and study models are provided to the students prior to the examination, and students are given one and a half hours to review and analyze the case prior to being asked questions in the OSCE format. At HSDM, the OSCE stations are faculty-based or written stations. At a faculty-based station, a faculty member asks students predetermined questions about the clinical case that the student previously reviewed. The faculty rate the students’ answers on a three-point scale (1=student answered correctly, 2=student answered with prompting, 3=student could not answer or did not know the answer). Faculty are also asked to evaluate students on their ability to problem-solve and their ability to explain concepts and articulate decisions in a clear and logical fashion (these two items are evaluated on a five-point scale). The OSCEs and MCQ examinations are scored on a basis of 100 points. HSDM employs a pass/fail grading system, and students’ scores are scaled according to the frequency distribution represented a normal curve. The class average occupies the center of the distribution. A "75" is set as the pass grade and can be adjusted according to the curve. Students who fail these examinations are offered remediation and a make-up examination.
Data were iteratively entered into a commercially available statistical software package (SPSS v.11.5, SPSS Inc., Chicago, IL) over the course of the study. Descriptive statistics (mean, SD, range) were computed for predictors and outcomes. Bivariate associations (ANOVA, Pearson correlations) were computed to identify predictors statistically significantly or near statistically significantly associated with OSCE performance (p<0.15). Associations with p0.15 were included in a multiple linear regression model, which was used to evaluate the simultaneous effects of multiple covariates. For the multiple models, only those covariates that occurred prior to the outcome of interest were included in the models (e.g., NBDE Part II is taken after OSCE 1 and, as such, was not used as a predictor of OSCE 1 performance). In the multiple regression models, p-values 0.05 were considered statistically significant.
|
Results
|
|---|
Data were available for sixty-two students from the Classes of 2006 (n=32) and 2007 (n=30). All students in each class participated in the study. The study sample consisted of thirty-two females (51.6 percent) and thirty males (48.4 percent) (Table 1). The average NBDE Part I and Part II scores were 93.32 ±4.02 and 84.63 ±4.25, respectively. Scores on the comprehensive written multiple-choice examinations 1 and 2 were 64.20 ±6.64 and 69.89 ±4.73, respectively. Average OSCE scores were as follows: OSCE 1 (72.34 ±8.89), OSCE 2 (81.74 ±6.25), and OSCE 3 (78.06 ±6.90) (Table 1). Scores on both NBDE parts were associated with outcomes on all OSCEs (p0.044) when analyzed independently of each other (Table 2). Bivariate association analysis also showed that performance on a previous OSCE was associated with performance on a future OSCE (p0.131). The only written comprehensive examination that was statistically associated with OSCE performance was examination 2 with OSCE 3 (Table 2).
However, when placed into a multiple regression model, virtually none of the didactic predictors were statistically significantly associated with OSCE performance (Tables 3, 4, and 5). The only didactic measure that was statistically significantly associated with an OSCE was NBDE Part II when compared to OSCE 2 (t=3.082, p=0.003) (Table 4).
In the multiple regression models, didactic predictors (NBDE examinations, HSDM multiple-choice examinations) explained between 24.4 and 32.0 percent (unadjusted R-square; adjusted R-squares ranged from 20.4 to 22.1 percent) of the variability in OSCE scores as shown by the R-square analysis for each OSCE (Tables 3, 4, and 5).
|
Discussion
|
|---|
The purpose of this study was to examine the relationship between students’ performance on the traditional written measures of dental school performance (NBDE Part I, NBDE Part II, and comprehensive multiple-choice examinations 1 and 2 at HSDM) and performance on OSCE examinations administered at HSDM. These results supported the hypothesis that there is no clear association between performance on these standardized written examinations and performance on the OSCEs administered at HSDM. The only statistically significant association in the data set was between NBDE Part II and OSCE 2. This correlated with our second hypothesis, that NBDE Part II would have a stronger association with OSCE performance, in comparison to the other predictor variables. The NBDE Part II contains a case-based section on day two of the exam that is very similar to the OSCE format used at HSDM. For both examinations, students are required to use critical thinking skills in combination with didactic knowledge to correctly interpret, analyze, and answer the questions.
Multiple studies have been done in the past to try to correlate performance on the OSCE, which is thought to be a more real-life (authentic) simulation due to its case-based nature and need for critical thinking skills, with other measures of dental student evaluation. Previous research and results from this study exemplify the difficulty of predicting future performance based upon performance in the past. Most DAT scores cannot be used to judge dental school performance,3,4,6 class rank cannot predict performance on the licensure examination,5 and didactic measures of performance (NBDE Parts I and II and HSDM comprehensive multiple-choice examinations) do not act as predictors of OSCE performance. It is interesting that, in our study, NBDE Part II was only associated with performance on OSCE 2 (and not OSCEs 1 and 3). An explanation for this may be due to the temporal proximity of these two examinations. Less than two months separate the NBDE Part II, which is taken in December of year 4, and OSCE 2, which is taken in October of year 4 (Figure 1). Given the proximity, knowledge gained during study for one would be more likely to be retained for the second.
Some previous studies have shown positive correlations among traditional measures of overall academic performance, GPA and class rank, and OSCE results. For example, Gerrow et al. found positive correlations (r=0.46, p<0.001) between the OSCE portion of the Canadian Dental Certification Examination and students’ final-year GPA results.9 In addition, Gerrow et al. found that, over a six-year study, no students in the top half of their class failed the Canadian Dental Certification Examination OSCE. Gerrow et al.’s research also showed positive correlation between performance on the written portion of the Canadian Dental Certification Examination and the OSCE portion.9
Other literature and the results from this study show poor correlations between performance on written examinations designed to assess students’ fund of knowledge and performance on OSCEs.23,24 One would expect this because most standardized written examinations, especially those in the one-best-response MCQ format, primarily measure the test-taker’s capacity to remember or recognize information, while exams like OSCEs in the format employed at HSDM are designed to test critical thinking and problem-solving abilities and capacity to communicate comprehension of concepts and explain the rationale for decisions.13–15,18,19 Another explanation for the discrepancies may be due to the different nature of the OSCE examinations administered. Although the general format for an OSCE is the same, differences may occur from types of questions administered, calibration of the evaluators, discipline in which the OSCE is used for evaluation, and the amount the examination focuses on clinical skills versus critical thinking skills.
The results from this investigation demonstrated that, within the limits of the analysis, didactic measures of performance studied (NBDE Parts I and II and HSDM comprehensive multiple-choice examinations) cannot be used to predict performance on OSCE examinations, except in the case of NBDE Part II and OSCE 2. The results also show that less than 30 percent of the variability in OSCE performance is explained by the aforementioned measures of didactic performance. This is similar to the results found by Gerrow et al., which showed that the written portion of the Canadian Dental Certification Examination explained approximately 30 percent of the variance in the OSCE portion of the same examination.9 This indicates that the OSCE may be a tool that allows educators to assess student capabilities that are not evaluated in typical standardized written examinations. These capabilities (critical thinking, communication skills, and the ability to problem-solve) are core attributes of professional behavior and consequently should be taught, practiced, and assessed in a dental curriculum.
Although standardized test scores such as the national dental boards can predict the performance of dental students on evaluations of knowledge recall and recognition that employ a similar testing format, a more sophisticated assessment strategy needs to be developed to predict the qualities of future dental practitioners in the areas of critical thinking, ability to form a hypothesis, capacity to collect appropriate information and analyze it objectively, ability to solve problems that arise during patient care, and capacity to explain the reasoning for decisions. In the end, the experience of faculty along with many different quantitative measures of assessment must be combined to form the best prediction of what students are actually learning. The OSCE can be used as an additional assessment tool to determine the readiness of dental students to function as health care professionals.
|
Author Information
|
|---|
Dr. Dennehy is currently a Resident in Orthodontics, University of Minnesota School of Dentistry and was a D.M.D. Candidate at Harvard School of Dental Medicine at the time of this study; Dr. Susarla is currently a Resident in Oral and Maxillofacial Surgery, Massachusetts General Hospital/Harvard School of Dental Medicine and was a D.M.D. candidate at Harvard School of Dental Medicine at the time of this study; Dr. Karimbux is Assistant Dean for Dental Education and Associate Professor of Periodontology, Harvard School of Dental Medicine. Direct correspondence and requests for reprints to Dr. Nadeem Y. Karimbux, Harvard School of Dental Medicine, Office of Dental Education, 188 Longwood Avenue, Boston, MA 02115; 617-432-1447; nadeem_karimbux{at}hsdm.harvard.edu.
|
REFERENCES
|
|---|
- Roberts J, Norman G. Reliability and learning from the objective structured clinical examination. Med Educ 1990; 24:219–23.[Medline]
- Taylor ML, Blue AV, Mainous AG, Geesey ME, Basco WT. The relationship between the National Board of Medical Examiners’ prototype of the step 2 clinical skills exam and interns’ performance. Acad Med 2005; 80(5):496–501.[Medline]
- Park SE, Susarla SM, Massey W. Do admissions data and NBDE Part I scores predict clinical performance among dental students? J Dent Educ 2006; 70(5):518–24.[Abstract/Free Full Text]
- Webb SC. Evaluating tests as predictors of dental school grades. J Dent Educ 1958; 22:33–45.[Medline]
- Hangorsky U. Clinical competency levels of fourth-year dental students as determined by board examiners and faculty members. J Am Dent Assoc 1981; 102:35–7.[Abstract]
- Bergman AV, Susarla SM, Howell TH, Karimbux NY. Dental admissions test scores and performance on NBDE Part I, revisited. J Dent Educ 2006; 70(3):258–62.[Abstract/Free Full Text]
- Brown B, Roberts J, Rankin J, Stevens B, Thompkins C, Patton D. The objective structured clinical examination: reliability and validity. Proceedings of the 2nd Annual International Conference on Assessing Clinical Competence, Ottawa. Montreal: Heal Publications, 1987:563–71.
- Barman A. Critiques on the objective structured clinical examination. Ann Acad Med Singapore 2005; 34(8): 478–82.[Medline]
- Gerrow JD, Murphy HJ, Boyd MA, Scott DA. Concurrent validity of written and OSCE components of the Canadian dental certification examinations. J Dent Educ 2003; 67(8):896–901.[Abstract]
- Roberts J, Norman G. Reliability and learning from the objective structured clinical examination. Med Educ 1990; 24:219–23.[Medline]
- Davenport ES, Davis JEC, Cushing AM, Holsgrove GJ. An innovation in the assessment of future dentists. Br Dent J 1998; 184:1923.
- Barzansky B, Etzel SI. Educational programs in US medical schools, 2002–2003. JAMA 2003; 290:1190–6.[Abstract/Free Full Text]
- Hendricson WD, Andrieu SC, Chadwick DG, Chmar JE, Cole JR, George MC, et al. Educational strategies associated with development of problem-solving, critical thinking, and self-directed learning. J Dent Educ 2006; 70(9):925–36.[Abstract/Free Full Text]
- Patel VL, Arocha JF, Branch T, Karlin DR. Relation -ship between small group problem-solving activity and lectures in health science curricula. J Dent Educ 2004; 68(10):1058–80.[Abstract/Free Full Text]
- Fincham AG, Shuler CF. The changing face of dental education: the impact of PBL. J Dent Educ 2001; 65(5): 406–21.[Abstract]
- Haden NK, Andrieu SC, Chadwick DG, Chmar JE, Cole JR, George MC, et al. The dental education environment. J Dent Educ 2006; 70(12):1265–70.[Abstract/Free Full Text]
- Ranney RR, Wilson MB, Bennett RB. Evaluation of applicants to predoctoral dental education programs: review of the literature. J Dent Educ 2005; 69(10):1095–106.[Abstract/Free Full Text]
- Harden RM, Stevenson M, Wilson-Downie W, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J 1975; 1:447–51.[Abstract/Free Full Text]
- Prislin MD, Fitzpatrick CF, Lie D, Giglio M, Radecki S, Lewis E. Use of an objective structured clinical examination in evaluating student performance. Fam Med 1998; 30:338–44.[Medline]
- Campos-Outcalt D, Watkins A, Fulginiti J, Kutob R, Gordon P. Correlations of family medicine clerkship evaluations and objective structured clinical examination scores and residency director ratings. Fam Med 1999; 31:90–4.[Medline]
- Regehr G, Freeman R, Hodges B, Russell L. Assessing the generalizability of OSCE measures across content domains. Acad Med 1999; 74(12):1320–2.[Medline]
- Muller ES, Harik P, Margolis M, Clauser B, Mckinley D, Boulet JR. An examination of the relationship between clinical skills examination performance and performance on USMLE Step 2. Acad Med 2003; 78(10 Suppl): S27–9.[Medline]
- Simon SR, Bui A, Day S, Berti D, Volkan K. The relationship between second-year medical students’ OSCE scores and USMLE Step 2 scores. J Eval Clin Pract 2007; 13(6):901–5.[Medline]
- De Ball S, Sullivan K, Horine J, Duncan WK, Replogle W. The relationship of performance on the dental admission test and performance on Part I of the National Board Dental Examination. J Dent Educ 2002; 66(4):478–84.[Abstract]
This article has been cited by other articles:
|
|
|
|
 
J. E.N. Albino, S. K. Young, L. M. Neumann, G. A. Kramer, S. C. Andrieu, L. Henson, B. Horn, and W. D. Hendricson
Assessing Dental Students' Competence: Best Practice Recommendations in the Performance Assessment Literature and Investigation of Current Practices in Predoctoral Dental Education
J Dent Educ.,
December 1, 2008;
72(12):
1405 - 1435.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Materials and Methods
