| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Educational Methodologies |
Key words: problem-based learning, dental, performance, graduate, postdoctoral, resident, independent learning
Submitted for publication 10/16/06; accepted 04/20/07
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionReferences |
|---|
Numerous studies have compared the effects of PBL and a traditional curriculum on the satisfaction in learning environment, perception, and performance of both faculties and students.3 The data revealed both pros and cons of PBL methods. However, the conclusion drawn from these studies indicated positive effects4–9 or, otherwise, neutral effects of the PBL method.10–13
PBL was first implemented in dental classes at the Harvard School of Dental Medicine (HSDM) in fall 1994. The HSDM predoctoral curriculum has transitioned from a traditional curriculum to a PBL-hybrid curriculum in both preclinical and clinical education, with an expectation that this pedagogy would enhance the students’ critical and interdisciplinary thinking and communication skills as well as their general professional competencies.4,14
Previously published studies have reported several aspects of the effects of PBL methodology on the competencies of HSDM students. These studies demonstrated that PBL has been effective at increasing National Board Dental Examination (NBDE) Part I scores, graduation rates, and percentage of graduates entering postgraduate education programs, as well as decreasing attrition rates.4 However, the implementation of PBL at HSDM has not universally increased student productivity in research.11
Several studies in medical education have shown the positive effect of PBL on the performance of graduates during their practice or residency training. A randomized controlled trial study conducted at Harvard Medical School showed that residents who graduated from the PBL-integrated New Pathway program rated their preparation to practice humanistic medicine higher than did traditional students and expressed more confidence in their ability to manage patients with psychosocial problems. A study by Antepohl et al. showed that PBL graduates tended to pursue a higher education and they were satisfied with their undergraduate education and preparation for communication with patients, collaboration with other health professionals, and development of critical thinking as well as scientific attitudes.3 Results from several studies carried out in other institutes also indicated superior performance of the graduates from PBL medical programs in professional practice or postdoctoral training especially in the areas of self-directed, lifelong learning, communication with patients,15 cooperation skill, problem-solving skill, and ability to work independently.16 Despite the differences between characteristics of PBL and non-PBL graduates, a recent study showed that there were no differences with respect to the main occupation (clinician or others), practice location (urban or rural), or employment sector (public or private) after graduation between medical student graduates from traditional and nontraditional curricula.17
It has been accepted that resident evaluation is more reliable if the assessments are based on impressions derived from multiple sources of information, for example, direct observation interactions as well as from more than one evaluator. Supervisors’ evaluation of residents’ performance has been included in several medical education studies15,18,19 and has been shown to be valuable in resident performance evaluation.19,20 However, the investigators were not able to identify any articles in the dental literature that reported the use of multiple evaluators in a PBL outcome study.
The goal of this study was to extend the information on the outcomes of PBL education by examining the performance of HSDM graduates during their postdoctoral training. The study was designed to test the hypothesis that HSDM graduates perform well during their postdoctoral training, especially in the areas considered to be enhanced by PBL curricula including critical thinking, teamwork, communication skills, and self-directed learning. To test this hypothesis, two types of data were collected and analyzed: 1) comparison of competency self-assessments completed by HSDM graduates and graduates of other schools, and 2) comparison of self-assessments completed by HSDM graduates and graduate program directors’ ratings of the competency of residents from HSDM. In addition, graduates’ future plans were collected to determine whether HSDM and non-HSDM graduates had different preferences for their future careers.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
A questionnaire was created and approved by the Harvard Medical School Office for Research Subject Protection. The questionnaire was a modification of a questionnaire created for a previous study by Santos-Gomez et al.15 and also was based on a questionnaire developed for the ACGME Outcomes Project by the Accreditation Council for Graduate Medical Education (ACGME). The questionnaire asked HSDM graduates to rate themselves in comparison with other residents (co-residents) in their advanced graduate programs who had graduated from non-PBL, lecture-based dental schools other than HSDM on twelve selected competencies on a four-point scale, which consisted of excellent, good, fair, and poor. These twelve competencies were organized into three categories. The first category includes competencies expected to be enhanced by PBL including communication with patients and staff, critical thinking, independent learning, performance in small group settings, patient education, self-assessment, and teamwork. The second category includes competencies known to be equal between PBL and non-HSDM students including general dental knowledge and knowledge that is specific for a specialty area; and the final category includes competencies that are speculated to be strengths of a traditional curriculum including preclinical and clinical skills. The competencies were placed in a random order in the questionnaire to control bias.
The HSDM graduates were asked to randomly give the questionnaire to one co-resident in the program who had been trained in a non-HSDM dental school. Each co-resident was asked to assess himself or herself on the twelve competencies. The director of each program was also asked to assess HSDM graduate(s) in the program on the selected competencies. The questionnaire included an overall evaluation in which the directors were asked to rate the HSDM graduates comprehensively, considering all aspects of performance, using a scale that consisted of superior, satisfactory, marginal, and unsatisfactory.
The questionnaire completed by the HSDM and non-HSDM graduates also included a question on the graduates’ future career plans with the response options being full-time clinical instructor, full-time researcher, part-time clinical instructor, part-time researcher, and private practitioner. The graduates were allowed to select more than one option that is in their plan. To prevent misinterpretation, definitions of the competencies were enclosed with the questionnaire. Follow-up letters were sent to the HSDM graduates who did not respond one month after the first letters were sent.
Differences between the responses provided by HSDM and non-HSDM residents were analyzed with the Mann-Whitney test with significance set at p<0.05. The responses by HSDM-trained residents and their directors were paired and analyzed with the Wilcoxon signed ranks test to determine the degree of correspondence and to investigate the direction of any differences that were identified from analysis of the responses.
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
). Figure 1 shows the distribution of the respondents by the type of residency program. More than 45 percent of the respondents were in orthodontic advanced graduate programs, and about 24 percent of the respondents were in periodontal programs. The others were in prosthodontics, end-odontics, pediatric dentistry, oral surgery, or General Practice Residency/Advanced Education in General Dentistry (GPR/AEGD) programs. However, since there was only one AEGD resident who graduated from HSDM responding to the questionnaire and there was no matched non-HSDM resident, those data were excluded. Results of the graduates’ self-assessments are shown in Table 2, which displays the percentages of the total number of responses in the four rating categories: excellent, good, fair, poor. The "non-applicable" responses were not included. The competencies expected to be enhanced by PBL are marked with asterisks in Table 2 and are highlighted in subsequent tables.
|
|
|
HSDM graduates rated themselves the highest for the competency "independent learning"; 80.95 percent self-assessed their skill level as "excellent" for this item. In contrast, 29.73 percent of the non-HSDM believed they were excellent in independent learning. The highest rating for the non-HSDM graduates was "ability in communication with staff "; 51.35 percent self-assessed their skill level as "excellent" for this item.
Table 3 compares the self-assessment ratings for all forty-one HSDM and thirty-eight non-HSDM residents combined without regard to type of postgraduate program. The statistical analysis displayed in Table 3 showed that there was no significant difference found for general dental knowledge, knowledge specific for specialty, preclinical skills, clinical skills, communication with staff, and patient education. However, there was a significant difference (p<0.05) comparing HSDM and non-HSDM ratings in communication with patients, critical thinking, independent learning, performance in small group settings, self-assessment, and teamwork.
|
shows responses from the graduate program directors who were asked to evaluate HSDM graduates’ performance in their program. The directors’ highest rating was in the category of independent learning, where 65.31 percent of the HSDM graduates were evaluated to be excellent. The statistical analysis shown in Table 5 shows no significant difference (p>0.05) between the directors’ evaluations and HSDM graduates’ self-assessments in twenty-seven instances where responses were available from both director and resident. However, there were seven competencies where the directors rated HSDM graduates less favorably than the HSDM graduates’ self-assessment, and there were five competencies where the directors rated HSDM graduates more favorably than the HSDM graduates’ self-assessment. However, none of these differences between residency director and HSDM graduates’ ratings were significantly different.
|
|
Residents’ future career plans are shown in Figure 2. Most of the HSDM and non-HSDM residents planned to pursue their careers either as full-time private practitioners (57.58 percent of all non-HSDM and 26.83 percent of all HSDM respondents) or as part-time private practitioners who also worked as part-time clinical instructors in dental schools (33.33 percent of all non-HSDM and 43.90 percent of all HSDM respondents). Interestingly, 70.73 percent of the HSDM graduates expected to be involved in the academic field, either full-time or part-time, whereas 45.75 percent of non-HSDM respondents planned to work in academia.
|
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
This study focused on the competencies relevant to professional practice, which could be categorized into three groups: the competencies expected to be enhanced by PBL, the competencies known to be equal between PBL and non-HSDM students, and the competencies speculated to be stronger in traditional curricula. The results from self-assessment showed that HSDM graduates rated themselves more highly than non-HSDM graduates on all competencies. This implies a higher confidence in their abilities. Statistical analysis showed significant differences (p<0.05) between the HSDM and non-HSDM graduate self-ratings for almost all of the competencies expected to be enhanced by the PBL method including ability in communication with patients, critical thinking, independent learning, self-assessment, team-work, and performance in small group settings. On all of these competencies, HSDM graduates rated themselves one to two scores higher than non-HSDM graduates. In particular, more than 80 percent of HSDM graduates considered themselves to be excellent in conducting independent learning. These results correspond with previous self-assessment studies, which also showed that the PBL graduates gave themselves higher ratings for these areas, especially in communication and independent learning skills as well as the ability to collaborate with others.15,21,22 However, a significant difference was not found in communications with staff and patient education, which are expected to be one of the strengths of a PBL curriculum.
The results also showed that HSDM and non-HSDM graduates believed that they performed equally well in general dental knowledge and knowledge specific for their chosen dental specialty. It has been demonstrated in previous studies that the performance of PBL students in these areas is typically equivalent but not superior in comparison to traditionally educated students.21,22 These results, which are generally consistent with previous research, indicate that the HSDM and non-HSDM students probably did not overestimate or underestimate their level of competency when completing the self-assessment surveys. More importantly, no significant differences were found between HSDM and traditional graduates for preclinical skills or clinical skills. There has been speculation that HSDM graduates’ level of competency in procedural skills would be adversely affected by the reduction of laboratory time resulting from implementation of the PBL program at Harvard. The results presented here confirm findings from another as yet unpublished study conducted at HSDM that showed the reduction of laboratory preclinical exercises at HSDM did not affect students’ didactic and clinical performance.
The HSDM graduates’ self-assessments and the evaluations completed by the program directors were generally similar. Program directors also rated HSDM graduates to be excellent in independent learning. However, the directors gave less favorable ratings compared to the graduates’ self-assessments in a few areas. These results are not consistent with several studies that have shown that directors tend to be more lenient evaluators than residents,23 and poor correlations between director and resident self-evaluations have been reported.15,18 One explanation that has been proposed is that supervisors’ ratings of residents are usually based on an overall impression instead of actual performance, while self-assessment tends to examine performance in discrete areas.15 However, most of the dental postdoctoral programs have considerably smaller size than medical residency programs. This could result in more stringent evaluations or more similar responses between self- and directors’ evaluations, which were found in our study.
In regard to future career plans, there was no statistically significant difference between HSDM and non-HSDM residents, which is consistent with previous findings.2,17 However, the data suggested that HSDM graduates appear to be more likely to get involved in academia as part-time clinical instructors.
The findings in our study suggest that some characteristics of the PBL curriculum appear to remain with graduates in their postdoctoral training. However, there may also be some confounding variables. Performance of the graduates might reflect other factors unrelated to the PBL method. For example, the selection procedures of Harvard University that target candidates who have specific characteristics and attributes may preferentially sort students by PBL characteristics prior to exposure to the PBL environment in HSDM. It is also possible that the prestige of Harvard University creates high confidence in its graduates, resulting in higher self-assessment ratings by HSDM graduates. In addition, the majority of the respondents were in orthodontic and periodontal programs. The findings of the study might have represented unique characteristics of the residents in these programs rather than characteristics of the PBL graduates. However, significant differences for certain competencies were found between HSDM and non-HSDM graduates within the same program including orthodontic and periodontal programs.
It is also possible that the positive responses by the HSDM subjects in this study reflect their gratitude to their school or their faculties. Similarly, the directors’ responses could be influenced by their relationship with HSDM or by their global impression of Harvard as a high-quality academic institution, which may have produced "halo effect" ratings for residents known to be HSDM graduates.
Despite the positive results, interpretation of the data in the present study was limited by the small sample size as well as the low response rate and the inherent disadvantages of using a questionnaire with a rating scale because respondents may avoid using the extreme ends of a rating scale. It is also possible that responses were subjective or biased. An evaluation of the effectiveness of a PBL curriculum was challenging because it is difficult to recruit subjects from a parallel traditional education track that could serve as better controls. More importantly, the results from self-assessment reflect the subjects’ perceptions of their own performance, which may or may not demonstrate their true performance.
Even though the present study illustrates the experience of a select group of dental residents who graduated from only one PBL program—the Harvard School of Dental Medicine—these findings extend the results of earlier studies and hopefully will be informative for dental faculty who desire to implement a PBL curriculum in other schools.
|
Footnotes |
|---|
|
REFERENCES |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  B. S. S. Cheng Problem-Based Learning and the Workplace: Do Dental Hygienists in Hong Kong Continue to Use the Skills Acquired in Their Studies? J Dent Educ., August 1, 2009; 73(8): 991 - 1000. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. A. Kramer, J. E.N. Albino, S. C. Andrieu, W. D. Hendricson, L. Henson, B. D. Horn, L. M. Neumann, and S. K. Young Dental Student Assessment Toolbox J Dent Educ., January 1, 2009; 73(1): 12 - 35. [Full Text] [PDF]
|
|
|
|
|
|
J. E.N. Albino, S. K. Young, L. M. Neumann, G. A. Kramer, S. C. Andrieu, L. Henson, B. Horn, and W. D. Hendricson Assessing Dental Students' Competence: Best Practice Recommendations in the Performance Assessment Literature and Investigation of Current Practices in Predoctoral Dental Education J Dent Educ., December 1, 2008; 72(12): 1405 - 1435. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
