JDE
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

J Dent Educ. 71(12): 1561-1573 2007
© 2007 American Dental Education Association
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Dalrymple, K. R.
Right arrow Articles by Shuler, C. F.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Dalrymple, K. R.
Right arrow Articles by Shuler, C. F.

Educational Methodologies

PBL Core Skills Faculty Development Workshop 3: Understanding PBL Process Assessment and Feedback via Scenario-Based Discussions, Observation, and Role-Play

Kirsten R. Dalrymple, Ph.D.; Shirley Wong, D.D.S.; Alvin Rosenblum, D.D.S.; Carol Wuenschell, Ph.D.; Michael Paine, D.D.S., Ph.D.; Charles F. Shuler, D.M.D., Ph.D.

Key words: problem-based learning, assessment, feedback, facilitator, faculty development

Submitted for publication 05/04/07; accepted 09/14/07


   Abstract
Top
 Abstract
Materials and Methods
 Results
 Discussion
 Conclusion
 References
 
Tutorial assessment in PBL is thought to be a valid assessment approach and is believed to exert a positive impact on the learning process. Reports, however, have demonstrated that assessment by the facilitator can be unreliable. Training of faculty to conduct this type of assessment has tended to be lacking and is a likely contributor to this inconsistency. This report describes the final in a series of foundation-building faculty development workshops focused on the instructional methodology of PBL. The PBL Assessment and Feedback workshop reported here introduced the theory and practice of conducting process-based assessment accompanied by formative feedback. Scenario-based discussions, mock group demonstration, role-modeling, and role-play were utilized as adult learning-appropriate strategies to familiarize participants with process-based assessment and feedback. Evaluation of the workshop by participants provided evidence that the majority of participants were satisfied with the methods and content of the workshop. Suggestions for additional training in these assessment methods included additional examples, practice, workshops, or observation and mentoring.

Assessment exerts a powerful influence over students and teachers in the educational environment. Educators and educational researchers have asserted what most students and teachers will anecdotally attest: that assessment drives the learning process.1,2 Rigorous assessment processes are intended to monitor student progress and used to demonstrate to outside stakeholders that educational programs are functioning and accountable.3 However, it is important to realize that student learning approaches can adapt to the types of assessments used, and assessments not aligned with curricular goals might actually result in unintended student outcomes.2 Thus, assessment methods should be aligned with intended curricular outcomes and with the teaching and learning methods employed to help students acquire the knowledge, skills, and attitudes that comprise the learning outcomes of the curriculum.

In problem-based learning (PBL), the small-group, tutorial environment is believed to support both the development of discipline-specific knowledge and the development of professional skills such as teamwork, clinical reasoning, and information literacy.4 Continuous, formative assessment and feedback on these skills as they are exhibited in the tutorial environment can therefore be thought of as a valid form of assessment and, by extension, a means to positively reinforce the learning process and outcomes.5 If the attributes of lifelong learning and critical thinking are intended educational outcomes, then assessment of the development of these skills is necessary to emphasize their importance in the learning environment.

Despite its positive impact on the learning process and development of professional skills, assessment in small-group PBL tutorials has been shown to vary more than is seen with objective multiple-choice type examinations.5–7 Investigators have suggested this may result from insufficient facilitator training in the methods and criteria necessary to conduct student tutorial assessment.5,7 Additionally, there may be other variables that contribute to the lack of reliability in tutorial assessment, including a lack of clarity regarding the true domains being assessed (i.e., skills related to the process of learning/self-directed learning versus acquisition of specific biomedical content), inadequate observation of relevant student performance, and a lack of support from teachers for the method itself or for the way the assessment is implemented.5,8–12

Faculty development activities in PBL assessment and feedback have been proposed as one means to help improve understanding of the tutorial assessment and feedback process.5 The underlying assumption is that providing training for faculty will enhance the reliability of the assessment process. Despite the fact that assessment is often a key role of the facilitator, helping faculty acquire the skills needed to assess student performance in tutorial groups has not often been a focus of faculty development programs.13,14 The University of Southern California School of Dentistry (USCSD) fully implemented PBL for its D.D.S. program in 2001. A comprehensive faculty development program with a series of linked workshops in core PBL facilitation skills has been developed to support curriculum implementation. The PBL Assessment and Feedback workshop serves as the third in the series of PBL Core Skills workshops that familiarize faculty members with the PBL process itself and with the roles, responsibilities, and skills of the facilitator. Previous reports in the Journal of Dental Education have described other components of this workshop series.15–17

The purpose of this report is to describe the structure, content, and methods utilized in the PBL Assessment and Feedback workshop and the post-workshop perceptions of the faculty who participated. The importance of the assessment process, the learning preferences of adults, and sociocultural variables in the rating process were considered in designing these workshops per best practice recommendations for conducting faculty development as outlined by Steinert et al. in the 2006 Best Evidence in Medical Education guidelines for faculty development programs.18


   Materials and Methods
Top
 Abstract
 Materials and Methods
Results
 Discussion
 Conclusion
 References
 
The PBL Assessment and Feedback workshop was advertised to the dental faculty in conjunction with the other PBL Core Skills workshops, and participation was required by the USCSD administration for those individuals involved with the education of the dental students. A total of 119 individuals completed this workshop during the thirteen times it was conducted between July 2003 and December 2004. Attendees included faculty members with dental and dental hygiene backgrounds as well as those with basic science backgrounds.

Workshop Development
The Faculty Development, Mentoring, and Evaluation (FDME) Subcommittee of the USCSD Curriculum Committee planned and implemented the PBL Core Skills workshop program. The overall goals for the PBL Assessment and Feedback workshop were to help faculty 1) develop an appreciation for the rationale behind assessment of individual students’ and group performance during PBL tutorials and 2) acquire skills needed to provide formative feedback to help students develop their PBL process skills. Specifically, the objectives were aimed to help participants understand:

  1. the importance of accurate individual PBL process assessment,
  2. how to define and emphasize the criteria for student performance evaluation in PBL cases,
  3. how, why, and when assessment is conducted in PBL cases,
  4. how to use the criteria-based evaluation system to record student performance in an objective format, and
  5. how to provide constructive feedback during the "facilitator to student" meetings based on performance criteria.

The workshop described in this report was designed to be the third in the series of faculty development programs, following workshops that first familiarized faculty with the overall PBL concept and process, and then introduced and demonstrated the roles, responsibilities, and methods associated with small-group facilitation.

Table 1Go demonstrates how the workshop activities unfolded, the time interval allotted to each activity, and how specific workshop activities were linked to learning outcomes. Learning objectives were shared with participants at the beginning of the workshop along with the approaches to be utilized during the course of the workshop. The teaching/learning methods incorporated into the workshop were informed by adult learning theory. Namely, the activities drew upon adult preferences for relevant, lifelike situations, and provided opportunities for the participants to share their prior knowledge and experience through active and collaborative discussions.19 These activities linked to the objective being addressed and drew upon multiple domains of learning where appropriate.


View this table:
[in this window]
[in a new window]

  		Table 1. Workshop program, resources, and learning outcomes
 
Subject-specific scenarios were used in this workshop to initiate discussions around the topic of qualitative assessment and feedback.17,20 Use of such scenarios directly involves the learner and incorporates adult preferences for relevant, lifelike problems. The scenarios utilized for this workshop (Figures 1Go and 2Go) described common misgivings shared by faculty about their role as assessors in the tutorial environment. The PBL Process Evaluation reporting form used at USCSD was built into the assessment scenarios to increase familiarity with the structure and function of the form.


Figure 1
View larger version (24K):
[in this window]
[in a new window]

  		Figure 1. Assessment scenario and visual aid

 

Figure 2
View larger version (13K):
[in this window]
[in a new window]

  		Figure 2. Feedback-specific scenario

 
The scenarios were used as follows. After tasking the participants to discuss the scenario in pairs, the moderator initiated a large-group discussion, applying participant-generated points back to the subject and providing specific responses to issues raised in the discussion. This activity served to probe participants’ prior knowledge and experience as well as their attitudes about qualitative assessment.

In preparation for the mock group observation that followed the scenario, the participants were given the criteria for student performance utilizing PBL process. The PBL process performance criteria have been organized into four broad domains: group skills, learning skills, reasoning skills, and feedback skills (summarized within Figure 1Go). The criteria associated with each domain describe a range of behaviors desired in students using the PBL process and also indicate performance levels designated as "below expected," "expected," and "above expected." The criteria and performance levels for the learning skills domain are included in Table 2Go.21 A full list of PBL process performance criteria utilized at USCSD can be obtained by contacting the corresponding author.


View this table:
[in this window]
[in a new window]

  		Table 2. Assessment criteria and performance levels for learning skills domain utilized at USCSD
 
Performance criteria related to group, learning, reasoning, and feedback skills in the "expected" range were distributed to the participants prior to the start of the mock group demonstration. Participants were instructed to observe a mock PBL group in action, make note of student behaviors exhibited, and match those behaviors with the performance criteria provided. Students participating in the mock PBL group were assigned specific behavioral characteristics (e.g., roles to portray), so they could demonstrate a range of behaviors, often observed in PBL groups, and faculty could thus observe specific student behaviors across the four assessment domains (Table 3Go). For example, one student was asked to propose hypotheses that were not linked to the information presented in the case, thus demonstrating a behavior that was inconsistent with aspects of reasoning skills described in the assessment criteria. Another student was asked to be passive and not participate, thus displaying a behavior related to poor group skills. The faculty member serving as the group facilitator was instructed to intervene minimally during the mock student group. The workshop participants were informed of the scaled-back role assigned to the facilitator.


View this table:
[in this window]
[in a new window]

  		Table 3. Mock group case and roles
 
After fifteen minutes of mock group demonstration, the workshop participants discussed what they had observed. The moderator reinforced that specific student behaviors could be linked to specific assessment criteria. When participants clearly misapplied the skills criteria to the demonstrated student behaviors, the audience was encouraged to correct the error with support from the moderator if necessary. For example, if a participant associated a student behavior with an inappropriate assessment domain, such as relating a learning skills student behavior to a group skills assessment domain, the moderator would point out a more appropriate use of the criteria from the specific list of learning skills behaviors.

Performance levels for each of the domains are determined not only by the frequency of behaviors exhibited but also by the range of behaviors displayed within the domain. These points are elaborated upon by the participants and the moderator as part of the discussion following the mock group demonstration. It is not expected that participants will be able to assimilate and apply the full range of criteria during the workshop. Rather, key points are derived from the exercises, namely, that behavior is linked to criteria; criteria are derived from desired performance goals; and performance levels are determined by the display of a range of behaviors, some of which are more challenging to exhibit than others.

The second portion of the workshop focused on the rationale for and delivery of formative feedback to students regarding their PBL case performance. This segment of the workshop was initiated with the use of a second short, feedback-specific scenario (Figure 2Go). This scenario was structured to cover the specific responsibilities of the facilitator to give feedback to students following completion of a case and to discuss characteristics of effective feedback delivery.

Subsequent to discussion of the scenario, poor and exemplary (i.e., "ideal") student feedback interactions were demonstrated to the participants by the faculty member running the workshop. Participants were asked to partner with a fellow participant and were assigned to give and receive feedback based on a specific mock student identified in the original mock student demonstration group. Content to be delivered during the feedback session included the assignment of a performance level to the student. One member of the pair was asked to assume the role of the assigned student, and the other the role of the facilitator. After first formulating a feedback strategy with one another, random pairs of participants were asked to demonstrate in front of the entire audience. After each role-play concluded, the audience was asked to critique the feedback that was provided to the student. The moderator positively reinforced aspects of the feedback and identified points that could have been incorporated into the feedback session.

The workshop concluded with a question and answer session, a wrap-up discussion, and the delivery of a survey to be completed by the participants. Selected literature on feedback strategies was distributed to the participants.

Post-Workshop Assessment
Figure 3Go displays the survey that was completed by workshop participants. Surveys were coded to remove personal identifiers from the analysis. Agreement that the workshop learning objectives had been met was queried using a four-point Likert scale (1=Strongly Disagree, 2=Disagree, 3=Agree, 4=Strongly Agree). Survey data were tabulated by assigning numerical values to the scale as indicated above. Mean values and standard deviations were calculated for each survey item. Participants were also asked to describe what workshop elements they felt were effective and which were ineffective. They were asked to offer suggestions to improve the workshop as well. Similar responses were grouped into themes where possible and their frequencies tallied.


Figure 3
View larger version (19K):
[in this window]
[in a new window]

  		Figure 3. Survey for the improvement of faculty development workshop

 

   Results
Top
 Abstract
 Materials and Methods
 Results
Discussion
 Conclusion
 References
 
Of the participants attending the Assessment and Feedback workshop, approximately 75 percent had a clinical dentistry or hygiene background. Nearly 10 percent had basic science backgrounds, and the remaining participants held dual, basic science, and clinical dentistry backgrounds. Over 80 percent of the workshop participants had served on the faculty for more than five years, with the remaining serving less than five years. Only a handful of participants were classified as postdoctoral researchers, and no dental specialty residents attended during the period studied.

Ninety-seven of the 119 participants completed the workshop evaluation survey for an 81.5 percent return rate. Mean participant agreement that the Assessment and Feedback workshop objectives had been met ranged between a minimum of 3.39 (SD ±0.57) for objective 3 and a maximum of 3.52 (SD ±0.62) for objective 5 (Table 4Go). Though there was agreement that all of the objectives had been met, participants expressed the most agreement with objective 5—namely, that they understood "how to provide constructive feedback during the ‘facilitator to student’ meetings based on standard performance criteria." Of the five objectives listed, numbers 3 and 4 were the two lowest ranked items; these workshop objectives were related to defining the rationale, basis, and formulation of "objective" PBL case assessments.


View this table:
[in this window]
[in a new window]

  		Table 4. Participant perceptions of workshop effectiveness relative to stated objectives
 
Effective Workshop Elements
Open-ended comments were grouped based on the overall subject of the comment. Multiple comments provided by the same individual were separated where unique items could be identified. Comments were tallied and ranked. From the ninety-seven total surveys completed, eighty-five individuals offered comments regarding effective elements of the workshop. Responses could be grouped into two major categories—workshop methods and workshop content—and one minor category—the role of the facilitator in assessment (Table 5Go).


View this table:
[in this window]
[in a new window]

  		Table 5. Most effective workshop elements
 
Participants identified the following as effective methodological elements of the PBL Process Assessment and Feedback workshop, with the following frequency: mock-group demonstration and role-playing (n=47), discussion amongst participants (n=27), and ability to apply assessment criteria (n=12). Less frequently mentioned items identified as being effective were the instructor(s), handouts, and workshop organization. One participant drew upon various components of the workshop stating, "The most effective element was the role-playing [mock group] observation and group discussion of what we have observed. The discussion gave me viewpoints from other individuals that were different from mine and things that I did not think of."

Content areas identified by participants as being effective elements of the workshop included improved awareness of assessment process and criteria (n=17), feedback strategies (n=16), and review of the PBL process (n=3). At times, participants would note valuing exposure to the assessment approach while still expressing skepticism about its objectivity. One faculty member described appreciating "exposure to written criteria—no matter the subjective nature of same."

Though mentioned infrequently, comments about the workshop in relation to the role of the facilitator in PBL were of interest especially as they drew, to some extent, upon the other core skills workshop. One participant commented, "This course really pulled it all together! The assessment [workshop] really ‘glued’ the PBL process. I better understand the role of facilitator."

Least Effective Workshop Elements
When queried about the least effective elements of the workshop, fewer and more varied responses were offered (Table 6Go). Fifty-one participants offered comments from which unique items were tallied. Twenty-three individuals remarked that all aspects of the workshop were effective. The remaining comments concerning least effective workshop components were grouped into four categories: 1) discussion-related, 2) methods-related, 3) insufficient time for topic, and 4) insufficient clarity/confidence with process/rationale.


View this table:
[in this window]
[in a new window]

  		Table 6. Least effective workshop elements
 
A handful of participants (n=6) indicated that some aspects of the workshop discussion were ineffective. These individuals felt that the discussion had gotten out of control, which was a reasonable observation from our perspective. During one implementation of the workshop, two or three participants shared strong opinions about the value of qualitative assessment methods.

A few participants found some of the methods utilized to be less than effective. The only pattern that emerged, however, was related to the delivery of feedback to a mock student, which was identified as a limitation by nine participants. One participant commented that it was "difficult to act out role-playing feedback session with no previous exposure or experience to process."

A still smaller number felt insufficient attention had been spent on assessment topics such as distinguishing between performance levels for the criteria and feedback topics. Others, with varying reasons, felt at odds with the type of assessment being explored in the workshop, citing disagreement with the assessment approach, ambiguity on the absence policy, and concern that the assessment criteria wouldn’t be applied uniformly.

Suggestions for Changes to Workshop
Participants were asked to provide suggestions on how to improve the PBL Assessment and Feedback workshop. Seventy individuals offered a range of suggestions that could be organized into two themes: 1) changes to existing content and methods, and 2) provision of additional facilitator support (Table 7Go). Twenty-seven participants offered no feedback.


View this table:
[in this window]
[in a new window]

  		Table 7. Suggestions for workshop improvement
 
Of those who suggested modification to the existing content and/or methods of the workshop, fifteen noted that further clarification on the use of the assessment criteria should be provided. Eleven participants thought that the mock group, role-modeling, role-play, or facilitator behavior could be modified. Thirteen felt the workshop could be improved by providing more time for discussion, more examples of group behavior linked to the assessment criteria, and more opportunities to practice the process of applying the criteria and giving feedback. Seven participants felt that the workshop would benefit from the provision of pre-workshop preparation materials such as pre-readings.

A substantial number of participants requested additional support for their roles as PBL facilitators. Twenty-two felt that additional workshops should be offered, and another thirteen expressed an interest in having the opportunity to observe facilitation and be mentored themselves.


   Discussion
Top
 Abstract
 Materials and Methods
 Results
 Discussion
Conclusion
 References
 
The PBL Assessment and Feedback workshop was the third in a series of PBL Core Skills workshops intended to familiarize faculty with the key skills involved in facilitating small-group learning. Considering the importance of assessment and feedback in the learning process and the assessment role played by the facilitator at USCSD, these skills were valuable areas to explore through a specific workshop program. Since assessment and feedback in PBL faculty training have been underrepresented both in medical and dental education, we believe that the addition of this program element is an enhancement in faculty development for PBL in dental education.13,14

A second and vital consideration for developing the PBL assessment and feedback workshop program was to address difficulties conducting performance-based assessments.5,12 Evidence from studies on in-training assessments in clinical placements and from appraisal processes in business suggested that there are a number of sociocultural variables that prevented raters from using rating instruments as they were intended.12 These variables include the raters’ personal views of what should be assessed and the purpose of the assessment as conveyed by the institution and interpreted by the rater. Faculty raters may also view their roles as both assessor and facilitator to be in conflict.12 It is important for faculty development efforts to explore the views that faculty have about the overarching purposes and uses of assessment, both broadly and within their own dental institution. It was for these reasons that we chose to use the assessment-related scenarios at the outset of the workshop. The workshop activities explored the PBL assessment criteria through their application in creating an assessment based on simulated student behavior and then explaining that criteria-based assessment to a mock student through role-play. The selection and sequence of methods were made in consideration of adult learning preferences for lifelike problems and active discussion, both of which are recommended by faculty development organizers in health professions education.14,18,19,22,23

When reviewing the perceptions of the participants and faculty development organizers from the PBL Assessment and Feedback workshop, several themes emerged that could guide future faculty development programs. These themes relate to 1) the general success of the range of methods used, especially the mock student group to demonstrate student behavior and serve as a mechanism for applying the student performance criteria; 2) the notion that assessment and feedback are topics that require extended exploration and application and therefore require additional learning opportunities for faculty; and 3) the fact that assessment practices are viewed differently by various faculty and may influence their application of the assessment process if not adequately addressed.

The primary indicator that the approaches used in the PBL Assessment and Feedback workshop were effective was reflected in the positive responses of the participants. The majority agreed that the objectives set forth had been met. When asked what had been most effective about the workshop, a large number of participants identified the mock group session, while a slightly smaller number singled out the group discussions and feedback-related role-play activities. A host of other aspects of the workshop were mentioned as being effective without being associated with a specific activity. Participants referred to effective aspects of the workshop as things that allowed them to apply the performance criteria. From our perspective, the performance criteria served as the strand running through the entire workshop, while the activities varied depending on the teaching task under consideration; for example, role-play was used to practice the skill of feedback delivery. We feel the participant comments support our choice of activities and are in agreement with the general findings of the Best Evidence in Medical Education (BEME) faculty development group, which notes "experiential learning, provision of feedback, effective peer and colleague relationships . . . and a diversity of educational methods within a single intervention" as elements of good practice.18

The outcome measures presented in this report are primarily focused on participant perceptions. This is a relatively easy outcome measure to conduct, and as one would expect, it is a widespread component of program evaluation.18,24 In terms of evaluating faculty development outcomes, the reactions of participants to training programs can serve as ground-level measurements and a starting point for determining program effectiveness and potential areas for program change.25 Other, more advanced measures that serve to determine program effectiveness are recommended and include measuring faculty learning through pre-post tests, behavioral change to teaching practice, and institutional change as a result of training endeavors. While positive participant perceptions about a program do not ensure learning or that faculty behaviors will be changed, Arthur et al. have shown that surveying participant reactions is associated with a medium and roughly similar effect size to other outcome measures such as learning and behavioral change.24 Delving further into the type of participant perceptions that are gathered, Alliger et al. have shown that surveying participants about the utility of the training program rather than just its affective impact (i.e., the enjoyment of the program) is more strongly associated with behavioral change.26 While the evaluation efforts in this report are limited to participant perceptions, we feel the organization of the survey around learning objectives improves our ability to gauge the effectiveness of the workshop with respect to its utility to participants. Clearly, to more robustly measure the effectiveness of the Assessment and Feedback workshop program, we will have to extend our analysis to the measurement of behavioral changes among the faculty with attention to such variables as changes to student perceptions of facilitators and the reliability of the facilitator assessment of student performance.

While the majority of participants expressed agreement that the workshop objectives had been met and that they approved of the methods used to learn about these areas, several factors are worth noting not only for their potential to impact the workshop program but also in terms of how the workshop participants will apply their new skills in practice. The participants who commented on less effective elements of the workshop generally referred to needing more exposure to examples of the criteria in use and to needing more opportunities to practice the skill of delivering feedback. Many of these individuals went on to suggest that further support was needed to develop these skills. During subsequent workshops, we have attempted to address these concerns by allotting more time to the participants to read the assessment criteria before starting the exercise. We have also introduced several short assessment scenarios related to specific student behaviors. Participants discuss these in pairs and report back to the larger group as an additional exercise in applying the assessment criteria.

Aside from these perceived and expressed needs to further develop the skills associated with PBL assessment, a number of individuals expressed concern about its rationale and purpose and wondered about its reliability. Some made these views known through the workshop discussion; others through their reflections on the surveys. Regardless of motivation, these comments underscore the difficulties in achieving a successful performance-based assessment system. Individuals may question their ability to assess in an unfamiliar system, or they may be unwilling to assess domains they don’t agree are necessary. Consequently, the value of the assessment may be compromised.

Despite our intentions to address these topics and provide practical opportunities for the faculty to use the PBL assessment criteria, it would seem that many participants feel that they would benefit from additional learning opportunities specific to PBL assessment and perhaps to assessment more broadly. Considering the complexity of assessing performance, this is perhaps no surprise. Govaerts et al. argued that a broader approach should be taken to looking at performance-based evaluation, of which assessing students in a PBL group may serve as one example.12 They argue that rating performance is akin to the clinical reasoning process in that it requires making judgments about what is under observation. Judgment is heavily influenced by both strong familiarity with the context and the specific viewpoints of the assessor. The former suggests that faculty would likely feel more secure in their judgments of performance if they were more familiar with what was taking place in the tutorial environment and how this aligns with the assessment criteria. To address this, additional learning opportunities should be provided for faculty to further familiarize them with the overall educational context in which student assessment takes place. This strategy was recommended by a substantial number of our workshop participants, which demonstrated their insight about their own learning.

Though meeting a distinctly different need, a similar argument could be made that the psychometric properties of PBL assessment could be enhanced by having peer discussions around the broader issues of assessment. As Govaerts et al. argue, the sociocultural variables that surround the assessment process itself should be addressed in a more meaningful way if we want to improve the standing of performance-based assessment.12 At this dental school, we have discussed developing a forum for facilitators to explore issues related to their roles and responsibilities. One goal for this forum would be to allow faculty to share their perceptions with a mind toward generating common goals and approaches for assessment of students in PBL tutorials. We envision this forum as being an additional venue for faculty to continue learning about the PBL pedagogy as well as other issues surrounding teaching and learning practices.

The importance of providing additional training for facilitators to help them acquire assessment and feedback skills is supported by findings from the arena of organizational training. Arthur et al. describe a gap between the learning that takes place in training programs and the application of that knowledge to job performance and propose that the favorability of the post-training environment plays a major role.24 Without support for application, the effectiveness of training may be attenuated. Considering the use of resources needed to run faculty development programs, it seems advisable to support faculty beyond the time allotted for workshop attendance alone.


   Conclusion
Top
 Abstract
 Materials and Methods
 Results
 Discussion
 Conclusion
References
 
This article has reported on the last of a series of PBL Core Skills workshops introduced to familiarize faculty with approaches to assessment and feedback in PBL. A range of activities including scenario-based discussion, mock group observation, and role-play were used to engage the faculty in the rationale and practice of PBL assessment and feedback. A majority of the participants expressed satisfaction with the methods used and, likewise, expressed agreement that the objectives set for the workshop had been reached. These findings are in concert with best practices for faculty development recommended in medical education18 and thus support the notion that such methods can be incorporated into the context of faculty development in dental education. Participant and program organizer input suggests that continued training and facilitator support is needed in the area of PBL student assessment and feedback delivery.


   Acknowledgments
 
The authors would like to thank Drs. Hsing Chi von Bergmann, Marian Bradford, Loris Abedi, Tae Jun Ahn, and Payem Sanjideh for their contributions to the creation of the PBL faculty development program at USCSD.


   Footnotes
 
Dr. Dalrymple and Dr. Wong are Assistant Professors, Division of Diagnostic Sciences, University of Southern California School of Dentistry; Dr. Rosenblum is Professor of Clinical Dentistry, Division of Health Promotion, Disease Prevention, and Epidemiology, University of Southern California School of Dentistry; Dr. Wuenschell and Dr. Paine are Assistant Professors, Division of Craniofacial Sciences and Therapeutics, University of Southern California School of Dentistry; and Dr. Shuler is Dean, University of British Columbia Faculty of Dentistry. Direct correspondence and requests for reprints to Dr. Charles F. Shuler, University of British Columbia, Faculty of Dentistry, 345-2194 Health Sciences Mall, Vancouver, BC V6T 1Z3, Canada; 604-822-5773 phone; 604-822-4532 fax; cshuler{at}interchange.ubc.ca.


   REFERENCES
Top
 Abstract
 Materials and Methods
 Results
 Discussion
 Conclusion
 References
 

  1. Newble DI, Jaeger K. The effect of assessments and examinations on the learning of medical students. Med Educ 1983; 17(3):165–71.[Medline]
  2. Newble DI, Entwistle NJ. Learning styles and approaches: implications for medical education. Med Educ 1986; 20(3):162–75.[Medline]
  3. Shumway JM, Harden RM. AMEE Guide No. 25: the assessment of learning outcomes for the competent and reflective physician. Med Teach 2003; 25(6):569–84.[Medline]
  4. Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of the evidence. Acad Med 1992; 67(9):557–65.[Medline]
  5. Eva KW. Assessing tutorial-based assessment. Adv Health Sci Educ Theory Pract 2001; 6(3):243–57.[Medline]
  6. Cunnington J. Evolution of student assessment in McMaster University’s MD programme. Med Teach 2002; 24(3):254–60.[Medline]
  7. Des Marchais JE, Vu NV. Developing and evaluating the student assessment system in the preclinical problem-based curriculum at Sherbrooke. Acad Med 1996; 71(3):274–83.[Medline]
  8. Hebert R, Bravo G. Development and validation of an evaluation instrument for medical students in tutorials. Acad Med 1996; 71(5):488–94.[Medline]
  9. Valle R, Petra L, Martinez-Gonzaez A, Rojas-Ramirez JA, Morales-Lopez S, Pina-Garza B. Assessment of student performance in problem-based learning tutorial sessions. Med Educ 1999; 33(11):818–22.[Medline]
  10. Eva KW, Solomon P, Neville AJ, Ladouceur M, Kaufman K, Walsh A, Norman GR. Using a sampling strategy to address psychometric challenges in tutorial-based assessments. Adv Health Sci Educ Theory Pract 2007; 12(1): 19–33.[Medline]
  11. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ 2005; 39(3):309–17.[Medline]
  12. Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment. Adv Health Sci Educ Theory Pract 2007; 12(2):239–60.[Medline]
  13. Hitchcock MA, Mylona ZE. Teaching faculty to conduct problem-based learning. Teach Learn Med 2000; 12(1): 52–7.[Medline]
  14. Farmer EA. Faculty development for problem-based learning. Eur J Dent Educ 2004; 8:59–66.[Medline]
  15. Dalrymple KR, Wuenschell C, Shuler CF. Development and implementation of a comprehensive faculty development program in PBL core skills. J Dent Educ 2006; 70(9):948–55.[Abstract/Free Full Text]
  16. Dalrymple KR, Wuenschell C, Rosenblum A, Paine M, Crowe D, von Bergmann HC, et al. PBL core skills faculty development workshop 1: an experiential exercise with the PBL process. J Dent Educ 2007; 71(2):249–59.[Abstract/Free Full Text]
  17. Wuenschell CW, Dalrymple KR, Shuler CF. PBL core skills faculty development workshop 2: training faculty in group learning facilitation skills through role-modeling and role-play activities. J Dent Educ 2007; 71(5):606–18.[Abstract/Free Full Text]
  18. Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M, Prideaux D. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach 2006; 28(6):497–526.[Medline]
  19. Knowles MS, Holton EF, Swanson RA. The adult learner. 5th ed. Woburn, MA: Butterworth-Heinemann, 1998.
  20. Friedberg S, Ash A, Brown E, Hallett D, Kasman R, Kenney M, et al. Teaching mathematics in colleges and universities: case studies for today’s classroom. Faculty ed. Washington, DC: American Mathematical Society & Mathematical Association of America, 2000.
  21. Mennin S. Faculty and student guide to problem-based learning tutorials in phase I curriculum. Office of Teacher and Educational Development, University of New Mexico School of Medicine. Paper presented at PBL Dental Education Conference, 2002.
  22. Pololi L, Clay MC, Lipkin M Jr, Hewson M, Kaplan C, Frankel RM. Reflections on integrating theories of adult education into a medical school faculty development course. Med Teach 2001; 23(3):276–83.[Medline]
  23. Wilkerson L, Irby DM. Strategies for improving teaching practices: a comprehensive approach to faculty development. Acad Med 1998; 73:387–96.[Medline]
  24. Arthur W, Bennett W, Edens PS, Bell ST. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol 2003; 88(2):234–45.[Medline]
  25. Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. 3rd ed. San Francisco: Berrett-Koehler Publishers, Inc., 2006.
  26. Alliger GM, Tannenbaum SI, Bennett W. A meta-analysis of the relations among training criteria. Personnel Psychol 1997; 50(2):341–58.



This article has been cited by other articles:


Home page
 J Dent EducHome page
D. C. Johnsen, M. W. Finkelstein, T. A. Marshall, and Y. M. Chalkley
A Model for Critical Thinking Measurement of Dental Student Performance
J Dent Educ., February 1, 2009; 73(2): 177 - 183.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Dalrymple, K. R.
Right arrow Articles by Shuler, C. F.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Dalrymple, K. R.
Right arrow Articles by Shuler, C. F.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS