J Dent Educ. 68(6): 644-655 2004
© 2004 American Dental Education Association
Educational Methodologies |
Non-Graded Clinical Evaluation of Dental Students in a Competency-Based Education Program
Mohsen Taleghani, D.M.D.;
Eric S. Solomon, D.D.S., M.A.;
William F. Wathen, D.M.D.
Dr. Taleghani, who created this assessment program, is Professor and Chair, Department of General Dentistry, Baylor College of Dentistry, The Texas A&M University System Health Science Center; Dr. Solomon is Executive Director for Institutional Research, The Texas A&M University System Health Science Center; and Dr. Wathen is Associate Professor, Department of General Dentistry, Baylor College of Dentistry. Direct correspondence and reprint requests to Dr. Mohsen Taleghani, Department of General Dentistry, Baylor College of Dentistry, 3302 Gaston Ave., #313, Dallas, TX 75246; 214-828-8414 phone; 214-828-8952 fax; mtaleghani{at}tambcd.edu.
Key words: clinical evaluation, non-graded assessment, competency-based dental education, clinical instruction, quality assessment, remedial education
Submitted for publication 11/26/03;
accepted 03/29/04
 |
Abstract
|
|---|
The objective of this article is to report the development, implementation, and early results of a non-graded normative dental student clinical performance assessment system based on our competencies documents. The normative system (student performance is compared to evidence-based clinical standards) was used and evaluated during the 200203 academic year and is now gradually replacing the traditional summative (numerical) grading system previously used at Baylor College of Dentistry. The methodology included: 1) consensus development of new clinical performance assessment forms over the summer of 2002; 2) concurrent fourth-year clinical faculty calibration to the new forms; 3) implementation at the beginning of the senior year for the Class of 2003; and 4) faculty and student evaluation surveys in May 2003. Every step of each clinical procedure was recorded, weekly performance summaries by both students and faculty were collected, and periodic workshops were held to refine the forms and further calibrate faculty. The results showed strong positive responses to the new system by graduates and faculty alike. We conclude that early results warrant broadened efforts toward a continuously improved schoolwide normative student clinical performance assessment system.
Much has been written in the past decade about dentists and dental practices of the future.13 Todays educational and practice environments demand change at an ever-increasing pace. The move to competency-based dental education adds to that demand.
Dentistry rests on an educational foundation, yet must thrive in the competitive milieu of a rapidly changing world that demands continuous quality improvement through both personal and professional growth.4,5 Our dental schools must produce dentists who are firmly rooted in the ethical/moral life of traditional professions, evidence-based science, and sound clinical decision making: individuals who are biologically oriented, technically capable, and socially sensitive. At the same time, it is imperative that our graduates maintain an abiding desire for lifelong collegial relationships with their dental school and within the communities they will serve during their practice lifetime. Our previous assessment system was inadequate at measuring these factors. Therefore, we recognized a need to develop and implement a different student performance assessment strategy that complemented a competency-based curriculum and emphasized the nontechnical as well as the procedural aspects of modern dental practice.
Todays dental administrators and faculty should be knowledgeable about our changing profession and supportive of appropriate innovation in dental education. The scope and scale of such nontechnical innovation have been discussed by numerous authors,68 who have suggested models for dental schools to consider. A secondary objective of our program innovation was to emphasize these aspects of the dental education process and to include them in our faculty calibration strategy.
Furthermore, we appreciate the subtle, critical change that occurs in maturing learners as they assume increasing responsibility for control of their learning environment (Figure 1
). Understanding the differences between levels of competencies and how to successfully mentor our students through that segue from passive to active learner915 must be mastered and practiced by faculty (who have largely been trained in a different model). Only after incorporating that understanding into the faculty mindset and behavior can schools more successfully mentor students into the professional worlds that await.

View larger version (38K):
[in this window]
[in a new window]
|
Figure 1. As learners move beyond formal education and mature toward mastery in their professions, they assume increasing responsibility for directing their own professional development strategies.
|
|
Collegial mentoring requires assessment and feedback systems that enhance learning, the students professional growth, and pride in being a dentist. Calibration workshops help ensure that faculty can and will model the defined competencies. Traditional dental school clinical assessment has often consisted of a faculty member simply looking at a students work and making snap decisions about the product. Such techniques are highly subjective and variable.16 Berrong et al.,17 as well as Mackenzie et al.,18 analyzed such discrepancies and identified common, specific factors that reduce assessment agreement among clinical faculty and create confusion among students (Table 1
). Our system attempts to eliminate these factors by requiring faculty to enter written explanation of any aspect of patient care that lies outside the clinically acceptable norms contained in our faculty calibration manual.
Early attempts to diminish the inter- and intra-rater discrepancies created by the traditional system consisted of rating scales that translated into the summative grading system with which many of us are familiar.19 Baylor College of Dentistry previously used a traditional block curriculum with the 04 grading system (F to A) to evaluate student clinical performance. As the curriculum changed to a comprehensive, clinical competency-based approach, the shortcomings of this evaluation method became increasingly evident.
Our traditional system was highly subjective, difficult to calibrate, and susceptible to individual faculty personalities, and it offered insufficient teaching opportunities and discussion time with students. Many faculty members tended to take the easier approach of giving out As because they didnt feel they had the time to discuss or argue with students who were graded down. Thus, the long-recognized assessment errors of leniency (some faculty usually give high grades), central tendency (some faculty usually give a B or C), and severity (some faculty usually give low grades) were predictably present, skewed dramatically toward the path of least resistance: leniency (Figure 2
). It was necessary to devise a simultaneous faculty evaluation and feedback system.20
To summarize, we wanted the new assessment system to:
- eliminate summative clinical grading,
- establish clinical performance norms based on the competency model,
- transition to a normative assessment model that compares student performance to those clinical performance standards,
- enhance faculty/student relationships and environment to better express professional collegiality,
- diminish student feelings of threat in the clinics,
- diminish faculty feelings of stress in a graded situation,
- bring collegial mentored education to the forefront of all clinical activities, as the primary faculty expectation, and
- ensure student transition from passive to enduring active learners with enhanced critical thinking and clinical decision-making skills.
 |
Materials and Methods
|
|---|
The transition to competency-based education, begun in 1992, is now a comprehensive clinical application. As previously reported,21 we began to construct a student evaluation system to pinpoint and document performance deficiencies that could lead to clinical failure. The Department of General Dentistry began the clinical implementation of our competencies documents by establishing fundamental assessment standards and criteria broadly based on clinical faculty input and identified ideals.4,1423 (See Table 2
.)
The process of developing new clinical performance assessment forms began in January 2002 and was concluded in the early summer of 2002. After general consensus was reached among clinical faculty from all departments, the fourth-year general dentistry faculty were calibrated to the clinical standards agreed to by consensus, and the forms were introduced in June at the beginning of the senior year for the Class of 2003.
A historical assessment problem involves the behavioral (nontechnical) aspect of dentistry. Baylor College of Dentistrys original competency document identified issues of Professionalism, Procedure and Patient Management, and Skills/Traits and detailed them in objective elements that specify those nonclinical aspects of being a dentist. These elements of professionalism are included in the assessment of all clinical activities that relate to specific nonprocedural aspects of practice (Figure 3
). The lines on the assessment forms under each of these headings define what is meant by each term: for example, under "Skills and Traits" the attending faculty member assesses 1) the degree of confidence and independence exhibited by a student during a patient encounter; 2) how the student reacts to stress that may occur during that encounter; 3) the level of manual skills demonstrated by the student during the encounter; and 4) the level of appropriate interpersonal skills related to patients, faculty, and staff members.

View larger version (67K):
[in this window]
[in a new window]
|
Figure 3. The assessment forms are two-part. At the conclusion of the appointment one copy of the form is given to the student and one to data entry.
|
|
Clinical procedures were similarly reviewed and disaggregated into key steps, sequentially arranged. Using the assessment sheets as a fundamental checklist, faculty assess these critical steps (both nontechnical and clinical) that predict success or failure of the procedure, privately discuss their observations with the student as the event occurs, and write their requirements of performance improvement (numbered to correspond to the line number of the assessment form) in the "comments" section of the clinical evaluation forms. Staff members then enter those comments into both the students and the facultys individual databases. Thus, the traditional Socratic teaching method is revived as targeted learning takes place in a mentored learning environment. Daily input into the colleges clinical management system allows extraction of such data and immediate identification of trends and areas needing improvement for both faculty and students.
The development of objective student evaluation forms encompasses the totality of general dental practice, tied to the Baylor College of Dentistry competency document. Thirteen forms linked to specific categories (disciplines/departments) of dental care are now in use as the college moves towards consistency among departments (Table 3
).
As clinical procedures are performed, appropriate assessment forms are begun as part of the pre-operative record. As the procedure progresses, both nontechnical and technical aspects are evaluated by attending faculty. Each procedure step must be adequately accomplished (meet the clinical standards contained in our calibration manual) before the student is allowed to proceed to the next step. When the procedure is complete, the two-part assessment form serves as instant feedback to the student and raw material for data entry. One copy of the form is given to the student immediately, and the other is submitted to data entry personnel. The entries are added to the students permanent electronic record, creating a real-time individual competency profile that supervising faculty can access to evaluate the student in all areas. Similar profiles of faculty performance are recorded (Figures 4
and 5
).
Fourth-year faculty and graduating students completed evaluation surveys about the effectiveness of the Non-Graded Clinical Evaluation System in May 2003. In addition, a statistical test was conducted that compared the grade distributions from the traditional grading system to the progress grades now used with the Non-Graded Clinical Evaluation System.
A final note about our grading methodology is important. A summative (4-0 or A-F) grading system remains in place for specific interim evaluations of competence. Under the Non-Graded Clinical Evaluation System, grades are awarded on progress examinations, conducted for specific procedures at prearranged times under controlled conditions. These grades are based on specific criteria, evaluated by multiple calibrated faculty, and serve to corroborate student performance in a traditional way that is consistent with requirements for graduate school applications and class rankings.
 |
Results
|
|---|
Ninety-nine percent of the fourth-year students completed the satisfaction survey of the Non-Graded Clinical Evaluation System (NGCES). The students were asked to compare this system, which was used in their fourth year, to the traditional grading system they worked under during their third year. The overall results were quite favorable for the new system (Table 4
). Almost four out of five students thought the NGCES provided a better learning environment. A similar percentage of respondents also believed their interactions with clinical faculty were improved using the system. Accordingly, it was not surprising that two-thirds of the students indicated they received more constructive comments on their techniques with the new system. The level of stress in the clinic has been a concern of the students for many years. Over 70 percent of the fourth-year students thought the system helped foster a less stressful clinic environment. As far as performance evaluation is concerned, over 85 percent of the respondents thought their performance was adequately evaluated under the new system. Since this was the first year for the NGCES, we asked the students whether their concerns were addressed. Eighty-two percent indicated their concerns had been addressed. Finally, we asked the fourth-year students whether they would recommend that the Non-Graded Clinical Evaluation System be instituted in the third year as well. Here the responses were positive overall, but there was some dissent. About 71 percent of the students were in favor of using it in the third year; however, 16 percent were neutral on the question, and 14 percent were not in favor of changing the third-year grading system.
A satisfaction survey was also administered to the faculty of the General Dentistry Department. One hundred per cent of the department faculty (seven full-time and twelve part-time) completed the satisfaction survey of the Non-Graded Clinical Evaluation System. The faculty were asked to compare the NGCES used during the 200203 academic year to the traditional grading system with which they worked previously. As in the student survey, the overall results were quite favorable for the new system (Table 5
). About 95 percent of the faculty thought the system provided a better learning environment. Over two-thirds of the respondents thought the learning environment was "much better." Eighty-four percent of the faculty also believed their interactions with students were improved and indicated they provided more constructive comments on their techniques with the new system. About 74 percent of the faculty thought the clinic environment was less stressful using the NGCES although most of these respondents thought the stress level was "less," not "much less." As far as performance evaluation is concerned, almost 90 percent of the faculty thought they were better able to evaluate performance under the system. We also asked the faculty whether they believed the new system helped them to be a more effective teacher, and almost 90 percent responded affirmatively. Finally, we asked the faculty whether they would recommend that the NGCES be instituted for all clinical and preclinical courses. As with the students, the responses were positive overall, but there was some dissent. About 74 percent of the faculty were in favor of using the system in all clinical and preclinical courses; however, 11 percent were neutral on the question, and 16 percent were not in favor of changing the grading system. Differences between the responses of full- and part-time faculty were explored, and no significant differences were found.
As noted, the traditional grading system had a tendency to foster leniency in grading. Figure 2
shows the distribution of grades in the fourth-year class using the traditional grading system for academic year 200102. The grades here are highly skewed, with over 91 percent of all grades in the two highest categories. The average grade was a 3.79 on a four-point system.
Under the Non-Graded Clinical Evaluation System, grades are only awarded on progress examinations. These examinations are conducted for specific procedures, at pre-arranged times, under controlled conditions, and are graded on specific criteria by multiple evaluators. Figure 6
shows the distribution of grades for the 200203 academic year. By using grades from progress examinations rather than daily grading, there are far fewer student grades and a greater distribution of grades. Grades still tend to be quite high, with almost three out of four grades in the top two categories and an average grade of 3.59.
Although the mean grade for each grading system was relatively similar, the distribution of progress grades appears to be greater. To verify this, a t-test was conducted to determine whether the means of each distribution of grades were statistically different. The results indicate that the average grades under each system were statistically significantly different (p
.001).
 |
Discussion
|
|---|
The results of the student and faculty evaluation surveys demonstrated a high level of acceptance of the NGCES. About 80 percent of students and 95 percent of faculty indicated the NGCES provided a better learning environment. Other related evaluation criteria were also positively rated by students and faculty. While many refinements will no doubt be incorporated in the future, students and faculty agree that better collegial relationships have resulted in a more relaxed atmosphere and a better teaching environment. Elimination of daily grades allows faculty and students to work together more easily as a team, which appears to have improved both the quality and quantity of procedures completed. Future satisfaction surveys of both faculty and students will elucidate this opinion.
The new clinical performance assessment has not only resulted in a high level of both student and faculty acceptance, but has also produced a broader grade distribution among the class. Under the old grading system, grades were not just highly skewed; there were a lot of them as well. During the 200102 academic year, fourth-year students received 33,473 grades. Only progress grades were awarded to fourth-year students during the 200203 academic year, resulting in a total of 817 grades. Thus, a largely meaningless accumulation of data was eliminated.
The faculty calibration and student assessment process recently implemented follows the mission-focused model proposed by Chambers et al.,20 concentrating on objective identification and documentation of areas that require improvement by students and faculty alike. The daily entry of performance data defined by the assessment forms allows identification of students who are not making satisfactory clinical progress, so remediation can begin immediately. These elements comprise the balance of the assessment forms.
From an educational standpoint, faculty members now have familiar templates that provide precious opportunities to teach. Under the previous system, typical verbal feedback at the chair went largely unremembered, and there was no reliable documentation trail for tracking either student or faculty performance. Teaching opportunities abound now because any failure to attain a level of predictable clinical success for each procedural step requires a check mark and may have written documentation of the step and the means used to correct the shortcoming. Daily entry of such raw data from the assessment forms allows both students and faculty to measure themselves against our normative clinical models.
The challenge while moving to a competency-based educational system was stated simply: eliminate the maximum number of problems identified in the introduction while creating a mentored learning environment conducive to enduring adult education and lifelong collegiality between our graduates and the school. The creation and implementation of such a program are difficult and have yet to be completed. At this point, however, it seems to be worth the effort. Our new system centralizes all student performance data for responsible faculty to review on a regular basis. Problems can now be identified and pinpointed. Faculty can rapidly engage appropriate remedial strategies to overcome student deficiencies in either technical or nontechnical areas.
Students meet regularly with their primary faculty mentor (Group Leader). These sessions cover all issues of concern, and students now typically report their realization that the new clinical assessment strategy is a nonthreatening evaluation system wherein the information gathered is used to help them achieve increased levels of clinical competency. Students and faculty alike are now more willing to engage in clinical discussions and the remediation process.
At the same time that daily and cumulative student performance profiles are generated from the individual assessment sheets, similar data are generated for faculty performance. Since our system is focused on "best practices" teaching, the information is now available for faculty to self-assess and strengthen their own weaker areas. A calibration manual has been written to detail each item of the evaluation forms and to discuss current clinical best practices.
Another way to use the data generated by this system is to assess curriculum deficiencies. We discovered early in the first year of using the evaluation system, for example, that an unusually high number of students were leaving caries and decalcification in their preparations. It was determined that more time in preclinical courses should be spent assisting students to recognize and remove decay from preparations, more attention paid on the clinic floor, more teaching opportunities sought, and immediate remediation with extracted teeth when the deficiency persists beyond rare isolated incidents.
After the 200203 year, the system was adopted by the curriculum committee for application in the second and third years. Therefore, the upcoming evaluation surveys for 200304 and 200405 will include students who have attended Baylor under both the old and the new systems, and all clinical faculty will be included henceforth.
 |
Conclusions
|
|---|
The new clinical performance assessment resulted in a high level of student and faculty acceptance and produced a broader grade distribution among the class.
We perceive better collegial relationships between faculty and students, a more relaxed atmosphere, and a better teaching environment. Elimination of grades seems to have allowed faculty and students to work together more easily as a team.
Faculty report that it is easier to give detailed written feedback to students and that the system is uncomplicated and user-friendly. Faculty calibration is better organized and sequenced, and helps new faculty in adjusting to a satisfying teaching career. The system creates a centralized information file for each student for ease of tracking their progress in all disciplines and departments, making it a simple mater to identify areas that require improvement. Faculty are able to quickly create customized remedial programs to overcome specific areas of concern. Department chairs and program directors can objectively measure faculty activity in the clinics to ensure ongoing calibration and development of faculty members.
The task is far from complete, but early outcomes of this assessment strategy are positive. The new evaluation process appears to have reduced or eliminated most of the problems listed in the introduction. Expanded surveys will now compare student and faculty responses to both old and new assessment systems. Performance on licensing boards will be compared, as will career satisfaction surveys among our graduates. Faculty surveys will search for the strengths and weaknesses of the new system and will continue to compare faculty responses to using the system.
In summary, we believe the new system has provided a better environment for students and faculty alike and will be a very positive factor in support of Baylors "Lifelong Colleague" initiative for the practicing community.
 |
REFERENCES
|
|---|
- Field MJ, ed. Dental education at the crossroads: challenges and change. Washington, DC: National Academy Press, 1995:94100.
- Future of dentistry. Chicago: American Dental Association, Health Policies Resource Center, 2001:19,21,88114.
- Oral health in America: a report of the surgeon general. Rockville, MD: National Institute of Dental and Craniofacial Research, National Institutes of Health, U.S. Department of Health and Human Services, 2000.
- Low DS, Kalkwarf KL. Assessing continued competency: an approach for dentistry. J Am Dent Assoc 1996; 127(3):3838.
- Drucker P. The new workforce in the next society: a survey of the near future. The Economist, November 3, 2001:811.
- Dugoni AA. University of the Pacific School of Dentistry: a dental school that serves the practicing profession. Cal Dent Assoc J 2002;30(10):7638.
- Christie CR, Coun M, Bowers DM, Paarmann CS. Curriculum evaluation of ethical reasoning and professional responsibility. J Dent Educ 2003;67(1):5563.[Abstract]
- Shuman SK. Doing the right thing: resolving ethical issues in geriatric dental care. Cal Dent Assoc J 1999; 27(9):694702.
- Glassman P, Chambers DW. Developing competency systems: a never-ending story. J Dent Educ 1998;62(2): 17382.[Abstract]
- Merriam SB, Brockett RG. The profession and practice of adult education. San Francisco: Jossey-Bass Publishers, 1997.
- Candy PC. Self-direction for life-long learning. San Francisco: Jossey-Bass Publishers, 1991.
- Brookfield SD. Understanding and facilitating adult education. San Francisco: Jossey-Bass Publishers, 1992.
- Knowles MS. The modern practice of adult education: andragogy versus pedagogy. Chicago: Follett Publishing Company, 1970.
- Mossey PA, Newton JP, Stirrups DR. Defining, conferring and assessing the skills of the dentist. Br Dent J 1997;182(4):1235.
- Hendricson WD, Kleffner JH. Curricular and instructional implications of competency-based dental education. J Dent Educ 1998;62(2):18396.
- OConnor P, Lorey RE. Improving inter-rater agreement in evaluation in dentistry by the use of comparison stimuli. J Dent Educ 1978;42(4):1749.
- Berrong JM, Buchanan RN, Hendricson WD. Evaluation of practical clinical examinations. J Dent Educ 1983; 47(10):65663.
- Mackenzie RS, Antonson DE, Weldy PL, Welsch BB, Simpson WJ. Analysis of disagreement in the evaluation of clinical products. J Dent Educ 1982;46(5):2849.
- Kerlingen FN. Foundations of behavioral research. New York: Holt, Reinhart, and Winston, 1966:5167.
- Chambers DW, Boyarsky H, Peltier B, Fendler F. Development of a mission-focused faculty evaluation system. J Dent Educ 2003;67(1):1022.
- McCann AL, Babler WJ, Cohen PA. Lessons learned from the competency-based curriculum initiative at Baylor College of Dentistry. J Dent Educ 1998;62(2):197207.
- Yip H-K, Smales RJ. Review of competency-based education in dentistry. Br Dent J 2000;189(6):3246.
- Yip H-K, Smales RJ, Newsome PRH, Chu FCS, Chow TW. Competency-based education in a clinical course in conservative dentistry. Br Dent J 2001;191(9):51722.
This article has been cited by other articles:

|
 |

|
 |
 
A. Polychronopoulou and K. Divaris
Dental Students' Perceived Sources of Stress: A Multi-Country Study
J Dent Educ.,
May 1, 2009;
73(5):
631 - 639.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
A. M. Hauser and D. M. Bowen
Primer on Preclinical Instruction and Evaluation
J Dent Educ.,
March 1, 2009;
73(3):
390 - 398.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
G. A. Kramer, J. E.N. Albino, S. C. Andrieu, W. D. Hendricson, L. Henson, B. D. Horn, L. M. Neumann, and S. K. Young
Dental Student Assessment Toolbox
J Dent Educ.,
January 1, 2009;
73(1):
12 - 35.
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
J. E.N. Albino, S. K. Young, L. M. Neumann, G. A. Kramer, S. C. Andrieu, L. Henson, B. Horn, and W. D. Hendricson
Assessing Dental Students' Competence: Best Practice Recommendations in the Performance Assessment Literature and Investigation of Current Practices in Predoctoral Dental Education
J Dent Educ.,
December 1, 2008;
72(12):
1405 - 1435.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
F. W. Licari and D. W. Chambers
Some Paradoxes in Competency-Based Dental Education
J Dent Educ.,
January 1, 2008;
72(1):
8 - 18.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
F. W. Licari, G. W. Knight, and P. J. Guenzel
Designing Evaluation Forms to Facilitate Student Learning
J Dent Educ.,
January 1, 2008;
72(1):
48 - 58.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
D. L. Hall
Oral brush biopsy technique instruction outcomes for senior dental students.
J Dent Educ.,
August 1, 2006;
70(8):
820 - 824.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
M. Taleghani, E. S. Solomon, and W. F. Wathen
Grading Dental Students in a "Nongraded" Clinical Assessment Program
J Dent Educ.,
May 1, 2006;
70(5):
500 - 510.
[Abstract]
[Full Text]
[PDF]
|
 |
|