JDE
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

J Dent Educ. 71(2): 235-241 2007
© 2007 American Dental Education Association
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Milani, J. E.
Right arrow Articles by Chambers, D. W.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Milani, J. E.
Right arrow Articles by Chambers, D. W.

Critical Issues in Dental Education

Comments as Part of a Clinical Competency Rating System

James E. Milani, D.D.S.; Howard H. Chi, D.M.D., M.A.; David W. Chambers, Ed.M., M.B.A., Ph.D.

Key words: clinical evaluation, faculty comments, ratings systems, professionalism, clinical dental education

Submitted for publication 07/07/06; accepted 10/13/06


   Abstract
Top
 Abstract
Materials and Methods
 Results
 Discussion
 References
 
The nature of comments faculty members make about students’ clinical performance and the relationship between comments and numerical scores given at the same time have not been studied in dental education. We developed a coding system for comments volunteered by faculty members to supplement the numerical ratings given as part of an established clinical competency evaluation system. Twenty-two hundred coded comments and their matched numerical ratings from more than 100 faculty members were compared for two classes of 146 and 157 students for four quarters. Thirty-seven percent of comments focused on dental performance, 43 percent on patient interactions, and 20 percent on relationships with faculty members or the clinic system. Forty-eight percent of comments were positive in tone. Where multiple comments were available for the same student in a quarter, 33 percent of them were confirmed by other faculty members; 9 percent were contradicted. Significant bidirectional associations were observed between comments (positive or negative) and numerical ratings in the area of patient interactions. For technical procedures and for interactions with faculty and the clinic, including professional demeanor, significant unidirectional associations predominated. The findings suggest that faculty members assume that students are skilled in these areas and tend to comment when their expectations are not fulfilled.

What do faculty comments tell us about students’ clinical competency? Some instructors tend to give corrective feedback on technical matters, while others offer blanket generalizations. Some tend toward the positive, and others only note problems. It is not even known for certain that faculty praise good performance and criticize weak students. This research was undertaken to characterize faculty comments in a mature clinical competency evaluation setting and to determine whether comments are associated in a regular fashion with numerical assessments made at the same time. We were especially interested in the degree to which faculty members comment on quasi-ethical student behavior such as performing treatment out of sequence, ignoring some patients, or working beyond their level of ability.

The literature on evaluation in dentistry focuses on reliability of numerical scales that are presumed to be objective.1–5 Validity studies (which require comparing preclinical or clinical ratings with outside standards) are rare,6,7 and no studies have been found that seek to explore the relationship between numerical ratings and subjective comments.

Several quantitative studies in the health fields report experience using comments in evaluation situations. Some of these studies measured the relationship between comments and outcomes such as board scores.8–14 Other studies examined student ratings and comments of their teachers15 or prediction of clinical performance from comments on admissions interviews.16,17 No studies could be located that compared comments and numerical ratings of students in a clinical context.


   Materials and Methods
Top
 Abstract
 Materials and Methods
Results
 Discussion
 References
 
At the University of the Pacific, Arthur A. Dugoni School of Dentistry, all students are evaluated at the end of each of the final eight quarters of their educational program in each comprehensive care clinical discipline by all faculty members who worked with them. A numerical competency rating system is used to assess clinical judgment, patient management, and skill in the disciplines of Diagnosis, Endodontics, Periodontics, Removable Prosthodontics, and Restorative Dentistry. This system and various characteristics of the numerical competency rating scale have been described in several published articles.1,7,18 The competency ratings are scaled to grading, promotion, and graduation decisions and are used, along with test case results, directly for those purposes.

At the time numerical evaluations are provided by faculty members, raters are invited to make comments as they deem appropriate. It is requested that ratings indicating the need for an intervention be accompanied by comment, which is almost universally the case. These comments are transcribed and grouped by student, resulting in twenty to thirty pages of comments per class each quarter. The comments are considered by a committee that meets separately from the Student Academic Performance and Promotion Committee for the purpose of identifying students who need special help. Group practice administrators, who manage teams of students, and department chairs also use the information for educational decisions and as part of the feedback they provide students.

We worked with several sets of comments from previous years, not included in this research, to develop a coding system. Comments tended to group naturally into three major clusters: 1) dental procedures, 2) interactions with patients, and 3) relations with the faculty and the clinic. Eighteen specific topics were identified across these three general categories, and seventeen of these could be meaningfully labeled bipolar (each having a negative and a positive end). "Overconfidence" or working beyond one’s ability only admits a negative coding. We continued to refine this coding system and check for its usefulness through independent scoring of new subsets. The coding system was considered satisfactory when we agreed that it covered all comments and an interrater agreement of 90 percent was achieved.

All numerical competency ratings and all comments were coded for the academic year 2004–05. These consisted of four quarters of ratings and comments for students in the Class of 2006 (the year before graduation) and four quarters of ratings and comments for students in the graduating Class of 2005.

Analysis consisted of frequency distributions and chi square tests to determine association with numerical ratings. Ratings were divided into 25/50/25 percent categories at the interquartile cuts for purposes of cross tabulation. Chi square analysis was performed on 2 x 3 tables (positive and negative poles on comments by interquartile categories for ratings) and in 1 x 3 tables, analyzing positive and negative poles of comments separately. Competency ratings in various disciplines were combined into a single technique rating. Students were rated by all faculty members regardless of department on clinical judgment and patient management.

This project was reviewed by the Institutional Review Board at California Pacific Medical Center and given exempt status, certificate #26.006.


   Results
Top
 Abstract
 Materials and Methods
 Results
Discussion
 References
 
Ratings were available from 146 students in the Class of 2006 and 157 students in the Class of 2005. Between 107 and 123 faculty members provided comments each quarter. There were 2,200 comments in all, for an average of 1.82 comments per student per quarter. (Figure 1Go shows the positive skew of comment frequency; a small number of students received many comments.)


Figure 1
View larger version (12K):
[in this window]
[in a new window]

  		Figure 1. Comments per student per quarter

 
A subsample of fifty comments was coded by two raters (the academic dean and a full-time clinical faculty member) for the sake of assessing consistency in applying the category system, resulting in 93 percent agreement.

The comments were approximately evenly divided between those that were positive in tone (48 percent) and those that were negative. In cases where more than one comment was offered for an individual student, 58 percent of comments were unique (no related comment was found among other comments for the student made during the quarter). In one-third of the cases, a comment was corroborated (also made by another faculty member). For example, one faculty member may have noted that a student "exhibits good interpersonal skills," and another commented that the same student showed "good patient rapport." In 7 percent of the cases where multiple comments were present, a contradiction was observed. An example of a contradiction would be "missing radiographs" and "neat, complete records."

Table 1Go shows the proportion of comments in each category. Thirty-seven percent of comments focused on dental performance, 43 percent on patient interactions, and 20 percent on relationships with faculty members or the clinic system. The most common specific comments were "good time management" (9 percent), "good treatment procedure" (9 percent), "good patient management" (8 percent), "poor treatment procedure" (8 percent), "poor time management" (6 percent), and "lack of independence" (5 percent). Only nine comment categories received less than 1 percent of the remarks—showing a wide variety in the topics commented on by faculty members.


View this table:
[in this window]
[in a new window]

  		Table 1. Comments on clinical performance by category (percentages are based on a total 2,200 comments; examples of faculty comments appear in italics)
 
Twelve percent of faculty comments could be classified as concerning professionalism. The most common of these was "devious; cuts corners" (2.9 percent), followed by "poor professional manner" (2.2 percent), "good professional manner" (1.9 percent), "working beyond ability" (1.8 percent), and "student’s interests ahead of patient’s" (1.2 percent). Where faculty commented on professionalism, 70 percent of the time the comments were negative.

Figure 2Go shows that the number of comments fluctuated by quarter, a factor under the control of faculty members. Chi square tests revealed that there was no difference across quarters (from Q5 through Q12) in proportion of positive/negative comments or distribution among the three major categories of dental performance, patient interaction, or relation with faculty and clinic.


Figure 2
View larger version (11K):
[in this window]
[in a new window]

  		Figure 2. Comments by major group across quarters

 
Table 2Go summarizes the analysis of associations between comments and numerical ratings made at the same time and for the same students. The arrows represent the results of chi square analyses that were significant at p<.05. The direction of the arrow indicates the directionality of the association. Where arrows point in both directions, the relationship is bidirectional. For example, negative comments about performance of treatment procedures are associated with low competency ratings for technique, and positive comments are associated with high scores (a bidirectional relationship). By contrast, the relationship between comments and ratings for understanding of treatment is unidirectional. Negative comments are associated with low scores, but positive comments are not associated with high scores. As a general pattern, features commented on by faculty in the area of interactions with patients are bidirectional; negative comments are associated with low ratings; and positive comments with high ratings (ten of fourteen significant associations). Generally, dental procedures and relationships with faculty and observance of clinical protocol are unidirectional: a relationship only exists between negative comments and low competency ratings (seventeen of twenty significant associations).


View this table:
[in this window]
[in a new window]

  		Table 2. Relationship between comments and numerical ratings
Chi square tests with comment present or absent (positive and negative pairs not combined) against three-group interquartile splits on average of technical skill, clinical judgment, and patient management scores.
 

   Discussion
Top
 Abstract
 Materials and Methods
 Results
 Discussion
References
 
It is noted that there was a roughly equal division between positive and negative comments and that a large range of comments was used by faculty members. It might have been expected that the ratio of comments about technical skills would be large at the beginning of clinical experience and that comments about patient interaction would constitute a larger proportion of the comments as graduation approached, but that was not the case. It was common that comments were confirmed by other instructors; but occasionally (7 percent of the time) opinions were disconfirmed. Faculty members do comment on students who exhibit questionable professionalism. Ringdahl et al.13 found in the case of ratings of family practice residents that 80 percent of comments were positive and tended to be general in nature. Similar generic and encouraging comments were reported to predominate in a medical technology board situation14 and a pediatric clerkship.11

The pattern of associations between comments and competency ratings does call for some consideration. First, it should be noted that comments about technical performance are associated with ratings in the areas of clinical judgment and patient management. Comments regarding patient interactions are associated with the patient management competency ratings (as expected) but also with technique and clinical judgment. The same generality appears in comments regarding relationships with faculty and observance of clinical protocol. This is not entirely unexpected in a clinical program that has a strong and long tradition of comprehensive patient care and which actively defines all competencies as reflecting the "combination of skill, understanding, and values necessary to begin independent dental practice." Chambers has presented evidence for a general dental competency.19

Laura Morgan Roberts, writing on the new literature in positive scholarship, notes that "positive states are not merely the opposite of negative states, and positive dynamics will not emerge by simply reversing negative dynamics" (p. 296).20 The general pattern of bidirectional association between comments and ratings for patient interactions and unidirectional (negative) associations for dental procedures and faculty and clinical protocol will also require further study and explanation. It is possible that faculty members place greater emphasis on patient interaction than on the other major categories or that they find it easier to comment in this area. It is even possible that students who feel confident in their dental skills are more at ease with patients. Faculty, however, are prompt to point out students "who only talk a good game."

The unidirectional association for dental performance and relations with faculty and clinical protocol where negative comments are associated with low competency ratings can also only be speculated on at this point. One explanation would be that faculty members assume that students will exhibit reasonable approximations to competence in dental treatments and their approach to learning. Confirmation of these expectations is unremarkable. It is only when a disappointment occurs that it must be noted, along with suggestions for improvement. The studies by Golnik et al.8 in ophthalmology, Hemmer and Pangaro10 among third-year medical residents, and Grams et al.9 of family practice residents were generally successful in their attempt to "retrospectively" identify marginal or troublesome students.

Some further credence to this "disappointment of expectations" hypothesis is gained by considering those comment categories associated with professional behavior. Except for "professionalism" (which includes some items such as dress code compliance), items such as "practicing beyond one’s ability," "receptiveness to faculty suggestions," and "deviousness" exhibit this negative unidirectional relationship with competency ratings. It may be that faculty members expect professionalism from students, and it calls for no comment unless it is absent.

Chamberlain et al.21 recently published an analysis of the relationship between personality traits and professional behavior in dental school. Those authors used a clinical rating item for "conscientiousness," which includes accepting responsibility, preparation, and pride. This end of the Chamberlain et al. scale for conscientiousness resembles the "professionalism" set of comments identified by faculty members in the present study. By contrast, the negative end of the Chamberlain et al. scale for conscientiousness includes "putting personal interests first," "poor time management," "defensiveness in the face of criticism," "lack of independence," and "lack of consideration for colleagues." It remains to be seen whether the Chamberlain et al. scale, which was constructed a priori, is a true bidirectional rating system or whether it confounds one set of attributes at the positive end and a different set at the negative end.

This investigation of comments from faculty members regarding student clinical performance given at the same time numerical ratings were provided demonstrates that faculty comments are wide-ranging and an independent source of information that augments the ratings. Faculty members comment on technical aspects of dentistry, patient management, and the relationship between students and faculty members and the clinic system. Comments are usually consistent across faculty members, but not always so. Comments were generally not merely "justifications" of their numerical marks, but meant to supplement or amplify number scores. Faculty members tended not to comment on the obvious, but were likely to make comments where their expectations for typical student performance were disappointed.


   Footnotes
 
Dr. Milani is Assistant Professor, Department of Restorative Dentistry; Dr. Chi is Assistant Professor, Department of Dental Practice and site director for the school’s Stockton AEGD Program; and Dr. Chambers is Associate Dean for Academic Affairs and Scholarship—all at the University of the Pacific, Arthur A. Dugoni School of Dentistry. Direct correspondence to Dr. David W. Chambers, University of the Pacific, Arthur A. Dugoni School of Dentistry, 2155 Webster Street, San Francisco, CA 94115; 415-929-6438 phone; dchambers{at}pacific.edu. Reprints will not be available.


   REFERENCES
Top
 Abstract
 Materials and Methods
 Results
 Discussion
 References
 

  1. Chambers DW, Glassman P. A primer on competency-based evaluation. J Dent Educ 1997; 61:651–66.[Medline]
  2. Meetz HK, Bebeau MJ, Thomas SJ. The validity and reliability of a clinical performance rating scale. J Dent Educ 1988; 52:290–7.[Abstract]
  3. Mecsher KD, Kerber PE. The effect of instructor variance on students’ grades. J Dent Educ 1982; 46:83–7.[Abstract]
  4. Reed TR, Feil PH, Greer DE. The reliability and agreement of subtask assessments. J Dent Educ 1988; 52:554–7.[Abstract]
  5. Weaver JR, Saeger W. A rater training program in removable prosthodontics. J Dent Educ 1984; 48:251–7.[Abstract]
  6. Chambers DW, Leknius C, Woodson R. Predictive validity of instructor judgment in preclinical technique courses. J Dent Educ 1997; 61:736–40.[Medline]
  7. Chambers DW. Do repeated clinical competency ratings stereotype students? J Dent Educ 1004; 68:1220–7.
  8. Golnik KC, Lee AG, Carter K. Assessment of ophthalmology residents’ on-call performance. Ophthalmology 2005; 112:1242–6.[Medline]
  9. Grams GD, Longhurst MF, Whiteside CB. The faculty experience with the "troublesome" family practice resident. Fam Med 1992; 24:197–200.[Medline]
  10. Hemmer PA, Pangaro L. The effectiveness of formal evaluation sessions during clinical clerkships in better identifying students with marginal funds of knowledge. Acad Med 1997; 72:641–3.[Medline]
  11. Lye PS, Biernat KA, Bragg DS, Simpson DE. A pleasure to work with: an analysis of written comments on student evolutions. Ambul Pediatr 2001; 1:128–31.[Medline]
  12. Rhoton MF. A new method to evaluate clinical performance and critical incidents in anesthesia: quantification of daily comments by teachers. Med Educ 1990; 24:280–9.[Medline]
  13. Ringdahl EN, Delzell JE, Kruse RL. Evaluation of interns by senior residents and faculty: is there any difference? Med Educ 2004; 38:646–51.[Medline]
  14. Teshima DY. Outcome measures of problem-based learning. Clin Lab Sci 2001; 14:68–9.[Medline]
  15. Leonard DL, Marion R, Niebuhr BR. The redundancy of student comments and numerical ratings in evolution of teaching. J Allied Health 1985; 14:141–7.[Medline]
  16. Meredith KE, Dunlap MR, Baker HH. Subjective and objective admissions factors as predictors of clinical clerkship performance. J Med Educ 1982; 57:743–51.[Medline]
  17. Raymond MJ, Sokol RJ, Vontver LA, Ginsburg KA. Candid candidate comments: the relationship between residency program selection factors and match list placements for ranked applicants. Am J Obstet Gynecol 2005; 193:1842–7.[Medline]
  18. Chambers DW. Faculty ratings as part of a competency-based evaluation clinical grading system. Eval Health Prof 1999; 22:86–106.[Abstract/Free Full Text]
  19. Chambers DW. Preliminary evidence for a general competency hypothesis. J Dent Educ 2001; 65:1243–52.[Abstract]
  20. Roberts LM. Shifting the lens on organizational life: the added value of positive scholarship. Acad Management Rev 2006; 31:292–305.
  21. Chamberlain TC, Catano VM, Cunningham DP. Personality as a predictor of professional behavior in dental school: comparisons with dental practitioners. J Dent Educ 2005; 69:1222–37.[Abstract/Free Full Text]



This article has been cited by other articles:


Home page
 J Dent EducHome page
J. E.N. Albino, S. K. Young, L. M. Neumann, G. A. Kramer, S. C. Andrieu, L. Henson, B. Horn, and W. D. Hendricson
Assessing Dental Students' Competence: Best Practice Recommendations in the Performance Assessment Literature and Investigation of Current Practices in Predoctoral Dental Education
J Dent Educ., December 1, 2008; 72(12): 1405 - 1435.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Milani, J. E.
Right arrow Articles by Chambers, D. W.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Milani, J. E.
Right arrow Articles by Chambers, D. W.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS