Improving interrater agreement in evaluation in dentistry by the use of comparison stimuli

Research shows that use of rating scales with precisely defined scale points improves interrater agreement. Instead of words, scale points may be defined by products for which ratings have been established by consensual agreement. These products serve as standards for raters. In two studies, raters assessed student products using conventional rating scales and identical scales together with comparison stimuli (photographic exemplars) of consensually established values of 3 (adequate) for each criterion. Results of the first study (but not the second) clearly confirmed the contribution of exemplars to interrater agreement. Findings suggested that single exemplars significantly enhance agreement for criteria on which the ideal is an end point on a continuum. Multiple exemplars may be necessary for other criteria. Comparison stimuli offered promise for improving interrater agreement, but further work is necessary to adapt their use for complex criteria that frequently define the quality of dental preparations.

This article has been cited by other articles:

				M. E. Jacks, C. Blue, and D. Murphy Short- and Long-Term Effects of Training on Dental Hygiene Faculty Members' Capacity to Write SOAP Notes J Dent Educ., June 1, 2008; 72(6): 719 - 724. [Abstract] [Full Text] [PDF]

				R. Haj-Ali and P. Feil Rater reliability: short- and long-term effects of calibration training. J Dent Educ., April 1, 2006; 70(4): 428 - 433. [Abstract] [Full Text] [PDF]

				M. Taleghani, E. S. Solomon, and W. F. Wathen Non-Graded Clinical Evaluation of Dental Students in a Competency-Based Education Program J Dent Educ., June 1, 2004; 68(6): 644 - 655. [Abstract] [Full Text] [PDF]