J Dent Educ. 68(10): 1104-1111 2004
© 2004 American Dental Education Association
International Perspectives on Dental Education |
Strategies for Training Standardized Patient Instructors for a Competency Exam
Hitoshi Amano, D.D.S., Ph.D.;
Tsukasa Sano, D.D.S., Ph.D.;
Kinuko Gotoh, D.D.S., Ph.D.;
Saburo Kakuta, D.D.S., Ph.D.;
Takeshi Suganuma, D.D.S., Ph.D.;
Yuichi Kimura, D.D.S., Ph.D.;
Hiroaki Tsukasaki, D.D.S., Ph.D.;
Hajime Miyashita, D.D.S., Ph.D.;
Tomohiro Okano, D.D.S., Ph.D.;
Nobuichi Goto, Ph.D.;
Haruko Saeki
Dr. Amano is Assistant Professor; Dr. Sano is Professor; Dr. Gotoh is Assistant Professor; Dr. Kakuta is Associate Professor; Dr. Suganuma is Assistant Professor; Dr. Kimura is Associate Professor; Dr. Tsukasaki is Associate Professor; Dr. Miyashita is Professor; Dr. Okano is Professor; Dr. Goto is Professor—all at the School of Dentistry, Showa University, where all are also members of the OSCE Committee. Mr. Saeki is Visiting Lecturer, School of Medicine, Showa University and head of the Tokyo Working Group for Simulated Patient Communication. Direct correspondence and requests for reprints to Dr. Hitoshi Amano, Department of Pharmacological Dentistry, School of Dentistry, Showa University, Hatanodai 1-5-8, Shinagawa-ku Tokyo, Japan 142-8555; 81-3-3784-8175 phone; 81-3-3784-8176 fax; amano{at}dent.showa-u.ac.jp.
Key words: OSCE, SP instructor, training program, medical interview
Submitted for publication 03/03/04;
accepted 08/18/04
 |
Abstract
|
|---|
The Common Achievement Test (CAT) in Japan, which will be implemented in 2005, involves a medical interview that is the core task to be completed by students during an Objective Structured Clinical Examination (OSCE). Standardized/Simulated Patient instructors (SPs), posing as patients in medical interviews, are trained in standard fashion in terms of expression of symptoms as well as the emotional affect of actual patients. Institution of appropriate training programs for SP instructors in the CAT is also necessary. We trained seven individuals to function as standardized patients (in-school SPs) during a three-day SP training program described in this article. Following completion of the OSCE, we conducted a comparison study among evaluations completed by the evaluators and two types of SP instructors. We observed high correlation, according to Spearman significance testing, between scores of evaluators and those of both newly trained in-school SPs and veteran SPs who had more than five years of experience. Correlation coefficients between the veteran SPs (r=0.77) and the in-school SPs (r=0.73) were nearly identical. These results suggest that our training program for SP instructors is an effective protocol, particularly with respect to reliability and efficiency.
In 2001, the Ministry of Education, Culture, Sports, Science and Technology of Japan proposed a Medical and Dental Education Reform Plan for the 21st Century that introduced clinical training in a patient care environment known as a "clinical clerkship." It was considered essential that our students participate in clinical training in order to demonstrate their ability in terms of well-mannered behavior, technical skills, knowledge, and problem-solving skills when providing health care to patients. To facilitate the implementation process for this curriculum reform, the Common Achievement Test Organization was established in cooperation with participating medical and dental universities throughout Japan. Prior to the 2001 reform plan, dental and medical students in Japan received limited clinical experience.
The Common Achievement Test (CAT) was designed to evaluate medical and dental students in terms of behavior, technical skills, knowledge, and problem-solving skills prior to the start of clinical training. The CAT consists of two components: 1) a computer-based evaluation of biomedical knowledge and problem-solving skills, and 2) an Objective Structured Clinical Examination (OSCE) that evaluates clinical examination skills as well as behavior during patient care. The trial stage of the CAT began in 2002, and official implementation is scheduled for 2005. The OSCE component of the CAT includes a medical interview as a required task. The medical interview station of the OSCE is based on the fact that inadequate communication by the medical practitioner can result in misunderstanding and, ultimately, medical malpractice. Communication skills are essential for medical practitioners to establish trust during treatment, so it is important that we evaluate this skill during the OSCE.
The concept of Standardized/Simulated Patients (SP) was first introduced by Harden et al.1 In recent years this concept has provided a very important human resource with respect to communication training in the medical2 and dental fields.3–9 SPs are trained to function as patients during medical interviews, but one of the most important roles of SPs is to evaluate the students’ manner of speech and behavior as well as their interviewing and physical examination skills. To develop SPs, individuals must be trained in standard fashion so that they learn how to express the symptoms as well as the emotional affect of actual patients.
It is necessary to institute appropriate training programs for SP instructors in the format of a qualifying examination. This article will describe how we trained seven in-school SPs during a three-day course designed by Haruko Saeki, head of the Tokyo Working Group for Simulated Patients. The training was conducted three months prior to OSCE implementation. We then examined the relationship between evaluation scores generated by faculty evaluators, experienced SPs trained elsewhere, and our in-school SP instructors. This article will report the results.
|
Methods
|
|---|
Training Program for SP Instructor
Volunteers for SP instructor training were recruited from staff (exclusive of dentists or dental hygienists) and second- through fourth-year dental students. The three-day SP training program was conducted three months prior to OSCE implementation under the direction of the program planner, Haruko Saeki, head of the Tokyo Working Group for Simulated Patients. The agenda and schedule for the SP training program are presented in Figure 1
.
Following observation of a model patient simulation by a veteran SP instructor on Day 1, basic communications training was provided. This training was provided by veteran SP instructors sent by the Tokyo Working Group for Simulated Patients. These veteran SPs had already completed classes in medical interviews at other medical universities and had at least five years of experience with OSCEs. On Day 2, each participant created a scenario depicting a patient with an oral health problem who comes to see the dentist. Then they were each videotaped performing the patient role in their scenario. At the end of the day, each SP trainee met with instructors to evaluate the videotape of their SP performance. On Day 3, standardization of simulated acting and evaluation among SP instructors were performed in collaboration with clinical resident dentists. Ultimately, seven in-school SP instructors were trained, including two students.
|
OSCE Implementation
|
|---|
Serving as a CAT trial, an OSCE was hosted by the School of Dentistry, Showa University. Implementation and administration were the responsibility of the OSCE Committee at the School of Dentistry, Showa University. The OSCE included five stations: 1) medical interview, 2) patient education—providing brushing guidance to patients, 3) applying moisture-proof rubber dam, 4) impression extraction, and 5) oral surgery skills. In the oral surgery skills station, incision and suturing were evaluated using OPESKIN (realistic skin for surgical training). Nine faculty from the dental school and nine individuals from the Common Achievement Test Organization (CATO) served as evaluators. A briefing session employing an evaluation manual was held for evaluators regarding the overall structure of the OSCE and the content of the five stations.
To prepare for the medical interview session, a special briefing was conducted for all evaluators. Evaluators then observed and evaluated medical interviews in which dental residents interviewed SP instructors. The evaluators’ evaluations were compared and discussed in an effort to increase standardization. Figure 2 presents the general and specific objectives for the medical interview station of the OSCE. Evaluators were carefully oriented to these objectives, which are based on the "Model Core Curriculum for Undergraduate Dental Education" produced by CATO.10
Fifth-year dental students (N = 106; forty-seven males, fifty-nine females) participated in the OSCE.
In addition to the assignment and instructions for the medical interview station presented in Figure 3, modified patient disease history (Figure 4) and writing materials were provided at each interview desk. The students were allowed five minutes to interview the SP, including one minute to read the station instructions. Upon completion of each interview, a feedback session continued for two minutes. Students received ninety seconds of feedback from the evaluators followed by thirty seconds of feedback from the SP. Veteran SP instructors sent by the Tokyo Working Group for Simulated Patients and new SP instructors trained by our dental school served as the simulated patients during the medical interview station of the OSCE and evaluated each student as described in the next section.
Based on general instructional objectives and specific behavioral objectives, an evaluation sheet and an evaluation manual were produced for the faculty evaluators who observed the medical interview stations of the OSCE. Evaluation items appeared in the form of a checklist, which adhered to that of evaluation manuals. To facilitate the evaluation and to reduce variability among evaluators, key words functioning as evaluation standards were added to the evaluation sheet. The evaluation checklist for the medical interview used by the faculty evaluators consisted of seventeen domains of performance. Faculty evaluators rated the students’ performance using a three-point scale (good=2, fair=1, poor=0) for five items and with a two-point scale (0 or 1) for twelve items. Faculty evaluators also provided a summary (overall) evaluation score that ranged from 0 (low) to 5 (high). Thus, the maximum possible total score was 27, since there were twelve items rated on a 0–1 scale, five items rated on a 0–2 scale, and the summary evaluation was rated on a 0–5 scale. Utilizing this evaluation form, two faculty members individually evaluated each student, and an average was calculated based on these two sets of scores. For the evaluation sheet used by the SPs, 20 was the maximum possible total score. The scores were converted to percentages, which served as the final score for the test subjects.
Following completion of the OSCE, a comparison study was conducted among the evaluations completed by the two faculty evaluators, four veteran SPs (experience >5 yrs) from the Tokyo Working Group for Simulated Patients (Tokyo SPs), and seven SPs trained at our school in the three-day program described above. In addition, all of the evaluators for this OSCE completed a survey that requested their feedback on the experience. SPSS ver. 11.0J for Windows was used for statistical analysis.
|
Results
|
|---|
For all students at the medical interview station, the converted average score by faculty evaluators was 67.09 ± 11.60. The highest and the lowest scores were 92.60 and 42.59, respectively. The converted average score by the SPs was 56.89 ± 20.79; the highest and the lowest scores by the SPs were 100 and 25, respectively. Average evaluation scores of the Tokyo SPs, who evaluated forty students, and the in-school SPs (sixty-six students) were 52.13 ± 23.01 and 59.77 ± 18.92 respectively (Table 1). No significant difference in students’ average evaluation score was observed between these two groups. Score difference between the two faculty evaluators at the medical interview station was 6.71 ± 5.39. The score differences between faculty evaluators at each of the five OSCE stations appears in Table 2. The between-rater score differences were the smallest for the medical interview station. For example, the between-evaluator score differences were significantly lower for the medical interview station in comparison to the oral surgery technique station by ANOVA analysis (p<0.05). Nine evaluators completed the questionnaires to provide feedback about the medical interview station. Six said the station was "easy to evaluate," and three said it was "reasonably easy to evaluate." No evaluator responded "difficult to evaluate."
High correlation was observed between scores of the faculty evaluators and those of both SP instructors (Figures 5 and 6), according to Spearman significance testing. Our results, which were consistent with those of Kopp and Johnson,6 indicated a higher concordance rate between veteran SP instructors and faculty evaluators (correlation coefficients r=0.77 vs. 0.73); however, the difference was very slight. Therefore, these results suggest that our training program for SP instructors was an effective protocol, particularly with respect to between evaluator reliability and the efficiency of training (three days).
|
Discussion
|
|---|
The Objective Structured Clinical Examination (OSCE) is thought to be an effective method for objective assessment of clinical skills and behaviors that are difficult to measure via written examinations.1,11 Utilization of the OSCE as an examination component requires evaluation standardization among evaluators and SP instructors. To achieve calibration among evaluators, well-designed evaluation forms need to be used and training must be provided.12 Several essential issues need to be addressed when conducting an OSCE including: the tasks and skills to be evaluated must be significant components of the professional responsibilities of health care providers; the tasks to be completed by students at each station must be consistent with the overall goals of the OSCE; and the evaluation must indicate the achievement level (expected standard of performance) of the targeted goals. In this case, the assignment "conduct a medical interview" is an essential component of a model core curriculum. The next considerations in planning an OSCE are logistical factors including where to conduct the examination, cost, and human resources. In the current investigation, volunteers were recruited from staff and students to serve as simulated patients. Using simulated patients facilitated our ability to evaluate the students’ communication skills and behavior toward patients.
A second essential issue with respect to conducting an OSCE is the evaluation procedure itself. Two evaluators per student were employed. Perhaps as a result of using a well-designed evaluation checklist, concordance rate between the two evaluators was significantly higher for the medical interview station than for the oral surgery technique station, and no significant differences were evident in comparison with other technical and interview assignment methods (Table 2
). Training of the evaluators in this study was approximately one hour in duration. However, it is our opinion that concordance rates among evaluators increased due to use of videotapes during training and because the evaluators were able to discuss their ratings during the SP practice.
Whether evaluation in medical interviews retains objectivity and repeatability has been the topic of debate.13 However, the high concordance rate indicated objectivity and repeatability. Moreover, high correlations between evaluations by SPs and evaluators indicated adequate validity. Nevertheless, the appearance of large disparity in some test subjects between evaluators and SPs suggests the continued existence of unresolved complications. Several cases, which received the highest score (5) from evaluators, also received the lowest score (1) from SPs. In the most extreme example, the score of evaluators was 81.48, whereas the score of SP instructors was 35, a differential of 45 points. Research indicates that evaluation discrepancy between evaluators and SP instructors also depends on attitude, aptitude, and experience.6 Therefore, our new in-school SP instructors will be required to gain more experience prior to formal implementation of OSCE.
Students who received high scores from evaluators asked the SP instructors numerous questions over the limited five-minute period. Students who received lower scores tended to bypass important items such as the chief complaint, patient demands, and main concern, opting instead to ask questions about cost of treatment and extremely detailed patient background. Assessment of the students’ question-asking strategy helped to identify priority questions and the sequence of questioning that should be reflected in the evaluation checklist used to observe and critique the medical interview process. For example, here are the interview strategies used by two students in the OSCE. Student A started the interview by asking the SP to describe her chief complaint in detail, then requested the patient to describe her expectations; finally, the student asked the patient to discuss her personal and medical background. During this OSCE station, the SP was comfortable with the process. In contrast, Student B started the interview by asking the patient about her background, followed by questions about treatment costs, and then finally asked the patient to briefly describe his chief complaint. The patient who encountered student B did not give high marks to the interview process. The evaluation checklist that we used for the OSCE did not allow evaluators to distinguish between the approaches of students A and B, but we now have modified our evaluation form to indicate the desired interview sequence.
Moreover, unclear questions, which were difficult to answer, unasked questions, and dominating advice, such as causes of dental caries and tooth-cleaning methods, were unpopular among the SPs. Furthermore, in some instances, question items on evaluation sheets were completed by repeating unclear questions (saying, for example, "anything else?"), leading ultimately to higher scores for the evaluation. To resolve these difficulties, additional improvement of the evaluation sheet and summary evaluation manuals, enhancement of evaluator enthusiasm, and enhancement of SP experience to clarify standardization of the evaluation method among SP instructors are necessary. High correlation between evaluator and SP evaluations is desirable in patient-centered medical settings. Currently, some researchers contend that evaluation by SP instructors is too subjective. They recommend that this component should be excluded from the evaluation,12 as it has not been used effectively.
The future expectation for our in-school SP instructors is extremely high. We firmly believe that, in addition to administration of the OSCE, our trained in-school SPs will contribute to faculty development at Showa University via implementation of student education related to patient data-gathering,8 preparation and presentation of dental treatment plans,9 geriatric dentistry,5 brushing instruction, improvement of interpersonal skills, and supervision of medication usage in dental pharmacology.
|
Conclusion
|
|---|
"Medical interview," an assignment in the OSCE, which can assess basic technical skills of dentists, requires training of SP instructors. In 2003, we successfully trained fifteen new SPs. Our training program for SP instructor affords the advantages of reliability and efficiency.
|
Acknowledgments
|
|---|
We thank Dr. T. Saitoh, Dr. Y. Shibata, and Dr. Y. Haseba for technical support and Miss E. Imamura for secretarial assistance. We wish to thank the SP volunteers, all staff and faculty members of the School of Dentistry, Showa University, for invaluable aid in the conduct of the OSCE.
|
REFERENCES
|
|---|
- Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J 1975;1:447–51.
- Norman GR, Tugwell P, Feightner JW. A comparison of resident performance on real and simulated patients. J Med Educ 1982;57:708–15.[Medline]
- Johnson JA, Kopp KC, Williams RG. Standardized patients for the assessment of dental students’ clinical skills. J Dent Educ 1990;54:331–3.[Medline]
- Johnson LA. Patient simulation consortium. J Dent Educ 1990;54:595.[Medline]
- Kay CJ, Johnson JA, Kopp KC. Standardized patients for teaching geriatric dentistry. Spec Care Dent 1994;14:229–32.
- Kopp KC, Johnson JA. Checklist agreement between standardized patients and faculty. J Dent Educ 1995;59:824–9.[Abstract]
- Johnson JA, Kopp KC. Effectiveness of standardized patient instruction. J Dent Educ 1996;60:262–6.[Abstract]
- Broder HL, Feldman CA, Saporito RA, Stilwell N. Implementing and evaluating a patient instructor program. Dent Educ 1996;60:755–62.
- Logan HL, Muller PJ, Edwards Y, Jakobsen JR. Using standardized patients to assess presentation of a dental treatment plan. Dent Educ 1999;63:729–37.
- A model core curriculum for undergraduate dental education (in Japanese). Tokyo: Common Achievement Test Organization, 2001:1–45.
- Stillman PL, Swanson DB, Smee S, Stillman AE, Ebert TH, Emmel VS, et al. Assessing clinical skills of residents with standardized patients. Ann Intern Med 1986;105;762–71.
- Murakami J, Takenaka H, Horikoshi A, Sawada U, Sato M, Ohi H, et al. Problems in the evaluation of medical interviewing skills with objective structured clinical examination: how can reasonable objectivity be ensured? Med Educ (Tokyo) 2001;32:231–7.
- Ban N. Objective structured clinical examination (OSCE): a new development in assessing clinical competence. Med Educ (Tokyo) 1995;26:157–63.
This article has been cited by other articles:
|
|
|
|
 
F. W. Licari and D. W. Chambers
Some Paradoxes in Competency-Based Dental Education
J Dent Educ.,
January 1, 2008;
72(1):
8 - 18.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
T. Komabayashi and W. F. Bird
Comparison of Written Examinations Required for Dental Licensure in Japan and the United States: Contents, Cognitive Levels, and Cultural Implications
J Dent Educ.,
August 1, 2005;
69(8):
930 - 936.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Methods
