| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Critical Issues in Dental Education |
Dr. Amyot is Professor and Director; Prof. Bray is Professor and Director, Graduate and Degree Completion; Dr. Branson is Associate Professor; Prof. Holt is Associate Professor; Prof. Keselyak is Assistant Professor; Prof. Mitchell is Assistant Professor, all in the Division of Dental Hygiene, and Dr. Williams is Professor, Department of Dental Public Health and Behavioral Sciences—all at the University of Missouri-Kansas City School of Dentistry. Direct correspondence and requests for reprints to Dr. Cynthia C. Gadbury-Amyot, Professor and Director, Division of Dental Hygiene, School of Dentistry, University of Missouri-Kansas City, 650 E. 25th Street, Kansas City, MO 64108; 816-235-2050 phone; 816-235-2157 fax; amyotc{at}umkc.edu.
Key words: competency, validity, portfolio assessment of student competency
Submitted for publication 09/15/04; accepted 12/17/04
 |
Abstract |
|---|
 Top Abstract MethodologyResultsDiscussionConclusionReferences |
|---|
Competency has been defined as the skills, understanding, and professional values of an individual ready to begin practicing independently.9–10 A one-shot initial clinical licensure examination is likely to be insufficient to evaluate the complexity of competency as it is defined above. It is more reasonable to assume that true competency can best be determined through longitudinal data involving multiple patient experiences and encounters with multiple evaluators providing assessment. The literature suggests that the hallmark characteristic of a competent individual is the ability to accurately assess his or her own competence.9,11 Clinical licensure examinations do not require the candidate to engage in meaningful self-assessment, nor do they provide multiple experiences by which evaluators can assess the student’s competency across different contexts. These issues suggest the inappropriateness of these examinations as a valid measure of a new clinician’s competency.
Problems related to the current one-shot testing for initial clinical dental and dental hygiene licensure are numerous and have spawned extensive discussion about their validity in the educational community. Inconsistencies between student performance at accredited schools and performance on clinical licensure examinations are one source of controversy.1,4–5,12–14 In a study published over two decades ago, results demonstrated a lack of concordance between dental students’ performance in dental school compared to performance on the North East Regional Board of Dental Examiners (NERB) clinical licensure exam.5 The study illustrated how nearly one-third of the candidates who failed the examination were ranked in the upper third of their dental class, while candidates ranked in the bottom 10 percent of their class all passed the NERB. Similar results were reported a decade later in a study conducted in Texas that examined pass rates for dental students on the Texas State Dental Board clinical licensing examination.14 The investigators found a very weak, negative (r = –.1487) relationship between class rank and pass rates. A 2001 study, undertaken to compare dental school performance with performance on NERB, demonstrated a lack of concordance between class rank and passing or failing the clinical licensure examination, which led the investigators to question the validity and reliability of clinical licensure examinations for determining entry-level competency of dental practitioners.1
Patrick conducted a national study of dental hygiene program directors using a Delphi process to gain consensus on the best way to determine clinical competence prior to issuing a dental hygiene license.7 Results showed that directors believed that dental hygiene clinical competence would be best determined through ongoing evaluations in an accredited dental hygiene program. Patrick’s study demonstrates that dental hygiene educators question the meaningfulness and usefulness (validity) of our current licensing system where a one-shot clinical examination is required for obtaining a license to practice dental hygiene. These studies collectively provide evidence that both dental and dental hygiene educators question the validity of one-shot clinical licensure examinations.
The issue of using live patients for a qualifying examination has been identified as a problem related to our current clinical licensure examinations and has raised questions of ethical treatment of humans. Reasons against using human subjects range from the impossibility of standardizing the level of treatment difficulty across student candidates, to patient discomfort with length of examination procedures and lack of follow-up, along with the problem of ethical lapses that occur when using live patients.3,15 Feil et al.3 conducted a randomized national survey of 1,000 general dentists and found that over half of the respondents knew with certainty of at least one instance of an ethical lapse related to the use of live patients for clinical licensure exams. Approximately 24 percent reported no follow-up care was sought for a patient even though it was needed; 32.5 percent reported that unnecessary radiographs were taken; 14 percent reported knowing of a classmate who coerced a patient into an inappropriate treatment choice; and 19.3 percent reported knowledge that a patient had been treated prematurely or overaggressively. Furthermore, the majority of the dentist respondents felt the exam was not a valid assessment of their clinical skills and made negative comments about the examination. Summarily, not only do educators question the validity and reliability of current dental and dental hygiene licensure examinations, but the professionals themselves express concern. Ethical and moral issues surrounding the use of live patients have also been reported by Hasegawa.2 Hasegawa argues that the use of live patients for purposes of licensure equates to seeing and using the patient as a means to an end, which goes against ethical and moral principles taught in dental and dental hygiene educational curriculums and as outlined in the ADA Code of Ethics and the ADHA Professional Code of Conduct.16–17 Ethical and moral issues surrounding the use of live patients are many, which may explain why no other health profession requires invasive and/or irreversible treatment as part of demonstrating competency for the purpose of licensure.
Other problems surrounding the current clinical licensure examinations include issues such as increased stress on the student as a result of retaking the examination, the increased expense to the student as a result of repeating the examination, and the need to identify and use additional human subjects in subsequent examinations. Many student candidates are required to travel great distances in order to retake the examination. This involves not only the expense of travel (ground or air), but also food and hotel costs for both students and their patients. For all of the reasons cited, it seems that more attention needs to be focused on the validity evidence for these high-stake examinations.
The Standards for Educational and Psychological Testing define validity as "the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests" (p. 9).18 Cook and Campbell discuss threats to construct validity in terms of mono operation and mono method bias.19 These two threats fall under a larger umbrella of construct underrepresentation. Specifically, Cook and Campbell state that single measures of a construct (i.e., dental/dental hygiene competency) threaten the investigator’s ability to establish validity due to underrepresentation and the chance for irrelevancies in interpreting the results. Failure on the one-shot clinical licensure examination, for example, could be due to poor patient selection as a result of a limited patient pool that meets the very specific criteria outlined by the clinical licensure examination testing services, a patient who is not cooperative during the exam, or the fact that a patient failed to show up for the examination. Clearly these factors can be interpreted as irrelevancies when it comes to measuring and interpreting dental and dental hygiene competency. To increase validity, Cook and Campbell suggest that the constructs, in this instance dental/dental hygiene competency, should be assessed by multiple measures in order to provide multiple perspectives (i.e., triangulation) about the construct. These measures should also capture the construct in different ways, thus addressing the threat of mono method bias. Most importantly, strong evidence should be available for the measures that demonstrate their validity and reliability.
One of those measures is the National Board examinations, which routinely report validity and reliability data and have demonstrated reliabilities above .90.20–22 Another defensible measure is grade point average. Several studies have demonstrated that GPA serves as a good predictor of student success in both dental and dental hygiene educational programs.20,23–25 In the midst of this ongoing controversy surrounding the validity and reliability of clinical licensure examinations, educators and researchers have suggested the use of portfolio assessment as a more appropriate measure of student competency than one-shot clinical licensure examinations.6,10–11,26 Educational measurement experts concur that the use of portfolios is appropriate for measuring student competency.27–28 Preliminary evidence would suggest that portfolios could serve as a valid and reliable measure of dental hygiene competency.6 If truly the intent of these high-stakes examinations is to protect the public from incompetent practitioners, then a strong argument for performance assessment in the form of portfolios that document evidence of student competency over time, based on multiple evaluations by multiple evaluators, could be made.
Therefore, the purpose of this study was to examine the predictive validity of traditional and nontraditional dental hygiene competency assessment measures on one-shot clinical licensure examinations in a baccalaureate dental hygiene program.
|
Methodology |
|---|
|
TopAbstractMethodologyResultsDiscussionConclusionReferences |
|---|
For the traditional measures, overall GPA is figured on the traditional 0–4.0 scale. Clinic GPA was calculated on a 0–2 scoring system where 0=No Credit, 1=Credit, and 2=A. Clinic grades for four consecutive semesters in the program were averaged to come up with the student’s overall clinic GPA. The National Board Dental Hygiene Examination (NBDHE) is a comprehensive examination consisting of approximately 350 multiple-choice test items and two components. The first component consists of 200 discipline-based items addressing the areas of the scientific basis for dental hygiene practice, provision of clinical dental hygiene services, and community health activities. The second component consists of 150 case-based times. The Central Regional Dental Testing Service (CRDTS) clinical licensure examination utilizes a criterion-based grading system to assess the student’s ability to perform procedures as follows: Patient Assessment, Scaling, Plaque/Stain, Radiographic Technique, and Treatment Standards. The examination has a maximum score of 100, with member state licensing boards requiring a 75 to be eligible for licensure.
The nontraditional measure is portfolio assessment. Portfolio assessment of student competency and the scoring rubric used in evaluation of student portfolios have been published and previously described in detail by the primary investigator.6,29 In short, each student develops a portfolio based on the University of Missouri-Kansas City Division of Dental Hygiene program competencies. Students must provide a reflective piece for each of the competencies, with accompanying evidence to support and document their claims of competency. Included in the evidence is longitudinal data collected over the student’s educational program, involving multiple patient experiences and multiple faculty raters. Examples include patient cases, competency testing results, community research and presentations, etc. The responsibility lies with the student to assess his or her own competency based on the evidence provided. The scoring rubric used to evaluate the portfolios contains a total of seven traits, which are evaluated in each portfolio based on the reflection and evidence provided by the student. These traits include evidence of growth, evidence of attainment of all program competencies, evidence of self-evaluation, evidence of lifelong learning, evidence of organizational skills, evidence of creativity in the development of the portfolio, and finally, evidence of communication skills. A four-point Likert scale (1= no evidence of the trait, 4=complete evidence of the trait) is utilized with possible scores ranging from a maximum of 28 to a minimum of 7. The validity and reliability of this rubric was explored through the examination of the degree to which evidence and theory support the interpretations of portfolio assessment within Messick’s framework for construct validity, and a detailed analysis can be found in a previous publication.6 A sound argument was made about the validity and reliability of this institution’s portfolio assessment based on both theoretical and empirical evidence.
This study examined the extent to which the predictor variables GPA, Clinic GPA, NBDHE scores, and Portfolios scores predict CRDTS (criterion variable) results. A nonexperimental research design was used to examine the validity of CRDTS clinical licensure examinations. The study population consisted of seventy-four dental hygiene graduates of the University of Missouri-Kansas City School of Dentistry Classes of 2001–04 and only included those graduates who had a minimum of three examiners evaluate their portfolios. Previous research has illustrated that the use of three examiners per portfolio produces reliable and defensible assessment results.6 This study was approved by the University of Missouri-Kansas City Social Sciences Institutional Review Board.
Descriptive data were used to summarize demographic characteristics of the sample. Predictor variables were subjected to principal components analysis to examine the underlying factor structure. Items were considered to be related to the underlying factor structure if factor loadings were >0.6. Using SPSS 12.1, factor scores were computed using a multiple regression approach. Factors were then used to predict one-shot clinical licensure examination scores using linear regression.
|
Results |
|---|
|
TopAbstractMethodologyResultsDiscussionConclusionReferences |
|---|
. All participants were female (n=74) and ranged in age from twenty-one to forty-eight years with a median age of twenty-five (semi-interquartile range= 4.38). Grade point averages at graduation ranged from 2.26 to 4.0 with a mean of 3.36 (SD=.43). Approximately 13.6 percent of the participants were of an ethnicity other than Caucasian.
|
. Correlations between overall grade point average (GPA), National Board Dental Hygiene Examination (NBDHE) scores, and Portfolio scores were moderate and positive (r =.41 to .63); however, the relationship between clinical GPA and overall GPA, NBDHE, and Portfolios was considerably lower (r=–.01 to .23). The relationship between all predictor variables with the criterion variable (CRDTS) ranged from r=.19 (Clinical GPA) to r=.31 (Overall GPA). Means and standard deviations of the predictors and criterion variables are shown in Table 3. Because the predictor variables were correlated, an exploratory factor analysis approach was used to examine the underlying latent structure of these competency measures.
|
|
displays the factor loadings for each of the competency measures. The first factor, Dental Hygiene Cognition, accounted for 51 percent of item variance; the second factor, Dental Hygiene Clinical Performance, accounted for an additional 27 percent of item variance. Factor scores were subsequently used in a linear predictive model, using stepwise regression, to assess the shared and unique contribution of factors to the one-shot clinical licensure examination score.
|
. These results demonstrate a statistically significant effect of Dental Hygiene Cognition on one-shot clinical licensure scores (R2 Change=.121, p=.003). A very small and nonsignificant increase in explained variance was noted for Dental Hygiene Clinical Performance (R2 Change=.018, p=.237). The full model explained 13.9 percent of variance in the outcome measure.
|
|
Discussion |
|---|
|
TopAbstractMethodologyResultsDiscussionConclusionReferences |
|---|
As previously stated, the Standards for Educational and Psychological Testing define validity as "the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests."18 In the case of the one-shot clinical licensure examination (CRDTS), the interpretation to be made is whether the test ensures a competent and safe dental hygienist who is ready to practice dental hygiene at an entry level. Educational measurement experts contend that central to most validation efforts is being able to demonstrate that persons who score high on a presumed indicator (CRDTS) of the construct (dental hygiene competency) should score high on other presumed indicators (NBDHE, Overall GPA, and Portfolios) of the construct being measured.30 The correlations in this study between the criterion variable (CRDTS) and the predictor variables of NBDHE, Clinic GPA, and Portfolios would all be considered small, according to the extensive work done by Cohen where the strength of correlations are defined as large being >.50, moderate .30 to .10, and small or trivial as <.10.31 The correlation between CRDTS and Overall GPA barely meets the criteria as a moderate correlation. This lack of congruence between the predictor variables and the one-shot clinical licensure examination is alarming considering that these clinical examinations are utilized by legally authorized state agencies for making decisions about granting licenses to practice.
Educational measurement experts would argue that a test’s validity is not established or refuted with one study; rather, validity is determined through an accumulation of data.19,30 In such a high-stakes testing environment, it is of great concern that the current licensing structure is unable to provide defensible validity evidence. While educators have questioned the validity of these one-shot clinical licensure examinations and conducted studies that have failed to support a validity argument, there seems to be minimal examination at a national level of how we might develop a more valid system for licensing both dental and dental hygiene students. Chambers has suggested in the case of dentistry a partnership between state licensing boards and schools where dental schools are held responsible for providing valid and reliable evidence of student competency and boards responsible for making their licensure decision based on that evidence.26
We would suggest that portfolio assessment of student competency is an appropriate model of assessment to use in determining qualifications for licensure to practice within the competency-based educational framework that we currently teach. The use of portfolios allows students to present multiple representations of their work evaluated by multiple faculty, with examples of multiple measures such as case studies and competency examinations and including in their portfolios those measures that have been previously validated or determined good predictors of success, i.e., competency (GPA, NBDHE). Not only must students produce the appropriate evidence in their portfolios; they also must self-assess their competency. This measure of competency is in alignment with the competency-based literature. Development and assessment of student portfolios are time-intensive endeavors for both students and faculty. However, given the ethical and validity issues surrounding one-shot clinical licensure examination, dental and dental hygiene educators and students may be able to justify the time involved in this nontraditional assessment measure.
On the same basis that validity can only be determined through an accumulation of data, there is still much to be done in terms of seriously looking at portfolio assessment of student competency as a valid and reliable measure for determining student competency. Few studies have been conducted to date on the validity and reliability of portfolio assessment of student competency. Limitations of the current study should be considered in the interpretation of the study results. The most obvious limitation relates to the setting in which these data were collected. This study was conducted in four consecutive dental hygiene classes at a single institution over a period of four years. Additionally, the one-shot clinical licensure examination (CRDTS) represents only one clinical licensure examination conducted across the United States. These factors limit the generalization to other dental hygiene programs and other clinical licensure examinations in the United States. Future studies should include further exploration of the validity of one-shot clinical licensure examination to predict dental hygiene student competency in regions and programs throughout the United States.
|
Conclusion |
|---|
|
TopAbstractMethodologyResultsDiscussionConclusionReferences |
|---|
|
Acknowledgments |
|---|
|
REFERENCES |
|---|
|
TopAbstractMethodologyResultsDiscussionConclusionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  M. E. Donaldson, C. C. Gadbury-Amyot, S. S. Khajotia, A. Nattestad, N. S. Norton, L. A. Zubiaurre, and S. P. Turner Dental Education in a Flat World: Advocating for Increased Global Collaboration and Standardization J Dent Educ., April 1, 2008; 72(4): 408 - 421. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. E. DeCastro, D. Bolger, and C. A. Feldman Clinical Competence of Graduates of Community-Based and Traditional Curricula J Dent Educ., December 1, 2005; 69(12): 1324 - 1331. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
