| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Educational Methodologies |
Key words: international dental students, National Boards Part I and II, dental examinations, dexterity tests, academic performance, clinical competency
Submitted for publication 09/10/04; accepted 11/23/04
 |
Abstract |
|---|
 Top Abstract MethodResultsDiscussionConclusionReferences |
|---|
The effectiveness of several selection measures for graduate school applicants has been previously evaluated including undergraduate and other predental school training,1,2 standardized tests including the Dental Admission Test, and admission interviews.3 The data confirm that no single measure by itself is a sufficient predictor of academic performance and clinical competency. Furthermore, and as expected, the data confirm that multivariate combinations provide greater predictive certainty. One combination included standardized measures (Graduate Record Examination scores), predental education (undergraduate GPA), admission interview scores, personality measures, and vocational suitability scores.2
Significant empirical evidence supports the inclusion of the identified predictors of academic proficiency4 with higher scores, at application, on standardized measures, such as the GRE, associated with eventual academic performance. This data has proved to be an important tool in the evaluation of U.S. dental school applicants. Overall, incoming students presenting with higher academic standard scores are more likely to successfully complete their course at dental school. Higher scores on formal coursework and clinical practice measure this success.
While there is little disagreement that standardized measures predict academic performance at dental school, it is acknowledged that dental competency involves more than achievement based on core knowledge. The profession of dentistry requires clinical practice competency as well, identified, in part, by dexterity and other fine-motor skills. This need for motor skill assessment has resulted in the examination of suitable measures,5 including standardized and/or unstandardized dexterity tests. The dexterity tests, it is hypothesized, improve the prediction of clinical practice competency.5–7
Research using standardized and/or nonstandardized dexterity measures provided divergent results. Walcott et al.,6 for example, found that higher dexterity scores increased the probability of clinical competency, whereas Simon and Chambers7 and Gansky et al.8 found little added value. Heintze et al.,9 in an examination of predictors of success in dental school, found mixed and limited use of spatial and dexterity abilities. Similarly, Oudshoorn10 found some support for carving dexterity, but limited practical utility predicting psychomotor performance in first-year dentistry. In a review and summary of the available data, Dailey5 found overall support for the inclusion of motor or dexterity tests, concluding they screen out applicants lacking ability to profit from technical instruction. Dailey also acknowledged that dexterity tests, such as chalk carving, are subjective, time-consuming, and messy to administer, thus restricting their value as predictors of clinical competency in dental education. Poor inter-rater reliability in assessing the dexterity tests, he said, added to the critical reviews. The need for additional investigation of the significance of dexterity measures is recognized. However, the available evidence supports combining motor skill measures with academic, language ability, and skill-based measures in evaluating dental competency.
An additional issue requiring clarification is research design. Evaluation of academic performance and clinical competency has generally focused on mainstream dental training programs (that is, programs in which U.S. college-trained students move on to U.S. graduate-level dental schools). Less research has examined specialized training programs focused on international students seeking graduate-level dental education in the United States.11,12 In these instances, the international students complete their undergraduate and professional schooling in their home country, practice as licensed professionals, and then seek graduate training in the United States. In most cases, the international students are assimilated into mainstream programs after completing their courses in dental training outside the United States.
The University of the Pacific (San Francisco) is one example of a specialized program in which students assimilate into the mainstream dental education program. National Board Part I scores, Test of English as a Foreign Language (TOEFL), pre-admission faculty interviews, and the submission of a realistic work sample are combined to aid prediction of academic performance and/or clinical competency.8 With this combination of predictor variables, the program addressed the challenges presented by internationally trained students—alternate grading methods, diversified national academic and clinical standards, and varying levels of English language mastery. Simon et al.11 concluded that these measures, plus the inclusion of TOEFL and preclinical technique bench tests, provided significant predictions of academic performance and clinical competency for international students.
Another example, the International Dental Program (IDP) at Loma Linda University, separates international dental students from the mainstream program. IDP students complete their initial dental training in their home country and then join the Loma Linda University D.D.S. graduate degree program. As for all dental schools serving international students in mainstream or specialized programs, the challenge for the IDP is to accurately evaluate potential students with varying academic and clinical training while maximizing the probability of achieving eventual academic performance and clinical competency required by U.S. standards.
Efficient and effective prediction of performance and competency for international students is defended as good for the student, the school, and the profession.11 As indicated, predicting performance and competency in mainstream or specialized programs has ordinarily focused on the use of standardized measures such as National Board scores, TOEFL, and previous academic performance (e.g., GPA). More recently, standardized and nonstandardized fine-motor measures (e.g., dexterity scores) have been included with generally positive but mixed results.
One further practical issue warrants inclusion. The challenge for all dental programs, especially those directing international students into specialized programs, such as Loma Linda University IDP, with numerous applicants for every open space, is to identify the most effective and efficient combination of selection measures.11 Competition for available open places, along with increased demands on the school’s personnel and financial resources in administering the selection process, mandates development of efficient selection procedures without any loss of accuracy or fairness. In statistical terms, this equates to finding the fewest, most accurate measures and eliminating those measures contributing no new information when predicting academic performance and clinical competency.
Establishing an efficient and effective combination of predictor variables has academic, clinical, and practical implications. One goal of our study was to replicate findings confirming the significance of measures, including National Board Part I and II, TOEFL, faculty interview, and dexterity scores, in predicting academic performance and clinical competency11 in the context of the Loma Linda University IDP program. The second goal was to identify the most efficient and effective combination of predictor variables. Three hypotheses drove the study design:
|
Method |
|---|
|
TopAbstractMethodResultsDiscussionConclusionReferences |
|---|
Applicants submit a packet that contains school transcripts, resumes, recommendations, and National Board Part I and TOEFL scores. Successful applicants ordinarily achieve TOEFL scores in excess of 213 and National Board Part I scores in excess of 81 and more commonly around 83. Applicants meeting these criteria are invited for the second stages of evaluation that are the dexterity tests and faculty interview. The top thirty-two scorers on the dexterity test, ordinarily with a composite dexterity score centering around 3.0 or over on a 5.0 scale, are invited for the final interview.
Teams of dental faculty and university personnel interview each of the top applicants. Interview issues include attitudes to schooling, goals, reasons for pursuing the IDP program, study practices, personal values, professional ethics, clinical judgment, attitudes to supervision, and clinical experiences. The stated goal of the interview is to qualitatively assess the match between the applicant and the IDP program. After the thirty-minute interview, each interviewer provides a score out of 10, with the interview team average entered as the applicant’s score. Clinical faculty then evaluate these scores, together with all the standardized scores. Successful applicants receive notification of final acceptance within the month.
Materials and Data Analysis
The data for this study were stored in archival records at the Loma Linda University School of Dentistry. All identifiers, including name, addresses, and university identification numbers, were separated from the raw data before extraction for analysis. The data included country of origin; gender; National Board Part I (NB I), National Board Part II (NB II), Test of English as a Foreign Language (TOEFL), and dexterity scores; admission interview scores; final academic grade; and final clinical grade.
National Board Part I is a prerequisite for all IDP applicants. Part I, according to the American Dental Association,13 is ordinarily taken after two years of dental school. For IDP, Part I is completed before application to the graduate program. Part I consists of four examinations on the basic biomedical sciences including anatomic sciences, biochemistry-physiology, microbiology-pathology, and dental anatomy and occlusion. Each of the four Part I examinations consists of approximately 100 multiple-choice test items.10 IDP at Loma Linda University requires a Part I score with a minimum of 81. However, because only the top 100 applicants on Part I are invited to the next stage of the admission process, the score is usually closer to 83.
For students in the regular predoctoral D.D.S. program, National Board Part II is completed before graduation, usually in December of the senior year. However, approximately 50 percent of IDP applicants completed Part II before entry into the program. Over the last ten years, more and more students completed Part II before entry, and Part II has now been included as a required element for application.
National Board Part II consists of a comprehensive, one-and-a-half-day examination and covers the clinical dental sciences (operative dentistry, pharmacology, endodontics, periodontics, oral and maxillofacial surgery/pain control, prosthodontics, orthodontics, pediatric dentistry and oral diagnosis, oral pathology/dental radiology, and patient management, including behavioral science, dental public health, and occupational safety). Approximately one fifth of the examination consists of test items based on patient cases. Part II consists of 500 test items: the discipline-based component (Component A) has 400 items in a multiple choice format, and the case-based component (Component B) has 100 multiple choice questions based on eight to ten case problems. Thirty percent of the Part II test items address basic science subject matter.13
The Test of English as a Foreign Language (TOEFL) is a standardized test of English language proficiency. It measures the ability of non-native speakers of English to understand North American English as it is used in university and college settings.14 TOEFL is used by over 4,300 two- and four-year colleges and universities, professional schools, and sponsoring institutions with norms available for nearly every language. TOEFL scores of 213 and above on the written form are required for application to IDP.
Loma Linda University IDP faculty developed the dexterity measures, and each year, different tests are drawn from a test pool. IDP faculty hypothesize that dexterity is an innate skill and prefer applicants do not train to the test. Examples include fashioning an article from a plaster block, carving on a sheet of plastic using an angle former, fabricating a model structure, and modeling out of clay. Each of the four tasks is allocated ninety minutes (total of six hours) with IDP faculty supervising and scoring the results. The assumption is that the tests provide a measure of fine motor skill, spatial perception, and hand-eye coordination. Final scores represent a composite from the four separate elements.
Clinical and administrative faculty from the School of Dentistry conduct face-to-face interviews for the final group of applicants. Business personnel, behavioral health specialists, educators, community leaders, clergy, and other university personnel join the faculty, and teams of five allocate thirty minutes to each applicant. Pre-session discussions by the interviewing teams help provide agreement on the interview content and process. The process, it is hypothesized, provides valuable insights into the applicant’s professional goals, values, attitudes, and a measure of "fit" with the school. The interview also allows the faculty and guests to draw on their subjective knowledge and experience to assess each student’s likely success in graduate school. The faculty interview is scored on a ten-point scale.
Two outcome measures—the Final Academic Grade and Final Clinical Grade—are the dependent variables for this study. The Final Academic Grade is a composite GPA derived from the formal coursework of the program’s two years. The Final Academic Grade is measured on the A through F scale, with A assigned a 4 for the grade-point average scores on this core coursework including tests, exams, formal papers, and research projects. The Final Clinical Grade is a composite score awarded for clinical work with patients at the dental clinic. For each quarter of the program, the student’s clinical work is assessed for quantity of product, quality of product, and faculty evaluation of competency. The faculty assess these required procedures in a series of quarterly competency tests. The Final Clinical Grade reflects the totaling of these competency evaluations and is scored on a 100-point scale. The two dependent variables are not independent of each other with 30 percent of the Final Clinical Grade included as part of the Final Academic Grade.
Data gathered during the application process was archived by IDP at the Loma Linda University School of Dentistry. The Statistical Package for the Social Sciences (SPSS) was used to analyze the data. SPSS is a tightly integrated, full-featured statistical analysis product and is suitable for data collecting, data access, data management and preparation, analysis, and reporting.15
|
Results |
|---|
|
TopAbstractMethodResultsDiscussionConclusionReferences |
|---|
Further examination revealed that, for statistical analysis, pairwise deletion of missing data points reduced the cases from 171 to approximately 100. The pairwise method of deletion in correlation matrices means that a correlation is calculated between each pair of variables from only those cases that have valid data on those two variables. To minimize the loss of cases, a mean substitution is acceptable if the missing data is less than 10 percent and randomly distributed. Because the data points in this study were randomly distributed across cases, the mean replacement method for dealing with missing data was used for Dexterity, Final Clinical Grade, and Interview with missing data points of 3 percent, 5 percent, and 8 percent respectively.
More significantly, the TOEFL and National Board Part II variables revealed forty-two (24.5 percent) and seventy-one (41.5 percent) missing data points. This amount of missing data is explained by the fact that LLU had not always insisted on TOEFL and or National Boards Part II for application to IDP. The decision to proceed with a replacement method was appropriate as the missing data points randomly occurred across the data set. Selection of the more statistically sophisticated use of regression equations minimized any reduction in variability and lessening of the predictor’s impact. Using TOEFL as the criterion variable and National Board Part I (r = .340, 
= .000) and Interview (r = .380, = .000) as the predictors, a regression equation was created for TOEFL. For the National Board Part II variable, a regression equation was created using National Board Part II as the criterion and National Board Part I (r = .656, = .000) as the predictor. The newly created predicted values were inserted to replace the missing values on both variables. The original and inserted (where applicable) variable means and standard deviations are listed in Table 1
. The mean and regression substitutions are included in brackets. A final examination indicated the data were within the normal range for skew and kurtosis (±1.0).
|
.452 on Final Academic Grade. Similarly, an independent samples t test demonstrated there was no significant difference between males (M = 85.95, SD = 7.14) and females (M = 86.74, SD = 7.47), t (160) = .688, = .494 on Final Clinical Grade.
Analysis of National Board Part I scores revealed there was no significant difference between males (M = 86.21, SD = 5.61) and females (M = 86.70, SD = 5.40), t (169) = .581, = .562. National Board Part II scores also indicated no significant difference between males (M = 89.92, SD = 5.31) and females (M = 82.88, SD = 5.08), t (98) = –.046, = .963.
Examination of countries or geographical areas of origin revealed IDP students came from the Asia-Pacific (7.6 percent), China (9.9 percent), Eastern Europe (10.5 percent), Latin and South America (9.9 percent), India (15.2 percent), Middle East (10.5 percent), Philippines (9.4 percent), Taiwan (21.6 percent), and Western Europe (5.3 percent).
Correlation
Previous research suggested that significant correlation would exist between the predictor variables. Table 2 lists the correlation between the predictor variables and the Final Academic Grade and the Final Clinical Grade. Correlations between the two outcome variables are included in the table, but not considered in the analysis, as they are not independent of each other.
|
). National Board scores Parts I and II, TOEFL, and Dexterity associated with higher Final Academic Grade scores. Significant association also existed between National Board Part II and Dexterity with Final Clinical Grades. The information gathered from the Faculty Interview had little correlation with eventual academic performance (r = .100, = .193) and clinical competency (r = .067, = .384). The faculty interview correlated with no other predictors except TOEFL (r = .401, = .000). While there was significant association between Dexterity and National Board Parts I and II, there was no association between Dexterity and TOEFL and the faculty interview. With regard to National Board Part I, and of interest to the current study, there was a strong association with National Board Part II, TOEFL, Dexterity, and Final Academic Grade, but no association with Interview and Final Clinical Grade.
Regression
Regression analyses examined the strength and combination of the predictors of Final Academic Grade and Final Clinical Grade. In a first analysis, all the predictor variables, National Board Part I, National Board Part II, TOEFL, Dexterity, and Interview variables were included in the model using the Enter procedure (Table 3) with Final Academic Grade as the criterion variable. The collection of predictor variables explained 24 percent of the variance and was significant as a group ( = .000). This analysis demonstrated National Board Part II (ß; = .371) exerting the greatest influence on Final Academic Grade, followed by Dexterity (ß = .157).
|
) addressed the study’s first hypothesis. This method enters the variables one at a time and ceases to add further variables when they no longer reflect additional significance. This regression explained 22 percent of the variance of Final Academic Grade ( = .023), with National Board Part II (ß = .421) exerting the greatest influence on the criterion variable, followed by Dexterity (ß = .158).
|
= .000), and it was important to scrutinize the interpretability of the results on Final Academic Grade given the significant association. Separate models omitted National Board Part II and National Board Part I in turn. Table 5 provides a summary using the Stepwise method with National Board Part II omitted from the predictor variables. National Board Part I provided the only significant influence (ß = .390) and explained 15 percent of the variance ( = .000). When National Board Part I results were omitted, the results were identical to the first model (Table 4).
|
). The model of predictor variables explained 9 percent of the variance and was significant as a group ( = .007). This result, similar to that with Final Academic Grade, indicated that National Board Part II (ß = .281) exerted the greatest influence on Final Clinical Grade followed by Dexterity (ß = .179).
|
). The model explained 7 percent of the variance of Final Clinical Grade ( = .002) with National Board Part II (ß = .194) having the greatest influence, followed by Dexterity (ß = .156)
|
= .000). Consequently, it was again reasonable to scrutinize the interpretability of the results predicting Final Clinical Grade. Table 8 provides a summary of the results using Stepwise regression with National Board Part II scores removed. Dexterity provided the only significant influence (ß = .183) and explained 3 percent of the variance ( = .017). When National Board Part I scores were omitted, the results were identical to Table 8.
|
) explained 22 percent of the variance of Final Academic Grade ( = .023), with National Board Part II (ß = .421) having the greatest influence on Final Academic Grade, followed by Dexterity (ß = .158). Similarly, with Final Clinical Grade as the criterion (Table 6), the model of predictor variables explained 9 percent of the variance, with National Board Part II (ß = .281) exerting the greatest influence on Final Clinical Grade followed by Dexterity (ß = .179). These results require some qualification. First, the retrospective design of the study with its restricted population sample limits the generalizability of the results. Furthermore, unsuccessful applicants to IDP, who may have joined another program, were not compared with the current participants or evaluated for final academic performance and clinical competency. In addition, the impact of the dexterity and interview measures must be evaluated cautiously as the scores were collected from only those students successfully meeting previous exclusion criteria. The truncated range of scores could reduce the influence of these two variables. Future research should address these design features.
|
Discussion |
|---|
|
TopAbstractMethodResultsDiscussionConclusionReferences |
|---|
The association between academic performance and the admission measures was expected and reassuring. The measures used by the admissions committee at IDP are clearly associated with academic performance as measured by final course grades. Higher scores on National Board scores, English language proficiency, and dexterity scores associated with higher academic performance scores at the end of the course.
The moderate association between National Board Part I and Final Clinical Grade warrants special attention. Knowledge of the basic sciences (National Boards Part I) is clearly associated with formal classroom performance, but only moderately associated with clinical competency for international dental students. This is not to suggest that Part I scores are unimportant. However, for international students—students who have completed dental training in another place—that knowledge of basic sciences says less about how they will perform on clinical measures in a D.D.S. program.
Interestingly, while National Board Part II, National Board Part I, TOEFL, dexterity scores, and final academic performance scores were associated, there was no association with the faculty interview. The absence of association suggests the interview process accessed some elements other than those measured by the other variables. The clear association of the interview with TOEFL suggests the interview process may be assessing the impact of the student’s communication abilities.
It was hypothesized that National Board Part I and TOEFL would be the primary predictors of academic performance, accompanied by National Board Part II and the faculty interview scores. Furthermore, it was hypothesized the primary predictors, National Board Part I and TOEFL, measures of basic scientific knowledge and language ability more commonly associated with academic progress, would emerge as the main predictors. However, the significance of National Board Part II and dexterity scores demonstrated that measures of specific dental knowledge and practical skills associated with technical ability were more important than knowledge of basic sciences (National Board Part I) and English-language ability (TOEFL) when predicting the academic performance of international students.
The lack of significance of National Board Part I may be partially explained in that international students have already demonstrated sufficient knowledge of the basic sciences by graduating from dentistry in their home countries. Similarly, the lack of significance for TOEFL scores suggests international students applying for U.S. graduate dental programs, such as IDP, have already achieved sufficient levels of English language mastery to successfully pass their National Boards. Consequently, the lack of variability in the student’s knowledge of the basic sciences and English language skills limits National Board Part I and TOEFL as significant predictors of academic performance.
While it is an open question whether dental programs, such as IDP, would feel comfortable omitting tried and true standards such as National Board Part I and TOEFL from the admissions process, the current results demonstrate that neither measure adds any additional help to that provided by National Board Part II and dexterity scores in predicting academic performance.
It was hypothesized that National Board Part II would be the primary predictor of clinical competency. However, as with academic performance, the lack of predictive significance of National Board Part I and TOEFL was unexpected. Based on the evidence from the current study, predicting the clinical competency of international students is best accomplished using measures that evaluate real-life dentistry knowledge (National Board Part II) and specific motor skills (Dexterity). The dexterity measure, often questioned by researchers and faculty, appears to add significant weight to the prediction of clinical competency of the international students.
In fact, the significance of dexterity scores predicting both academic performance and clinical competency is at variance with some of the earlier research. It was hypothesized that dexterity skills would predict clinical competency, but not academic performance. However, international students with higher dexterity scores achieved higher grades in clinical competency and academic performance at IDP—thus strongly supporting the inclusion of dexterity measures in the selection process. It is acknowledged, however, that generalizing this finding from the IDP study to U.S.-trained students in mainstream dental programs is a research opportunity for the future.
The lack of significance of the faculty interview raises questions as to its relevance to the admissions process. The subjective assessment of interviewing faculty was not a significant factor for either academic or clinical outcomes. One possible explanation is that the faculty interview, while not associated directly with academic or clinical outcomes, may be associated with other outcomes more indirectly related to the student’s eventual success at dental school. For example, the interview may be a more indirect measure of a student’s language and communication skills. It may also help faculty, based on their experience, assess a student’s ability to handle the pressures of the graduate school experience. Further, even though it may not clearly identify the most successful students, it may serve as a form of assessment of "fit" between the students and the unique values of the school. In all likelihood, the interview could add to the prediction of clinical competency and academic performance if it was standardized. The more subjective process at IDP, while possibly and potentially valuable, likely dilutes any impact the faculty interview has on the academic and clinical outcomes.
|
Conclusion |
|---|
|
TopAbstractMethodResultsDiscussionConclusionReferences |
|---|
|
Footnotes |
|---|
|
REFERENCES |
|---|
|
TopAbstractMethodResultsDiscussionConclusionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  L. E. Itaya, D. W. Chambers, and P. A. King Analyzing the Influence of Admissions Criteria and Cultural Norms on Success in an International Dental Studies Program J Dent Educ., March 1, 2008; 72(3): 317 - 328. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. M. Neumann and R. L. MacNeil Revisiting the National Board Dental Examination J Dent Educ., October 1, 2007; 71(10): 1281 - 1292. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. Kingsley, J. Sewell, M. Ditmyer, S. O'Malley, and G. M. Galbraith Creating an Evidence-Based Admissions Formula for a New Dental School: University of Nevada, Las Vegas, School of Dental Medicine J Dent Educ., April 1, 2007; 71(4): 492 - 500. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. M Breit and W. Kunzel Effect of the Recently Established Admissions Procedure on Success in the First-Year Exams at the University of Veterinary Medicine Vienna, Austria J Vet Med Educ, January 1, 2007; 34(3): 335 - 339. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
W. Kunzel and S. M. Breit Admissions Procedures at the University of Veterinary Medicine Vienna, Austria J Vet Med Educ, January 1, 2007; 34(5): 639 - 644. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
