HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Andersen, R. M. Articles by Carreon, D. C. Search for Related Content PubMed PubMed Citation Articles by Andersen, R. M. Articles by Carreon, D. C.

Pipeline, Profession, and Practice Program: Evaluating Change in Dental Education

Key words: evaluation of dental education, dental health services, dental schools

Submitted for publication 11/05/04; accepted 11/24/04

	Abstract

Top Abstract Conceptual and Analytical... Methods Baseline measures for... Conclusions References

 
This article describes the conceptual and analytical framework that will be used to assess the effectiveness of the Pipeline, Profession, and Practice: Community-Based Dental Education Program. The evaluation will use a mixed method qualitative and quantitative data collection, analysis, and triangulation. Baseline measures are reported using data from the 2003 ADEA survey of dental school seniors. Baseline measures show the dental schools are confronting a major recruitment challenge that will require short and long pipeline efforts to attract and retain underrepresented and low-income (URM/LI) persons. Gaps were found between the perceptions of URM and non-URM students in the adequacy of the curricula. The majority of all seniors described the current extramural clinical rotations as positive experiences, but URMs were more likely to report the experience improved their ability to care for diverse groups.

The Pipeline initiative was developed to address the critical shortage of oral health care services for underserved and disadvantaged populations. The nation’s dental care safety net is limited, and access problems are likely to become more acute in the next decade as the relative supply of dentists declines.^1,2

The RWJF funded programs in eleven of the fifty-six U.S. accredited dental schools, and one year later, TCE funded four additional programs in dental schools in California. Additionally, TCE required all five of California’s dental schools to develop a regional recruitment program for underrepresented minorities and a health policy initiative to sustain the Pipeline program after funding ends. In total, the two foundations invested more than $25 million dollars in the Pipeline program and its evaluation. Additionally, the W.K. Kellogg Foundation and TCE contributed $1.6 million for financial aid to underrepresented minority and low-income students recruited by the Pipeline schools.

The National Evaluation Team (NET) for the Pipeline project, based at UCLA, will determine if the three Pipeline objectives are achieved: 1) increase recruitment and retention of underrepresented minority and low-income (URM/LI) students; 2) revise didactic and clinical curricula to support community-based educational programs; and 3) establish community-based clinical education programs that will provide dental students and residents with sixty days of experience in this patient care environment.³ The specific evaluation questions the NET will address are:

• Underrepresented and Low-Income (URM/LI) Recruitment: Did the Pipeline schools increase the recruitment, admittance, and retention of underrepresented minorities and/or disadvantaged students?
  
• Revisions to the Curriculum: Was the curriculum revised, and were students prepared to provide culturally competent care to underserved and disadvantaged populations?
  
• Extramural Clinical Rotations: Did students and residents spend an average of sixty days in patient-centered community delivery settings, and was more care delivered to underserved patients? What is the financial and clinical training effect of shifting clinical programs from traditional sites to community sites?
  
• Practice Decisions: Upon graduation, what factors influence the decision to provide care to low-income and underserved populations?
  
• Health Policy: Did the schools work cooperatively, and were they successful in developing and implementing health policy initiatives to sustain the program?
  
• Sustainability: Are the Pipeline program components developed under the RWJF and TCE grants likely to be sustainable after the program ends?
  

	Conceptual and Analytical Framework

Top Abstract Conceptual and Analytical... Methods Baseline measures for... Conclusions References

 
The evaluation framework presented in Figure 1 was adapted from previous conceptual and analytical work on health care access.^4–8 The framework is based on the assumption that educational programs are influenced by multiple inputs, including the contextual environment and stakeholder groups. The major components of the Pipeline program (recruitment, curricular revisions, and extramural clinical rotations) are expected to influence the structures, processes, and outcomes of dental education. Structure refers to Pipeline plans, personnel, and resources. Processes are the strategies and approaches used to execute Pipeline plans, programs, and services. Outcomes are the intermediate and longer-term results of the Pipeline project. Ongoing monitoring and program management are provided to improve the performance of Pipeline programs including NET feedback reports and coaching and technical assistance provided by the National Program Office (NPO) at Columbia University. In turn, all of these factors influence the longer-term outcomes, including practice decisions of dentists, program sustainability, and ultimately, sustainable improvements in dental care access for underserved communities.

View larger version (31K): [in this window] [in a new window]   Figure 1. Conceptual and analytical framework

Stakeholders are individuals, groups, or organizations having a vested interest in the progress of the Pipeline program.¹⁰ All of the stakeholders—RWJF, TCE, NPO, dental associations, students, faculty, dentists, allied health professionals, and underserved communities—are viewed as inputs that facilitate or impede changes to dental education and dental care access.

Pipeline Program Components
The structure and process measures of URM/LI recruitment are considered predisposing conditions for dental care access. The evaluation examines the structure of the recruitment and retention programs, as well as the strategies and approaches used by the schools to increase URM/LI enrollment. The outcome includes the number of URM/LI students who are admitted to dental school, matriculate, and graduate each year. These intermediate term outcomes are linked to longer-term outcomes since increased numbers of minority providers increase the likelihood that underserved populations will receive care.^11–13

Similar to recruitment, the structure and process measures of curriculum revision are considered predisposing to improved access for underserved populations. Curricular revisions are designed to improve, for example, provider patient communication skills and cultural competency. Curriculum committee leaders, faculty, and graduating seniors are the sources of information used to assess the perceived strengths/weaknesses in the existing curriculum and barriers/facilitating factors to curriculum change. These intermediate outcomes represent potentially beneficial influences on long-term access outcomes since students completing didactic and experiential courses may develop a greater willingness to serve disadvantaged populations as well as better communication skills for interacting with diverse individuals.

Moving clinical training from the main school clinic to extramural rotation sites is expected to improve access because more dental students will be providing more services in underserved communities. We will examine the structure and process of developing these clinical rotation sites as well as outcomes of revenue generation and the productivity of students in extramural sites.

Annually, the NET provides feedback reports to all Pipeline schools to help them continually improve their programs. The reports describe and compare Pipeline schools in the baseline year and document structure, process, and outcome changes over the duration of the project. In addition to feedback reports, the NET presents an overall progress report at the annual grantees meetings, summarizing evaluation findings and their implications.

The NPO provides technical assistance to the funded schools through meetings, information sharing, program website, workgroups, and regular communications with the NET. Each year, the NPO convenes an annual grantees meeting(s) with a focused theme for improving the Pipeline program, for reporting comparative evaluation data, and for exposing grantees to best practices and innovative models for Pipeline program components. The NPO conducts ongoing program monitoring to assess the progress of the schools in achieving their objectives. The other major NPO activity is developing and executing an annual communications plan.

Long-Term Outcomes
The Pipeline initiative was developed to help address the critical shortage of oral health care for underserved populations. Thus, critical outcomes of the Pipeline initiative are the practice settings selected by recent graduates and the percentages of underserved populations these entry-level dentists expect to serve in their practice. Determinants of practice decisions will be investigated including demographic, social, and economic characteristics of seniors, their attitudes and beliefs, the influence of Pipeline program components, and the contextual environment.

Based on qualitative and quantitative results, evaluators will assess the probability that Pipeline program components will be sustained after foundation funding ends. Additionally, statewide recruitment efforts and health policy interventions will be closely monitored in California and elsewhere to determine implications for sustainability and lessons learned worthy of national replication.

	Methods

Top Abstract Conceptual and Analytical... Methods Baseline measures for... Conclusions References

 
Table 1 presents each data source, time schedule for data collection, and use in addressing the major evaluation questions. It also shows site visit interviews are being conducted to collect baseline data prior to Pipeline program implementation and subsequently to assess progress in achieving Pipeline goals. Interview data are collected using a uniform bank of questions for each of the three major program components: recruitment, curriculum, and extramural clinical rotations. For example, interview data will be used to assess the community-based dental education curriculum at baseline, the content of existing courses, plans for changing the curriculum, and barriers and enabling factors in proposing curricula changes.

ADEA conducts an annual survey of dental school seniors. The NET works closely with ADEA to modify the survey in an effort to capture changes in perceptions and preferences of graduating seniors related to the Pipeline program. Also, the annual American Dental Association (ADA) Survey of Predoctoral Dental Education serves as a critical source of information on the number of URM students entering dental schools.

A faculty survey is being conducted twice during the Pipeline years. The faculty survey reflects items in the ADEA survey of dental school seniors, allowing the NET to assess the gaps between student and faculty perceptions of the Pipeline program. Course syllabi are collected during the site visit years to assess curricula revisions and to identify innovative courses and teaching methods. The NET and NPO provided technical assistance to the schools for developing clinical information systems containing a uniform set of data elements collected at the extramural rotation sites. These data will show the kinds and volume of services provided at the sites. Contextual variables are being constructed to measure the policy, delivery system, university and school, and population characteristics influencing the Pipeline program and outcomes.

	Baseline Measures for Longitudinal Analysis

Top Abstract Conceptual and Analytical... Methods Baseline measures for... Conclusions References

 
In this section, we provide examples of measures that have been collected at baseline and will be followed over the course of the Pipeline Project. These measures are from the ADEA survey of dental school seniors collected annually using a self-administered questionnaire distributed to graduating seniors in accredited U.S. dental schools. The survey collects information about students’ financing of dental education, graduating indebtedness, practice and postdoctoral education plans following graduation, decision factors that influenced postgraduation plans, and impressions on the adequacy of time that was directed to various areas of predoctoral instruction.¹⁴ Each school uses its own distribution and collection system to conduct the survey. Surveys are returned to ADEA for analyses and reporting. Fifty-three accredited U.S. dental schools returned surveys in 2003, resulting in an overall student response rate of 83.2 percent. This response rate is based on the total number of students who completed the survey from the fifty-three schools returning surveys divided by the total number of students graduating from those schools.

From this survey, selected questionnaire items were identified that best represent recruitment, curriculum, and clinical services. These are critical baseline measures the NET will track longitudinally for the next five years to examine changes in dental education associated with the Pipeline program. Ethnicity and parents’ income as reported by graduating dental students are used to identify URM/LIs. Hispanics, African Americans, and Native Americans were combined into an underrepresented minority (URM) group, while whites and Asian/Pacific Islanders were kept separate. Parents’ income was divided into less than or equal to $30,000; $30,000-$50,000; and greater than $50,000.

Curriculum measures are based on the following questions: 1) do you believe the amount of time devoted to your instruction in each of the following areas was excessive, appropriate, or inadequate for cultural competency and social and behavioral determinants of health? 2) how would you rate how patients were treated as people (excellent, very good, fair, poor, very poor) in the main clinic? and in the extramural clinical rotations? and 3) you are prepared to integrate knowledge regarding cultural differences into treatment planning and care delivery (strongly agree, agree, disagree, strongly disagree). Clinical services measures included mean number of weeks students spent providing care in extramural rotations; whether number of weeks spent in last year at extramural rotations was inadequate; whether extramural clinic experience improved ability to care for diverse groups much or very much; and extramural clinical rotations were positive/very positive experiences.

In Table 2, responses to the recruitment, curriculum, and clinical services items are reported for National Pipeline schools (ten funded Pipeline schools excluding those in California), California Pipeline schools (all five dental schools in California), non-Pipeline schools (all other dental schools in the United States and Puerto Rico), and all schools combined. Furthermore, all responses are broken out by race and parents’ income. ADEA data will be collected and analyzed each year to assess progress of the schools in attaining Pipeline objectives inferred from the perspective of graduating seniors.

Curriculum Measures.
One quarter (25 percent) of all seniors rated the time devoted to instruction in cultural competency as inadequate (Table 2). URMs were most likely to rate the time as inadequate (36 percent) while whites were least likely (21 percent) to do so. Perceptions of URM and white students were fairly comparable among school types (National Pipeline, California Pipeline, and non-Pipeline). Asian/PI students in California Pipeline schools, however, were less likely (24 percent) to believe time was inadequate than their counterparts in the National Pipeline (36 percent) and non-Pipeline (32 percent) schools.

Sixteen percent of all seniors regarded time devoted to instruction in social and behavioral determinants of health as inadequate. URMs (23 percent) and Asian/Pacific Islanders (22 percent) were most likely to judge the instruction as inadequate while whites (13 percent) were less likely to do so. In describing how patients were treated as people at the main school clinic, 68 percent of all seniors reported patient treatment as very good/excellent. Results were comparable for the extramural clinic rotations, where 64 percent described this aspect of care as very good/excellent. Seniors in the California Pipeline schools were more likely to describe this aspect of patient treatment as very good/excellent than seniors in the National Pipeline schools for both the main school clinics (69 percent vs. 61 percent) and the extramural rotations (68 percent vs. 63 percent).

Most seniors (87 percent) for all schools described themselves as prepared to integrate knowledge of cultural differences into treatment planning and this percentage varied little by race or income. However, URMs in the National Pipeline schools were the most likely (94 percent) and the URMs in the California Pipeline schools somewhat less likely to (81 percent) to describe themselves as prepared.

Clinical Services Measures.
Key measures of community-based dental education show that the median number of weeks students spent providing care in extramural rotations was three weeks for National Pipeline schools, four weeks for California Pipeline schools, and three weeks for non-Pipeline schools (Table 2). A higher percentage of seniors described the number of weeks spent in the last year at extramural rotations as inadequate in the National Pipeline schools (43 percent) compared to the California Pipeline schools (24 percent) and non-Pipeline schools (29 percent). About one-third (35 percent) of all seniors agreed the extramural clinic experience improved ability to care for diverse groups much/very much. Seniors in the California Pipeline schools were most likely to agree (40 percent). URMs across all schools were also more likely to agree (45 percent) than Asian/Pacific Islanders (33 percent) or whites (35 percent). URMs were generally more likely to report the extramural clinic improved their ability to care for diverse groups much/very much (45 percent) than Asian/Pacific Islanders (33 percent) or whites (35 percent). Finally, 65 percent of all seniors described the extramural clinical rotations as positive/very positive experiences. URM seniors were most likely to describe the experience as positive/very positive in both the National Pipeline schools (81 percent) and the California Pipeline schools (87 percent).

	Conclusions

Top Abstract Conceptual and Analytical... Methods Baseline measures for... Conclusions References

 
This baseline article lays the foundation for a series of manuscripts to follow from the Pipeline program National Evaluation Team (NET). Evaluation questions were posed reflecting the major program components and intermediate and longer-term outcomes of the national program. The conceptual and analytical framework described here will be used to guide data collection and analysis longitudinally. A wide net has been cast to collect data from multiple stakeholder groups, who will report progress of the Pipeline initiative over the next five years. The evaluation will be a mixed method qualitative and quantitative data collection, analysis, and triangulation. ADEA baseline measures were reported for subsequent use in longitudinal analysis to track changes reported by successive cohorts of dental school seniors in Pipeline and non-Pipeline schools.

Not surprisingly, baseline measures show the dental schools are confronting a major recruitment challenge that will require short and long pipeline efforts to attract and retain diverse URM/LI persons. Gaps were found between the perceptions of URM and non-URM students in the adequacy of the curricula, with URM and Asian/PI students reporting the need for more instruction in cultural competency and the social and behavioral determinants of health than whites. The majority of all seniors described the current extramural clinical rotations as positive experiences, but URMs were more likely to report that the extramural clinical experience improved their ability to care for diverse groups.

	Acknowledgments

 
We are grateful to our foundation officers who have provided expert advice on the design and implementation of this national evaluation: Laura Leviton, Ph.D., and Mary Ann Scheirer, Ph.D., The Robert Wood Johnson Foundation; and Diane Manuel, Ph.D., The California Endowment. Additionally, Richard Weaver, D.D.S., M.S.D., Associate Director of the Center for Educational Policy & Research at the American Dental Education Association, has generously collaborated with the UCLA evaluation team to redesign the annual survey of dental school seniors to measure changes in students’ perceptions of dental schools during the Pipeline years.

	Footnotes

 
Dr. Andersen is Professor Emeritus, Department of Health Services, UCLA School of Public Health; Dr. Davidson is Associate Professor, Department of Health Services, UCLA School of Public Health; Dr. Atchison is Professor, UCLA School of Dentistry; Dr. Hewlett is Associate Professor, UCLA School of Dentistry; Dr. Freed is Clinical Professor Emeritus, UCLA School of Dentistry; Dr. Friedman is Director of the Workforce Development Center, West Los Angeles College; Dr. Thind is Associate Professor, Department of Health Services, UCLA School of Public Health; Mr. Gutierrez is Project Manager, Department of Health Services, UCLA School of Public Health; Mr. Nakazono is Programmer Analyst, Department of Health Services, UCLA School of Public Health; and Ms. Carreon is Research Associate, Department of Health Services, UCLA School of Public Health. Direct correspondence and request for reprints to Dr. Pamela L. Davidson, UCLA School of Public Health, Department of Health Services, CHS 31-293, 650 C.E. Young Drive South, Room 31-269CHS, Campus Box 951772, Los Angeles, CA 90095-1772; 310-825-7188 phone; 310-825-3317 fax; Davidson{at}ucla.edu.

	REFERENCES

Top Abstract Conceptual and Analytical... Methods Baseline measures for... Conclusions References

 

1. Health Resources Center, American Dental Association. Dental workforce model 2000–2020. Chicago: American Dental Association, 2002.
2. Guay HG. Access to dental care: the triad of estimated factors in access-to-care programs. J Am Dent Assoc 2004;135(6):779–85.[Abstract/Free Full Text]
3. Bailit H, Formicola A, Herbert K, Stavisky J, Zamora G. The origins and design of the dental pipeline program. J Dent Educ 2005;69(2):232–8.[Abstract/Free Full Text]
4. Andersen RM, Davidson PL. Improving access to care in America: individual and contextual indicators. In: Andersen RM, Rice TH, Kominski GF, eds. Changing the American health care system: key issues in health services, policy and management. San Francisco: Jossey-Bass, 2001.
5. Andersen RM, Davidson PL. Ethnicity, aging, and oral health outcomes: a conceptual framework. Adv Dent Res 1997;11:203–9.[Abstract]
6. Chen M, Andersen R, Barmes D, Leclercq MH, Lyttle CS. Comparing oral health care systems: a second international collaborative study. Geneva: World Health Organization, 1997.
7. Oral health in America: a report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, National Institute of Dental and Craniofacial Research, National Institutes of Health, 2000.
8. Davidson PL, Andersen RM, Hilberman DW, Nakazono TT. Transforming health services management education and development: a challenge for the new millennium. J Health Adm Educ 2000;18:63–110.[Medline]
9. Davidson PL, Andersen RM, Wyn R, Brown ER. A framework for evaluating effects of the safety net and other community-level factors on access to health care. Inquiry 2004;41:21–38.[Medline]
10. Rossi PH, Lipsey MW, Freeman HE. Evaluation: a systematic approach. Thousand Oaks, CA: Sage Publications, 2003.
11. Mofidi M, Konrad TR, Portefield DS, Niska R, Wells B. Provisions of care to the underserved population by National Health Service Corps alumni dentists. J Pub Health Dent 2002;62(2):102–8.
12. Weaver RG, Haden NK, Valachovic RW. Annual ADEA survey of dental school seniors: 2001 graduating class. J Dent Educ 2002;66(10):1209–22.[Medline]
13. Weaver RG, Haden NK, Valachovic RW. Annual ADEA survey of dental school seniors: 2002 graduating class. J Dent Educ 2002;66(12):1388–404.[Medline]
14. Weaver RG, Haden NK, Valachovic RW. Annual ADEA survey of dental school seniors: 2003 graduating class. J Dent Educ 2004;68(9):1004–27.[Free Full Text]

This article has been cited by other articles:

				The Macy Study Team, A. J. Formicola, H. L. Bailit, T. J. Beazoglou, and L. A. Tedesco Introduction to the Macy Study Report J Dent Educ., February 1, 2008; 72(2_suppl): 5 - 9. [Full Text] [PDF]

				B. R. Johnson, P. M. Loomer, S. C. Siegel, E. S. Pilcher, J. E. Leigh, M. J. Gillespie, R. K. Simmons, and S. P. Turner Strategic partnerships between academic dental institutions and communities: Addressing disparities in oral health care J Am Dent Assoc, October 1, 2007; 138(10): 1366 - 1371. [Abstract] [Full Text] [PDF]

				R. M. Andersen, D. C. Carreon, J.-A. Friedman, S. E. Baumeister, A. A. Afifi, T. T. Nakazono, and P. L. Davidson What Enhances Underrepresented Minority Recruitment to Dental Schools? J Dent Educ., August 1, 2007; 71(8): 994 - 1008. [Abstract] [Full Text] [PDF]

				E. L. Davis, D. C.L. Stewart, M. Guelmann, A. G. Wee, J. L. Beach, K. M. Crews, and R. S. Callan Serving the Public Good: Challenges of Dental Education in the Twenty-First Century J Dent Educ., August 1, 2007; 71(8): 1009 - 1019. [Abstract] [Full Text] [PDF]

				P. L. Davidson, D. C. Carreon, S. E. Baumeister, T. T. Nakazono, J. J. Gutierrez, A. A. Afifi, and R. M. Andersen Influence of Contextual Environment and Community-Based Dental Education on Practice Plans of Graduating Seniors J Dent Educ., March 1, 2007; 71(3): 403 - 418. [Abstract] [Full Text] [PDF]

				B. S. Graham Educating Dental Students About Oral Health Care Access Disparities. J Dent Educ., November 1, 2006; 70(11): 1208 - 1211. [Abstract] [Full Text] [PDF]

				H. L. Bailit, A. J. Formicola, K. D. Herbert, J. S. Stavisky, and G. Zamora The Origins and Design of the Dental Pipeline Program J Dent Educ., February 1, 2005; 69(2): 232 - 238. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Andersen, R. M. Articles by Carreon, D. C. Search for Related Content PubMed PubMed Citation Articles by Andersen, R. M. Articles by Carreon, D. C.