| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Milieu in Dental School and Practice |
Key words: electronic patient record, dental, ROI, performance, systems evaluation
Submitted for publication 05/14/08; accepted 07/02/08
 |
Abstract |
|---|
 TopAuthor information Abstract MethodologyResults/findingsDiscussionReferences |
|---|
The use of electronic records is hypothesized to improve clinical quality and patient safety through reduction of medical and medication errors, since improved legibility should improve communication. These systems also allow records to be easily accessed by multiple parties, which is especially useful in a multiprovider clinic where patients might see different dental providers on each visit. Electronic records also should standardize the documentation of care, which could lead to improvements such as streamlining manual processes and eliminating paper.
Accordingly, many practicing dentists and dental schools are beginning to embrace the concept of the electronic patient record and move from paper charts to more expensive and comprehensive commercial systems and applications. The American Dental Association (ADA) and the ADA’s Standards Committee on Dental Informatics have set out a number of specifications and technical reports to ensure confidentiality, interoperability, and sound data architecture. The ADA and the American National Standards Institute have also developed Specification Number 1001 to help dental schools ensure appropriate designs of software for educational purposes, although it is not customized for the dental domain.2
Very little empirical research (i.e., research that is not anecdotal) is available to estimate or predict the actual total cost of implementing an EPR for dentists and dental schools. Our colleagues in medicine, however, provide a useful comparison. Electronic record systems in physicians’ offices have been studied frequently and are suggested to range in cost from $15,000 to $50,000 per physician.3 If we assume that these systems have comparable costs for dental offices, this becomes quite a substantial investment. Under most cost-effectiveness guidelines, one would expect technology investments to cover their expenses or produce some value in return, in the form of safety, quality, or clinical, operational, or financial benefits.4,5
Unfortunately, many comprehensive evaluations of electronic systems, primarily performed on large medical systems, have found evidence that EPR systems have costs that are greater than their combined benefits. Such studies have concluded that EPRs often do little more than automate an already existing record process and thus do not have a significant impact on outcomes, yet their costs are often measured in millions of dollars of investment and are met with widespread resistance from providers and staff.6
One of the most often-cited studies on electronic medical records (EMR) and benefits was conducted by the RAND Corporation to estimate the adoption and benefits of current systems.7 This study, published in 2005, found adoption of electronic records was somewhere between 15 percent and 25 percent, depending on the size, organizational type, and other factors. Using a number of data sources, the authors attempted to evaluate the potential savings impact of these technologies over a fifteen-year period, which they considered to be the typical adoption period based on studies of innovation, and to suggest a framework for identifying performance areas, defining costs and efficiency, and using financial modeling to compare these benefits against costs. Another recent national study has suggested physician adoption of the EMR is around 17 percent for limited functionality and only 4 percent for a fully functioning system, with the main barrier to further adoption rates being the high financial costs.8 In our research, we build upon the RAND framework for assessing integration of EPR into dental school clinics.
The electronic dental record has experienced mixed results similar to that of the electronic medical record. In general, there are extremely low adoption or penetration rates for a comprehensive EPR in dental offices, estimated by one research study at between 0 and 10 percent.9 It is assumed that dental schools have higher penetration, but no empirical studies have been reported. Although computing and technology systems have begun to make limited progress at chair-side and as instructional tools in the dental classroom, most have not yet achieved paperless status.10
If there are clearly documented gains in performance—whether in reduction of errors, improvement in the quality of care or teaching, or higher safety—this could be enough to justify the large investment in an EPR. Most institutions that implement the EPR claim that the systems are "quality initiatives" and not undertaken to make profits. Quality changes in dentistry could include improvements in the following:
In the absence of clearly documented gains in these areas, however, it is easy to question the actual value (i.e., cost-benefit) of the system.11 At a minimum, most institutions should expect to fully recover costs and typically should expect to earn at least marginal positive economic outcomes in the long run. Typical ways that costs can be recovered are by reducing labor costs, reducing cycle time of manual processes, and eliminating paper and film expenses, among others.
In a recent article, Atkinson et al. discuss a comprehensive set of criteria, features, and functionality for dental EPR systems.12 Their research, which evaluated the commercial EPR systems for dentistry, augments the ADA’s Technical Report of EPR criteria with practical applications of the data contained in the repository for improving research, education, service, and clinical activities. That study did not, however, provide guidance on how to evaluate whether a system implementation was able to recover costs or produce substantial gains in performance.
Besides initial costs, resistance of the health care provider has been identified as one of the primary barriers or obstacles to adoption and usage of these systems. A number of failed implementations have been widely discussed, such as the Cedars- Sinai EMR debacle in which staff rebellions led to basically unplugging the system in 2002, after running up costs of nearly $34 million.13 In the dental community, most of the barriers are similar to those experienced in other health care environments: finding a way to make the technology accessible, quick, and usable; incorporating the right features and criteria; and ensuring the underlying data model incorporates appropriate terms, taxonomies, and structures.14 Legal issues, including patient security and confidentiality through HIPAA, also have been widely discussed as barriers to implementation success.15 These types of challenges to the EPR could effectively mitigate any savings or benefits otherwise derived from the implementation.
The literature on value and outcomes attributable to electronic patient records in dentistry is very limited. There are few, if any, empirical findings exploring the relationship between benefits and costs of the EPR in a dental environment and, more specifically, the use of the EPR in a dental school. Based on the literature and theoretical foundations, it was evident to us that an evaluation should be performed to compare the theoretical value of the dental EPR versus actual results. While benefits can come from many areas, we specifically wanted to comprehensively evaluate a single implementation of an EPR and focus primarily on two key factors: 1) economic value, attributable to increases in revenues and efficiency or decreases in cost, which was explored in the study reported here; and 2) provider satisfaction, to estimate the support or magnitude of barriers to implementation success, which this research team is exploring in a current study to be reported in a future article. The findings from this evaluation should prove useful to other dental schools embarking on their own digital record implementations.
|
Methodology |
|---|
|
TopAuthor informationAbstractMethodologyResults/findingsDiscussionReferences |
|---|
This protocol was augmented by the gaps identified in the literature necessary for conducting a comprehensive EPR implementation, as described above. The evaluation was conducted for a single case study dental environment, and Institutional Review Board approval was granted. Details about the organization that was studied follow the methodology section. This EPR included a picture archiving and communication system (PACS) as well as the electronic dental record system, which together we refer to as the EPR. We systematically evaluated the impact of an electronic patient record on clinical and operational outcomes one year post-implementation, which is the first of a multiphase evaluation process to be conducted over the next several years. To do this, we followed a six-step methodology aimed at uncovering the value of the system during this initial time frame. Figure 1
shows a summary of the methodology.
|
Data Collection and Sources
Based on these documents, we identified the data to be collected, as well as their sources or repositories for both baseline and current data. Here, we obtained the existing data and explored gaps. There were a number of data sources that we relied upon. Structured, qualitative interviews with project participants, managers, and other stakeholders were performed. User perception surveys were also administered, using identical pre- and postsurvey designs. Other data relied upon were costs, productivity, revenues, process workflow, and space utilization. A number of other variables that we attempted to explore, such as reductions in medical errors or increase in quality or safety, were difficult to assess because our institution did not have existing performance metrics in these areas that were measurable. Although student admissions have increased annually, there was no specific evaluation of the quality of students’ educational experience, beyond a survey of their satisfaction with the system.
Ideally, all data should be available for at least twelve to twenty-four months pre-EPR implementation, to serve as baseline comparison against the post-implementation outcomes. As such, the research design we used was essentially a one sample, pre-test-posttest design. Development of the baseline information for comparison was paramount. To do this, we looked retrospectively for two years prior to implementation and forward twelve months post-implementation. Twelve months was sufficient for an initial evaluation and to establish a methodology for future iterative reviews, but is insufficient to gauge long-term efficacy results. Figure 2 shows the time-line used for this study.
|
One of the primary disadvantages with traditional economic evaluations, however, is its reliance on a single estimate of value. To improve on this, we performed advanced bootstrapping using Monte Carlo simulation analyses in Crystal Ball software. Bootstrapping is a form of computer simulation in which repeated computation of results is performed as sample parameters are changed simultaneously or as the sample is replaced by other observations.18,19 Bootstrapping is a widely used technique to evaluate a range of scenarios and outcomes and to incorporate risk and probability into otherwise static or deterministic models. It has been used to evaluate hypotheses and inferences in a variety of areas, including medicine and other aspects of health care.20
We analyzed the values of the data collected, and estimated variances and standard deviations for key parameters using descriptive statistics. Most of the data followed assumptions of normality in their distributions. Using Monte Carlo simulation, we tested 1,000 iterations (i.e., trials), applying sensitivity changes in several of the key parameters. These parameters included cost of capital (ranging from 6 to 10 percent); inflation rates (ranging from 2 to 5 percent); annual incremental labor costs (ranging from $0 to $60,000); and many others, which will be discussed in the Results section. Finally, we compared qualitatively and quantitatively the benefits, features, and savings expected prior to implementation against the actual results. Gaps, or differences between these, serve as recommendations for enhancements as well as lessons learned for future systems implementations.
Case Study Institution
The University of Texas Health Science Center at Houston Dental Branch (UTDB) is located in Houston, Texas, and is one of the fifty-seven U.S. dental education programs accredited to award the D.D.S./D.M.D. by the Commission on Dental Accreditation of the American Dental Association. Founded in 1905 as the Texas Dental College, it joined the University of Texas System in 1943 and serves as the only dental school in the Houston area. There are currently about 450 students, enrolled in one of many degree programs, including the D.D.S., M.S., B.S. in Dental Hygiene, and Dental Hygiene Certificate programs.
UTDB purchased an EPR in 2006. The EPR included scope of both imaging (PACS) and digital patient records. Two commercial systems were purchased and integrated, although for purposes of this research this is not a vendor evaluation and so we will keep this "vendor-neutral" or "-agnostic." By that we mean that the specific vendor or application was not important for this analysis, but rather the generic process of implementing any EPR system, so we will not discuss vendors here. The combined EPR system was primarily purchased to automate the relatively large clinical practice that occurs in the main facility, which provides nearly 150,000 treatments annually for the community. However, the solution also serves as an educational delivery tool, in which images and records can be shared throughout the "simulation classrooms" to improve pedagogy. Benefits of the planned acquisition, as stated in the institution’s original documents, included
Other potential benefits that could have been added to this original list include a) the opportunity to mine data for clinical research, which is only recently starting to happen at this institution; b) improved patient experiences, through higher satisfaction levels in clinics; and c) extending the use of the EPR in preclinical simulation to improve student transition to real patient care.
In most of these areas, however, there was not a specific, quantifiable target for the improvement, which would make it easier to evaluate post-implementation. Additionally, these figures should have been used to define a pre-investment "business case." The organization did document existing workflow, identify potential changes, and establish "to be" versions of clinical and academic processes. Additionally, not all ideal data around transactions (processes) and outcomes (performance) were available because of system or process limitations. Table 1 summarizes the data sources and their availability.
|
|
Results/Findings |
|---|
|
TopAuthor informationAbstractMethodologyResults/findingsDiscussionReferences |
|---|
Incorporating all of these benefits into a deterministic model over a five-year period, with the basic assumptions and distribution parameters from our data collected, the net present value was estimated at $425,191, which indicates fairly significant economic value. In most project evaluations of this size, a five-year NPV of this magnitude would be seen as quite a success. The reality, however, is that a large number of factors or variables influence this figure, and making a small change in simply one area has a huge impact on the outcome; thus, for each variable operationalized, there is a range of estimates and likely patterns of behavior. For example, a small increase in the cost of capital (or average cost to fund investments at a specific institution; for example, if bond rates yielded higher debt service payments) from 7 to 8 percent would have a 26 percent reduction on total net present value. To accommodate this variability, we used a bootstrapping simulation approach, which required us to make choices about variances and standard deviations of our assumptions and data and incorporated a greater degree of probability into each of the predictor variables. Table 2 shows each of the dynamic predictor variables in which we incorporated distribution models. Note that the static data (e.g., initial license costs, initial hardware) and certain other data are not listed because they are considered static and are not subject to change.
|
presents the range of estimates for economic value produced in the simulation models.
|
To test a scenario, we ran the same set of data using the same distributions for these variables, with one exception—we eliminated the cash inflow due to student fees. This way, we could better answer the question: would EPR produce positive economic outcomes if the dental school (or any other school) was not able to have fee increases designated for this purpose? (The UTDB was in a good position to raise fees, since the previous level was quite low in comparison to fees typically paid by students in other professional programs.) The answer to this question was "no," at least within the same five-year horizon. Without this sizeable benefit from student fees, the value of the EPR is significantly reduced. There is not enough clinical revenue generated (or quantification of other benefits, such as reductions in errors) to maintain a positive NPV without student fees as a benefit. The median net present value falls to –$74,575, which indicates that during the first five-years post-implementation this project would not have sufficient value. In the scenario in which student fees are excluded, and the analysis is extended to ten years (which is roughly equivalent to our projection on the payback period), then the project would break even financially. In this scenario (with no increase in student fees), the range of economic value produced in these simulation runs was between –$410,000 and +$160,000. Figure 4 presents this scenario.
|
|
Discussion |
|---|
|
TopAuthor informationAbstractMethodologyResults/findingsDiscussionReferences |
|---|
One of the key findings, or lessons learned, from this case is the need for comprehensive, quantitative expressions of the potential benefits up front, prior to the purchase and implementation of any system. Many institutions claim that they purchase an EPR for its potential impact on quality outcomes or safety, such as reduction in clinical errors or remakes, which is one contributing factor for why the Health Insurance Portability and Accountability Act (HIPAA) is helping to boost implementation rates of electronic systems in many health organizations. In addition, EPR has the potential to impact students’ educational process, although we were not able to specifically measure any changes in this study except to say that, in the survey process, the students’ perception of their education has increased due to the system and that they felt the system would improve their ability to practice after graduation. Without quantifiable estimates and measures for each benefit area, plus detailed views of initial and ongoing costs, the probability of creating positive outcomes is fairly minimal. Our institution did a good job of documenting many aspects of the project, but failed to concretely define precise values in many areas. With quantitative benefits documented, post-implementation decisions can be made to evaluate where and how the process and system should be changed. Without it, there is no clear rationale for implementing an electronic patient record.
Similarly, each of these benefits represents an outcome, or dimension of performance. If a clinical benefit is an improvement in patient safety, and if it is expressed in quantitative terms, there must be a systematic process for routinely measuring this metric pre-, during, and post-implementation. Relying on single points in time, without consistent measurement techniques, is a recipe for disaster. Our institution had certain metrics clearly defined and measurable at most points in the baseline period, but not all. This causes problems because one can really never look back and determine if the EPR produced any differential effect on those areas.
The final lesson learned is that while some benefits produce economic value, there are many others that do not. For example, the EPR in this study has the potential to improve student learning and educational processes. The ability for a faculty member to review a student’s work (e.g., case notes based on images on file) is very difficult to assign values to, but presumably (hopefully) these capacities result in higher-quality students and academic outcomes at some point. In addition, the "forcing functions" that support embedded rules and choices (e.g., choosing specific procedures or treatments in a drop-down list) also presumably (in a "makes sense" deductive appraisal) improves quality of care and reduces error, but is similarly difficult to quantify.
Revenue inflows at dental schools are fairly limited. Sources include tuition, grants, clinical revenue, state appropriations (for some schools), and student services fees. The issue of including any of these sources, beyond clinical revenue, can be debated for a number of reasons. For instance, if funds are secured through appropriations from the state specifically to purchase the system, is that really an incremental cash flow that should be attributed to the system? Accountants and economists typically view those as "unearned" and sometimes leave them out of analyses, as we did here with one exception—student fees. As we stated above, student fees reflect pricing decisions, similar to tuition, which are quantitative expressions of value received. If students perceive high value from the educational process, they are (perhaps or hopefully) more willing to pay higher prices. We realize that the UTDB case study was somewhat unique in that the current fees were low at approximately $95 per student/year, and a survey of the students indicated they would be able and willing to pay more if the technology was implemented. In other institutions, obtaining student support for enhancing fees might be more difficult to accomplish. One suggestion for institutions that are considering increasing fees to offset EPR costs would be to utilize economic simulation models, as described here, to understand the economic impact of a range of fee schedules. Such simulation would help to minimize the fees, but ensure they were sufficient to keep long-run net present value positive.
Dental schools and practicing dentists who are either considering or currently implementing electronic patient records should think carefully about the performance benefits to the practice (i.e., the expected benefits that the system should produce). A "scorecard" or dashboard is useful for improving management of resources and for aligning decisions with changes in outcomes. Performance management also ensures that benefits, costs, savings, and outcomes are known prior to embarking on the journey. Ensuring that data for transactions and outcomes are defined, documented, accessible, measurable, and quantifiable is essential. Understanding the process workflow, and redefining the workflow to take full advantage of the system, is also a key lesson learned.
A process and system for ensuring complete views on performance data, such as current dental error rates, really should become a higher priority. It has been suggested that the EPR improves safety in a number of ways: for example, by ensuring treatment plans are being followed; by reducing duplicative or contradictory tests and procedures; by reducing the time between an error and its discovery; by monitoring a patient’s allergic or other reactions to specific medicines; or by documenting all services performed to allow for quality assurance and monitoring. Although we have not seen any published empirical findings on these safety and quality dimensions in dental schools, the ADA and the Centers for Disease Control and Prevention have found increased prevalence and have issued reports and guidelines for monitoring infection control and other adverse events.25 Therefore, each institution should develop its own process for tracking and monitoring these metrics; the EPR could help in this area. Clearly, one could hypothesize that digital charts improve legibility, and reviewing students’ diagnoses and notes allows faculty to detect errors faster and minimize patient impacts. These benefits, if real, could produce significant value that needs to be quantified. Absent a process for clearly defining and tracking these incidents, however, no organization can clearly delineate the true impact of the EPR.
While there are no universal rules or assumptions that will apply to all institutions, in this particular case study, there were positive economic outcomes from the electronic patient record, even given a thousand iterative changes in probabilities and other risks, but enhancement of student fees was a key component in positive value outcomes. Continuing to expand and refine these evaluation models over a longer time period, and with a broader sample of institutions, is one suggestion for future research in this domain.
|
Author Information |
|---|
|
TopAuthor informationAbstractMethodologyResults/findingsDiscussionReferences |
|---|
|
REFERENCES |
|---|
|
TopAuthor informationAbstractMethodologyResults/findingsDiscussionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  M. F. Walji, D. Taylor, J. R. Langabeer II, and J. A. Valenza Factors Influencing Implementation and Outcomes of a Dental Electronic Patient Record System J Dent Educ., May 1, 2009; 73(5): 589 - 600. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
