J Dent Educ. 70(12): 1328-1338 2006
© 2006 American Dental Education Association
Educational Methodologies |
Use of Handheld Computer Technology to Monitor General Practice Residents’ Clinical Experiences
Marc D. Dyer, D.M.D.;
Emory A. Presler, D.M.D.;
Ted P. Raybould, D.M.D.;
Cindy Burklow, M.B.A.;
Timothy A. Smith, Ph.D.
Key words: Palm Pilot, residency programs, computer technology
Submitted for publication 01/23/06;
accepted 08/24/06
 |
Abstract
|
|---|
The Commission on Dental Accreditation (CODA) requires that General Practice Residency (GPR) programs report a summary of each resident’s clinical experiences. The goal of this study was to determine the effectiveness of handheld computer technology in tracking residents’ clinical procedures. In the academic years 2004–05 and 2005–06, University of Kentucky GPR residents were provided a Palm M130® with customized forms. These menu-driven forms allowed residents to record procedures. Residents synchronized the data to a centralized database. A separate reporting system interfaced with the database to allow administrators to produce detailed reports on each resident. At the end of the academic year, data were tabulated, clinical procedures were correlated with respective costs, and annual production was determined. Data from fourteen residents were analyzed. Types and frequencies of various clinical experiences were tabulated, financial production summaries assessed, and percentage distribution of procedures by discipline was examined. Palm technology has proven to be an effective tool in monitoring each resident’s clinical experiences. This methodology was beneficial to residency directors by allowing the examination of the experience distribution of each discipline in dentistry to determine the range of experiences that residents achieved.
The Commission on Dental Accreditation (CODA) requires that General Practice Residency (GPR) programs report an annual summary of each resident’s clinical experiences.1 Residents in GPR programs often encounter difficulty in accurately tabulating, documenting, and reporting diagnostic and therapeutic experiences completed during their residency training. In our experiences, a resident’s compliance with a particular documentation method often correlates with the method’s ease of use. Assessment of past methods indicated a need for alternative methods to collect and analyze residents’ clinical procedural data. Paper reporting was not an efficient method to collect data, and time-consuming data entry into a computer file was needed for in-depth statistical analysis.
In the University of Kentucky General Practice Residency, residents rotate through seven different clinical venues. At each clinic, the residents perform many aspects of general dentistry. Each clinical venue has its own proprietary computer system that does not interface with other clinical systems. Historically, most residents’ procedural data were obtained through computerized billing systems. However, data on resident out-of-unit experiences (off-service rotations) were very difficult to obtain. Therefore, at the conclusion of the academic year, each clinical venue reported to residency administrators annual production summaries in different formats. From this, it was an arduous task to compile procedural data for seven to eight residents from the different clinics per annum. The administration absorbed the clerical, materials, and hardware expenses of data accumulation and assimilation, but an improved method of data collection was necessary to streamline the data accumulation and reporting process. Interestingly, Garvin et al. reported that, in 1987, Schneeweiss et al. cited costs of $153 per resident per year to merely tabulate annual resident data.2 Moreover, previous reporting systems focused more on the aggregate information than detailed data on individual residents and site locations. Therefore, an improved data collection method could allow administrators to examine the effectiveness of different clinics and the experience distribution per individual resident. Finally, none of the previous methods allowed documentation at the time of the procedure. It has been suggested that making medical information more accessible at the point of care can improve patient outcomes and treatment results.3,4
It was anticipated that resident usage of Palm handheld technology in documenting clinical experiences would be an extremely effective tool in assimilating and analyzing the data. Also, it was anticipated that the use of Palm handheld computers would streamline the process of resident procedure documentation. Moreover, residency administrators anticipated that the project costs would compare equally to other methods of procedural documentation.
The Commission on Dental Accreditation (CODA) urges residency programs to employ technology in the training of their residents and also requires that each program "apply scientific principles to learning and oral health care. This includes using critical thinking, evidence- or outcomes-based clinical decision making, and technology-based information retrieval systems."1 Palm handheld technology has been extensively used in medical residency programs,2,5–8 but a review of the literature found no reports on its usage in dental residency programs.
The use of Palm technology in medicine and dentistry has increased dramatically over the last ten years. It is estimated today that between 26 and 50 percent of physicians currently use Palm technology as a delivery tool to access information at the point of patient care.7 Barrett et al. report that "[usage] rates appear higher among residents, with one recent study finding that over two-thirds of family practice residencies use handheld computers in their training programs."7 In clinical dental practices today, many dentists use Palm handhelds to carry a mobile record of their patient base and refer to this to obtain patient demographic information when after-hours issues arise, such as postoperative problems or other dental emergencies.9 Furthermore, dentists use Palm handhelds for many other functions. For example, many major practice management software companies have created Palm-based programs that allow clinicians to transfer information from the practice software programs to the handheld.10 Also, dentists are using the Palm handheld computers to function as drug references, act as calendars and address books, retrieve email, and obtain and store patient data chairside.11 Additionally, dentists have used their Palms clinically to serve as references for fluoride supplementation tables, pediatric dosages for SBE prophylaxis, and oral infection management references.11
|
Materials and Methods
|
|---|
The General Practice Residency (GPR) program at the University of Kentucky College of Dentistry is a hospital-based residency program with a total of seven first-year residents and one second-year resident in the 2005–06 class. There were seven first-year residents in the 2004–05 class that participated in the study. In 2003, this GPR initiated a program designed to measure the effectiveness of the usage of Palm handheld technology as a reporting medium to document resident clinical procedures as required by the Commission on Dental Accreditation. Through a U.S. Health Resources and Services Administration (HRSA) grant (Grant Number 30 46904600), the University of Kentucky GPR program purchased Palm handhelds, developed proprietary software, and purchased external memory cards in order to track clinical experiences for the 2004–05 and 2005–06 academic years. After review of options, the Palm M130® was purchased for use by the residents. This handheld unit was introduced into the marketplace in March 2002. Table 1
shows the handheld’s specifications.13 The handheld units were purchased with 256 MB expansion cards at an estimated cost of $350 per resident.
In the academic years 2004–05 and 2005–06, each resident was provided a Palm M130 with customized Pendragon® Forms software that took advantage of touch-screen technology. Pendragon Forms software was selected due to its flexibility in developing customized forms for personal digital assistants (PDAs) that support both Palm OS and Microsoft® Windows CE platforms. Pendragon Forms provides the programmer the environment to design and write scripts specifically for creating forms for PDAs for any particular application, not just for dental-related forms. All of the forms created for this project were newly created and specially programmed to meet the needs of collecting resident procedural data and were not prefabricated forms. These menu-driven forms allowed residents to record procedures. Also, the menu-driven forms were formatted to reflect all disciplines of dentistry and allowed residents to quickly record tasks. Figure 1 shows a representation of the software that was used.
After recording the procedures in the hand-held units, the residents synchronized the data to a centralized database by "Hot-Syncing" on a desktop computer. Residents had the option of syncing with computers in the clinics with the Pendragon Forms software, or they could sync their handhelds via the Internet with their home computers, which would send the information to the university database. If the resident opted to sync from home, a special set of programs—such as virtual private network (VPN) software—had to be installed on his or her home computer. VPN software provides a secure private communication network over the Internet that ensures the secure transmission of data. All procedural data were transferred from the Palm into a secure enterprise-level Microsoft SQL database server by using an Internet connection. Pendragon Forms enable the user to select any database of choice to store the data as well the capacity to export the data to any reporting system. For this project, we developed a separate customized reporting system utilizing Microsoft Access interfacing with the enterprise-level database to allow administrators to produce detailed reports on each resident. This reporting system enabled the administration to generate individual resident reports for any specific period of time, aggregate data for location, or aggregate group data. At the academic year’s conclusion, the annual data were electronically tabulated, clinical procedures were multiplied with respective procedure cost, and annual production summaries were generated. By taking advantage of Palm touch-screen technology, in combination with electronic forms, database, and Internet application servers, it was possible to electronically collect resident procedural data in an efficient manner without need of paper.
In addition to the use of electronic tools and handheld computers to collect procedural data, we sought to examine the opinions of the residents involved in the study. A questionnaire was designed to elicit resident opinions regarding the usage of Palm handheld computers, hardware and software functionality, and overall resident opinions (Table 2). Surveys were administered to each resident and subsequently analyzed and frequency distribution and correlations calculated. With the survey results, a correlation analysis was performed to examine resident opinions and frequency of PDA usage.
|
Results
|
|---|
This study examined data from two residency classes (2004–05 and 2005–06) over a two-year period. Overall, data from fourteen residents were analyzed. From resident experience data, total clinical experiences were accumulated, financial production summaries assessed, and percentage distribution of procedures by discipline was examined. From the annual reports, residency administrators examined how each clinical venue fared with respect to the other clinics. They were able to determine what percentage of procedures residents performed at each clinic. Also, we were able to quantify the annual production of billed procedures to see exactly how much each resident produced.
Based upon the reported procedures, in the 2004–05 residency class, each resident produced approximately $100,000 per annum. This production total was based upon a ten-month schedule because residents in this program complete two one-month rotations with anesthesiology and internal medicine. During these months, residents do not treat patients in the dental clinics. The highest levels of patient care experiences and financial production came from restorative dentistry, oral surgery, and diagnostic procedures. Figures 2 and 3 present the distribution of clinical experiences by discipline and generated revenues for the 2004–05 residency class. Figure 4 also shows a summary of additional clinical experiences from the 2004–05 residency class.
Survey Results
Twelve of fourteen residents responded to the questionnaire. Table 2
presents the seventeen-item questionnaire with the numbers or percentages of respondents selecting each response. First, 50 percent of residents reported using their Palms on a daily basis, 41 percent reported usage on a weekly basis, and 8 percent reported using their Palms on a monthly basis (Item #1). Second, 41 percent of residents reported using a PDA/Palm handheld prior to this study, and 58% reported never before using handheld technology (Item #2). A majority of residents (58 percent) reported that their Palm handheld allowed them to be more efficient in their daily practice (Item #4). Residents reported that the Palm Pilot components that were the most difficult to use were the daily maintenance required for the handheld device functioning followed by entry of procedures (Item #5). Approximately 75 percent of residents found the software to be somewhat effective in recording data (Item #7), and 92 percent found the menu-driven forms to be user-friendly (Item #9). However, only 66 percent of residents perceived the addition of handheld computers to be beneficial for the residency program (Item #8). With respect to future usage, 50 percent reported they would use a PDA in their future practice, and the other 50 percent reported they did not have an interest in using a handheld in their future clinical practice (Item #13). Most of the residents used their handheld as a drug reference source, and 25 percent utilized medical reference software (Item #15).
Furthermore, a correlation analysis was performed with the survey data. Table 3 presents the correlations between other questionnaire items and Items #1 and #9. In our analysis, the P value indicates that these correlations exceeded the 0.05 level in statistical significance, and thus are not likely due to chance. Table 3 displays the correlations with respect to the frequency of Palm Pilot usage. Those individuals who reported the highest rates of PDA usage reported that their PDA allowed them to be more efficient in their daily practice. Moreover, these individuals tended not to store any personal information in their PDA. Furthermore, they found the menu-driven forms easy to use, and half of these people planned to use a PDA in their future practice. Second, examiners correlated the data for those residents who reported finding the recording software to be user-friendly. These residents had the highest PDA usage rates, reported that keeping their Palm functioning was the most difficult item they found in using their Palms, and had the lowest rates of drug reference utilization. Also, these individuals rated the Forms 5.0 program easy to use and found the program to be effective for its intended purpose. Important to note, there was no relation between previous PDA experience by residents entering the study (Item #2) and the judgment of the PDA’s increasing clinical efficiency (Item #4) and being considered beneficial to the residency (Item #8). There did not appear to be significant differing opinions among the residents with respect to experienced versus novice users with respect to the handheld’s usefulness. Also, the residents did not report substantial difficulties in learning to use the recording software. Most software problems, with respect to software navigation, were reported to come from the medical reference software, not the recording application.
Project Costs and Technical Difficulties
The largest cost for the project was associated with purchase of the handheld units. Additional costs have been in the technology personnel associated with the maintenance of the database and continued upkeep of the Palm units. However, project costs have compared favorably to other methods of data accumulation and analysis.
This project encountered a few technical issues. First, 58 percent of the residents reported never before using Palm handheld computers. Each resident attended an instructional class during resident orientation pertaining to the basics of handheld computing, extensively focusing on the proper use of the software designed to record clinical procedures. The first difficulty some residents reported was not maintaining the battery’s charge so that the hand-held unit "died" and the recording software had to be reloaded by the Palm coordinator. After further investigation, it was determined that the battery did not maintain a charge because the residents failed to regularly charge their Palm handheld, not due to a hardware failure. At times it was found that residents would often go for extended periods of time without charging their handheld. By the time the resident discovered that the handheld had "died," it was too late since the handheld had sat idle for days without an electric charge, causing the memory to be erased. Palm M130 handhelds do not have flash memory like the newer Palm models; hence, they require a consistently charged battery to avoid loss of memory. After the battery was completely depleted on the Palm M130, it was discovered that the residents had only a day or two to recharge the Palm handheld before complete memory loss would occur. To counteract the data loss by using synchronization technology with the resident’s local computer, the lost data was restored onto the Palm handheld up to the point where the last performed hot-sync backed up the handheld on the local computer. Therefore, it was strongly emphasized to the residents that they should hot-sync their Palm handhelds on a daily basis to keep their data up-to-date on the local computer in case of hardware failure.
Furthermore, synchronization problems arose during the second year of implementation with the residents who opted to sync their Palm from an off-site location. Residents reported that during attempts to hot-sync from off-site locations, error messages appeared on the handheld. After further investigation, the study’s technical coordinators discovered that the university had implemented a new security policy that required that all Palm handheld devices utilize special software called a virtual private network (VPN) client to hot-sync with any university server when syncing from an off-site location. VPN software allows a secure network connection between a remote computing device and a host VPN server and to access intranet and extranet networking resources that are restricted to certain IP addresses. After installing the secure VPN client, residents were able to successfully synchronize their handhelds to the university database.
Additionally, another difficulty reported was the Palm handheld "freezing" during use. The source of this problem arose after residents had downloaded other "free" software onto their Palm device. Often these software packages are bundled with other unnecessary programs that consume large portions of memory, causing the Palms to behave erratically. Once specific components of the free software were removed, the Palm handheld operated normally. In response, study technology personnel advised the residents on what free software was permitted to be installed on their Palm handhelds and warned of risks associated with downloading other software packages.
|
Discussion
|
|---|
In this study, the use of handheld computer technology proved to be an extremely effective tool to record and analyze resident procedural data. The handheld units that were purchased functioned well for their intended purpose. Residents reported additional uses beyond simply using the handhelds as a procedural recording device. Furthermore, residents reported being able to carry and use the handheld units in the clinic with ease. Moreover, in addition to the numerical summary of recorded experiences, additional uses in this project were discovered. Among the beneficial uses were the following.
Prospective Resident Practice Management Tool.
At the completion of the first academic year, Excel spreadsheet programs were created that combined the reported quantified procedural tally with a resident fee schedule. From this, residents could get a numerical production value for the year based on the number of reported procedures multiplied by an established fee schedule per procedure. This information has proven invaluable in discussions with potential financial lenders by providing a practice management tool illustrating residents’ prospective clinical production capabilities. Residents greatly benefited from the use of financial analysis summaries that demonstrated their production capability. This type of methodology can be used as a practice management tool to teach residents the business value of clinical production as the residents move into private practice.
Residency Program Evaluation Tool.
This methodology was beneficial to residency directors as well by allowing the examination of the experience distribution of each discipline in dentistry to determine the range of experiences that residents achieved. As the study monitored each resident’s progress, it also evaluated the success of training sites in exposing residents to all dental disciplines. Administrators found that residents in this program are adequately exposed to all disciplines of general dentistry.
Hospital Reporting Tool.
The University of Kentucky General Practice Residency is a hospital-based residency program. As a result of their training, many of the residents proceed to perform various aspects of hospital dentistry after residency training. Often, "hospitals where our [residency] graduates apply for staff privileges ask us to certify not only their general training, but the exact number of various procedures the resident has done."14 When applying for privileges, the data reported during residency training can easily be supplied to hospital credentialing committees to document resident experiences. By accessing their electronic records, residency administrators can assure themselves of a resident’s experiences before certifying them at area hospitals. Moreover, by adequately documenting their procedures, residents can be assured of obtaining the privileges they indeed want. Finally, hospitals may in the future require general dentists to be board-certified. In the certification process, often dentists must document resident experiences. Using the handheld recording device and software, these numbers are readily available.
Recruiting Tool.
The annual experience summaries served as a recruiting tool to prospective residents by displaying the variety of procedures future residents may expect to perform. At the most recent interview session, interviewees were shown a summary of clinical experiences. The interviewees stated that they found this information valuable and would factor this information into their residency program choices.
Resident Comparison Tool.
The generated reports permitted enrolled residents to examine which areas of dentistry consumed their time. Also, the reported data functioned as a comparison among residents to gauge progress and productivity.
Unexpected Resident Uses of Palms.
In addition to using the PDAs for recording procedures, the residents reported that they used their handhelds for other functions too. Among the most popular reported ancillary functions, the residents used drug and medical reference software that helped them in the clinical setting. Most residents reported using the Epocrates® drug reference program in their Palm Pilots (www.epocrates.com). This software is very popular in medicine and dentistry today and is extensively used by physicians and dentists. In the University of Kentucky GPR, residents often treat patients who have complex medical problems. These medically compromised patients frequently have extensive medication lists that residents must examine. It is prudent to examine possible drug-drug interactions, and the pharmacology reference programs in the Palms proved invaluable. Fishman et al. comment that "an expeditious analysis of drug interactions is almost impossible without computer assistance. . . . the utility of pocket sized computers for this purpose can not be overemphasized."3 Moreover, one resident also used a medical reference program entitled "5mDental"® (www.skyscape.com/estore/ProductDetail.aspx?ProductId=317). This program also served as an invaluable tool in being an excellent reference source for the resident by allowing the examination of complex medical histories and how to manage patients with these disorders. In today’s clinical setting, dentists are challenged by massive amounts of medical and drug information. Often, the appropriate reference materials are of limited value because they are not available at the point of care. This study demonstrated that this important information can be retrieved at the point of care and improved treatment results obtained. In this, by having voluminous amounts of medical information and drug reference software available at the point of care, residents could make better informed decisions in the management of their patients. Importantly, handheld information retrieval has proven to be much more effective, user-friendly, and productive than previously attempted methods. A literature review showed that previous methods attempted to create an "evidence cart" of clinically applicable information, which was found to be too bulky and cumbersome to utilize and carry around.15
Future Goals.
As the study progresses, administrators continue to discover areas of improvement and expanded uses. First, the residency director hopes to create a program that will interface with clinical scheduling programs, so residents can view their clinical schedules on their Palms. Next, administrators hope to provide more software tools on the Palms such as increased medical reference programs. Finally, technology personnel are examining the idea of incorporating PDA attachments such as a digital camera to further aid in educating residents.
|
Conclusion
|
|---|
This study demonstrated the benefits of using Palm technology to track residents’ clinical experiences in advanced training programs. This approach to documenting each resident’s clinical experiences and providing residents with a summary of annual production revenues was more cost-effective and time-efficient than paper-based monitoring systems used in the past. Using these data, residency directors can assess residents’ educational experiences and guide residents to other activities and types of patient encounters that will augment their training. Use of PDAs also provided an effective method for collection of required CODA resident procedural data and served as a practice management tool for the residency program director. In summary, careful analysis of annual resident performance facilitates assessment of current program strengths and weaknesses and, more importantly, can dramatically aid in helping to improve the quality of the residents’ educational experience.
|
Footnotes
|
|---|
Dr. Dyer is Chief Resident, General Practice Residency, Division of General Dentistry, Kentucky Clinic Dentistry; Dr. Presler is Resident, General Practice Residency, Division of General Dentistry, Kentucky Clinic Dentistry; Dr. Raybould is Professor, General Dentistry and Preventive Care, Division Chief, General Dentistry, University of Kentucky College of Dentistry, Director, Kentucky Clinic Dentistry, and Director, General Practice Residency, Kentucky Clinic Dentistry; Ms. Burklow is Senior IT Consultant and Developer, BizSoft, Inc.; and Dr. Smith is Professor, Behavioral Science, University of Kentucky. Direct correspondence and requests for reprints to Dr. Marc D. Dyer, 804 N. Mulberry St., Elizabethtown, KY 42701; 270-769-3990 phone; 270-737-7344 fax; mddyerdmd7{at}hotmail.com.
|
REFERENCES
|
|---|
- Commission on Dental Accreditation. Accreditation standards for advanced education programs in general practice residency. Chicago: American Dental Association, 1998. At: www.ada.org/prof/ed/accred/standards/gpr.pdf. Accessed: November 20, 2005.
- Garvin R, Otto F, McRae D. Using handheld computers to document family practice resident procedural experience. Fam Med 2000; 32(2):115–8.[Medline]
- Fishman SM, Prince JB, Herman JB, Cassem NH. Technical progress: pocket-sized electronic referencing. MD Comput 1996; 13:310–3.[Medline]
- Ebell M, Rorner D. Information in the Palm of your hand. J Fam Pract 2000; 49(3):243–51.[Medline]
- Beasley BW. Utility of palmtop computers in a residency program: a pilot study. South Med J 2002; 95(2): 207–11.[Medline]
- Rao G. Introduction of handheld computing to a family practice residency program. J Am Board Fam Pract 2002; 15(2):118–22.[Medline]
- Barrett JR, Strayer SM, Schubart JR. Assessing medical residents’ usage and perceived needs for personal digital assistants. Int J Med Inf 2004; 73(1):25–34.[Medline]
- Fishman SM. Palmtop computer on the medical wards. JAMA 1992; 267(1):169.[Medline]
- Kehoe B. Practice management in the palm of your hand. Dentalproducts.net. At: http://dentalproducts.net/xml/display_printer_friendly_asp?file=1205&type=textonly. Accessed: October 31, 2005.
- Flucke J. Handheld computing: you can take it with you! Dentalproducts.net. At: http://dentalproducts.net/xml/display_printer_friendly.asp?file=2014&type=textonly. Accessed: October 31, 2005.
- Emmott L. A guide to handheld computers in dentistry. Dentalproducts.net. At: http://dentalproducts.net/xml/display_printer_friendly.asp?file=268&type=textonly. Accessed: October 31, 2005.
- Taylor MH. Handheld computing in dentistry. Dent Clin North Am 2002; 46:539–51.[Medline]
- Palm One support knowledge library. At: http://kb.palmone.com/SRVS/CGI-BIN/WEBCGI.EXE?New,Kb=PalmSupportKB,ts=Palm_External2001,Case=obj(16608). Accessed: December 3, 2005.
- Wheeler RP. The advantages of a microcomputer-based procedure log. MD Comput 1989; 6(1):24–6.[Medline]
- Sackett DL, Straus SE. Finding and applying evidence during clinical rounds. JAMA 1998; 280(15):1336–8.[Abstract/Free Full Text]
Top
Abstract
Materials and Methods
