HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Kim, S. Articles by Shaul, C. Search for Related Content PubMed PubMed Citation Articles by Kim, S. Articles by Shaul, C.

Implications for Designing Online Oral Health Resources: A Review of Fifty-Six Websites

Dr. Kim is Assistant Professor of Family Medicine; Dr. Mouradian is Clinical Associate Professor of Pediatrics, Pediatric Dentistry, and Health Services; Dr. Leggott is Professor of Pediatric Dentistry; Dr. Schaad is Associate Professor of Medical Education and Biomedical Informatics; and Ms. Shaul is Program Coordinator, Pediatric Dentistry—all at the University of Washington. Direct correspondence and requests for reprints to Dr. Sara Kim, Box 356390, Department of Family Medicine, School of Medicine, University of Washington, Seattle, WA 98195; 206-543-9425 phone; 206-543-3821 fax; sarakim{at}u.washington.edu.

Key words: oral health, Internet, learning, computer-aided design, user-computer interface

Submitted for publication 02/13/04; accepted 04/08/04

	Abstract

Top Abstract Methods Results Discussion Conclusions References

 
We conducted a review of websites in oral health to identify content areas of our target interest and design features that support content and interface design. An interprofessional team evaluated fifty-six oral health websites originating from nongovernmental organizations (NGOs) and associations (28.6 percent), regional/state agencies (21.4 percent), federal government (19.6 percent), academia (19.6 percent), and commercial (10.7 percent) sources. A fifty-two item evaluation instrument covered content and web design features, including interface design, site context, use of visual resources, procedural skills, and assessment. Commercial sites incorporated the highest number of content areas (58.3 percent) and web design features (47.1 percent). While the majority of the reviewed sites covered content areas in anticipatory guidance, caries, and fluorides, materials in risk assessment, oral screening, cultural issues, and dental/medical interface were lacking. Many sites incorporated features to help users navigate the content and understand the context of the sites. Our review highlights a major gap in the use of visual resources for posting didactic information, demonstrating procedural skills, and assessing user knowledge. Finally, we recommend web design principles to improve online interactions with visual resources.

The Internet has also served as a major vehicle for delivering core and supplemental educational materials in dentistry and medicine. Many guidelines are available for the development of credible and user-friendly web-based educational materials. Some of these guidelines include: American Dental Association’s Guidelines for the Design of Educational Software,⁴ Health on the Net,⁵ the Quality Criteria for Electronic Publications in Medicine,⁶ and the guidelines developed by the Standards Committee for Dental Informatics.⁷ While general guidelines are useful, our understanding of which design features are most frequently incorporated to deliver and present content in a particular domain is limited.⁸

In spite of many challenges in conducting studies with websites due to inconsistent availability of sites over time, several studies have applied a range of design criteria to evaluating existing websites. Eysenbach et al.⁹ described the characteristics of seventy-nine studies of online consumer-oriented health information based on the following criteria: assessment target (accuracy, completeness, technical criteria, readability, design); search quality (search tool, search terms, consumer involvement); and evaluation quality (number of raters, inter-rater reliability, criteria standard for accuracy and completeness). The results from these studies varied widely across health-related websites because of disparate study methods, rigor, quality criteria, study population, and target topics. In an updated study of an earlier work, Gagliardi and Jadad¹⁰ reviewed the quality of online instruments developed for rating health information available over the Internet. The review criteria included whether the rating instrument covered a specific scope on health, authorship, attribution to reference sources, disclosure of ownership, method of scale development, and reliability of the instrument. They found that many of the instruments reviewed in the original study were no longer available, only a few instruments met the evaluation criteria, and none of the instruments were validated.

In this article, we describe a methodology developed for assessing fifty-six websites in oral health and present key results of our review, including content areas and design features. Our study uses methodologies from evaluative studies of online educational resources, primarily in anatomy.^11,12 The choice of these methods is based on the importance of visual media in dental education.⁴ The study by Frasca et al.¹¹ ranked web-based anatomy sites using a Likert scale based on navigational ease, quality of images, value and amount of textual information, and presentation styles. Most sites received a score between two and three out of a maximum score of five. Kim, Brinkley, and Ross¹² reviewed forty anatomy sites in four categories: 1) site background information (institution and authors’ names, contact information, content posting dates); 2) content (image sources, use of 3D models, labels, text, self-evaluation features); 3) interactivity features (sound, zooming, rotation, assembling/disassembling of anatomical structures); and 4) user interface (help menu, site map, navigational links, minimum scrolling). The average score of the reviewed sites was 3.3 out of a possible score of ten, suggesting that many of the websites lacked features that enhanced the usability of the site and that facilitated learning in a predominantly visual domain, such as anatomy.

The purposes of our study were to compile a list of websites with oral health content in our targeted area of interest; identify key design features for supporting content and interface design; and develop web design principles for improving learning in domains strongly facilitated by visual resources, such as in oral health. We expect that our review would benefit a wide scope of constituents, ranging from dental and medical practitioners and trainees who may search for specific content materials in oral health, to faculty and web designers who may seek exemplar sites and design principles that optimize features for organizing and presenting content materials.

	Methods

Top Abstract Methods Results Discussion Conclusions References

 
The websites reviewed in this study were identified through a keyword search using the Internet search engine google.com and national organizations dealing with oral health issues. We searched the websites using the following keywords: caries, role of fluorides, nutrition and dietary practices, anticipatory guidance, and health counseling for families. In addition, we were directed to several sites recommended by key professional contacts. Initially we identified 115 web resources in oral health. Sixty-six sites were selected for review after applying the following exclusion criteria: 1) sites that did not include relevant informational or educational materials in oral health; 2) sites that required payment or passwords for access; 3) sites that did not include institutional identifications or authorship; 4) sites that were designed primarily for advertisement purposes; and 5) sites that only offered materials in PDF format, which made it difficult to assess web design features in these sites.

The URL addresses and institutional names of these sites were posted on the web for access by an interprofessional team of evaluators (they are available at faculty.washington.edu/~sarakim/oralhealth.htm). Our study focused on fifty-six sites, all of which were still accessible as of April 1, 2004. These sites originated from non-governmental organizations (NGOs), professional associations, and foundations from both dental and medical communities (28.6 percent); regional/state agencies (21.4 percent); the federal government (19.6 percent); academia encompassing undergraduate training through continuing education (19.6 percent); and commercial (10.7 percent) sources.

We developed a fifty-two item evaluation instrument to review two areas: content and web design features. Eighteen content areas were identified by two content experts (PJL, WEM) (see Table 1). We identified thirty-four web design features, including interface design (seven items), site context (nine items), use of visual resources (nine items), teaching of procedural skills (four items), and self-assessment (five items). Our evaluation placed a special emphasis on features integrated to enhance visual learning. This emphasis reflects the effective roles visual resources can play compared to text in mediating complex information and in promoting long-term retention.^13,14 Content areas were identified based on focus groups with dental and medical practitioners.² Two experts (PJL, WEM) reviewed content areas in the websites. Two other members (SK, CS) reviewed items pertaining to website design features. We calculated the inter-rater reliability by summing the number of items agreed upon by reviewers for presence or absence of content or design features and dividing the sum by the total number of items in a category. The level of agreement was 75 percent between the two experts who reviewed the content based on 20 percent of sites randomly selected from fifty-six sites. The inter-rater reliability in identifying design features was 88 percent (81 percent for both interface design and site context, 78 percent for features supporting visual resources, 100 percent for both procedural skills and self-assessment). Twenty percent of the websites were randomly selected each time for calculating the inter-rater reliability for individual design categories.

	Results

Top Abstract Methods Results Discussion Conclusions References

 
Content Areas
Table 2 presents the overall percentage of content coverage and specific topic areas covered in the reviewed sites. The range of content coverage varied from 5.6 percent to 83.3 percent out of eighteen oral health content areas. Overall, commercial websites covered the highest number of topic areas (58.3 percent) compared to academic institutions (37.4 percent), NGOs and associations (33.7 percent), federal government (26.8 percent), and regional/state organizations (24.5 percent). Sites that included the majority of the eighteen content areas were Proctor and Gamble (83.3 percent), Simple Steps to Better Dental Health (72.2 percent), Evidence-Based Dentistry (77.8 percent), World Health Organization (66.7 percent), American Dental Association (61.1 percent), and Columbia University School of Dentistry (61.1 percent). No site included all of the identified content areas. While the majority of the reviewed sites covered content areas in anticipatory guidance (66.1 percent), caries (64.3 percent), and the role of fluorides (62.5 percent), materials related to risk assessment (19.6 percent), oral screening exam (17.9 percent), cultural issues (7.1 percent), and dental/medical interface (5.4 percent) were lacking.

Site Context.
The site context scores ranged between 22.2 percent and 100 percent. Commercial sites attained the highest score of 75.9 percent, followed by federal government sites (66.7 percent), NGOs and associations (54.2 percent), academia (53.5 percent), and regional/state agencies (47.2 percent). The majority of the sites specified institutional identifications on the home page (96.4 percent) and included contact information for user inquiry (87.5 percent), date of information posting (76.8 percent), user feedback form (64.3 percent), references to support cited evidence (57.1 percent), and authors’ names (48.2 percent). Several sites noted that the design of the websites adhered to published guidelines, such as Health on Net (HON) (12.5 percent) and provided customized content for professionals and patients, respectively (8.9 percent). Features more frequently incorporated by federal government sites than other sites included user feedback forms (90.9 percent) and references to published work (81.8 percent).

Features Supporting Visual Resources.
Of the fifty-six reviewed sites, thirty-nine (69.6 percent) used visual resources, including photos, video clips, clinical images, and drawings. Of these thirty-nine sites, twenty-eight (five commercial, three federal government, eight NGOs and associations, six regional/state agencies, and six academic institutions) incorporated features to enhance user interactions with visual resources. Commercial sites embedded many more features (31.5 percent) than academic institutions (14.1 percent) and NGOs and associations (11.1 percent). Labels/annotations of images (33.9 percent) and descriptive text interpreting visual features (21.4 percent) were the predominant design components observed in the sites. A few sites showed animations of images (14.3 percent), sequenced still images for illustrating the progression of pathology (14.3 percent), and compared normal and abnormal examples (12.5 percent). Few sites allowed users to zoom images for closer view (7.1 percent) or placed images in proximity to facilitate visual comparison (5.4 percent). Only one site included images showing oral lesions from multiple views (front and rear).

We highlight a site, Simple Steps to Better Dental Health developed by Aetna Corp. (www.simplestepsdental.com/), that offers various interactive tools and illustrations to aid a user’s understanding of tooth eruption at different ages, sequential stages of dental caries, and progression of periodontal (gum) disease. The site uses clearly annotated and descriptive texts that accompany images for illustrating the carious process beginning with a healthy tooth and showing the development of decay involving enamel, then dentin, and finally pulpal involvement. Examples of periodontal diseases are illustrated by comparing pictures of healthy gums and gingivitis. These examples are accompanied by text that describes salient visual features characterizing the extent of diseases. Examples are placed in close proximity to afford an immediate comparison without involving scrolling of the screen. Users can also roll the mouse over predefined hot zones in the image to display the names of the surfaces of the tooth.

Teaching of Procedural Skills.
Eleven sites out of fifty-six (19.6 percent) provided modules for teaching procedural skills, such as placement of dental sealants, application of fluoride varnish, or oral screening examination. Nine of these sites included images or video clips for illustrating the procedures, and two sites described the procedures using text without images. Five sites displayed images in a sequence that showed steps involved in the procedures. One site (Nova Scotia Dental Association) made available still images from a video clip that illustrated each procedural step with an option to play the video clip through the whole procedure.

Self-Assessment of Knowledge.
Only four out of the fifty-six sites (7.1 percent) included self-assessment of knowledge (National Maternal and Child Oral Health Resource Center, British Dental Association, Phoenix Continuing Dental Education, and University of California, Los Angeles, School of Dentistry). Two sites used still images in the questions, and two sites provided feedback on the correctness of submitted responses with further explanations. None of the sites tracked users’ scores or provided a summary of the performance at the end of the test.

	Discussion

Top Abstract Methods Results Discussion Conclusions References

 
We conducted this evaluation of existing websites in oral health in order to understand key content areas covered in these sites as well as to identify website design features embedded to augment user interactions with the content for potential use in educating family physicians. While the spectrum of websites reviewed in this study may be varied, our evaluation illustrates for readers how specific content areas and a range of web design features are incorporated in the sites we evaluated. We note several limitations of this study. First, we did not seek to assess the accuracy or completeness of the content materials. We reported proportions of topics covered in the reviewed web resources as a way to determine areas of oral health that are available to the public and providers over the Internet. Second, we did not seek to assess the appropriateness or effectiveness of the methods the reviewed sites used to deliver content materials. Our attempt was to highlight design features of our target interest and report the sites that included these features. Third, the aggregate number of website design features does not necessarily translate into the quality of presentation of these resources. We did not judge the stylistic quality of web design, including color scheme, font type, font sizes, and other aesthetic considerations. Fourth, there may be websites not captured in our evaluation in spite of our search strategies for identifying sites for review. Finally, we recognize that the evaluation criteria we developed and applied in this study are specific to our review purposes. Therefore, scores we assigned to the reviewed sites do not reflect the sites’ success in achieving their own goals and objectives.

Overall, commercial sites included the highest number of content areas and web design features. Federal government and regional/state agencies covered less than 30 percent of the targeted eighteen topic areas we sought to identify in these resources. The majority of the reviewed sites used features to help users navigate the content and understand the context of the sites. We note that NGOs and associations, regional/state organizations, and academic institutions used less than 30 percent of the web design features we tracked. Many factors, particularly costs, may explain the reasons behind the low usage of key web design features in academic sites. The increased call for web-based educational resources may compel these organizations to incorporate design features in their sites that are accessible to the public.

Our review pointed to a major gap in the use of visual resources for posting didactic information, demonstrating procedural skills, and assessing user knowledge. Similar results were reported in a study that reviewed 157 online continuing dental education courses.¹⁵ These results would indicate that attention to instructional design principles and review of the literature in visual learning could guide web developers in the use of design features to augment and enhance user interactions with content materials. While the selected design principles we list below may not apply to some oral health sites, we recommend these principles to complement the existing web design guidelines we outlined earlier in this article:

Principle 1: Compare and contrast visual examples from the same and different pathology to help transfer of learning when encountering a novel example.

Generalization of an example to the same pathology is facilitated when similar or overlapping critical attributes in an example are highlighted. Generalization to a different pathology is aided by focusing on discriminating visual features across examples.¹⁶

Principle 2: Sequence images for teaching simple to complex concepts (tooth numbering, anatomy) and for illustrating steps in procedural skills (fluoride varnish application, oral cancer screening, treatment of dental emergencies).

Learners can best process multiple images presented in order from easy to difficult and that remain in full view to reduce cognitive overload from having to remember key features in individual images.¹⁷

Principle 3: Place images in close proximity for visual integration.

The principle of proximity compatibility posits that perceptual configuration and conceptual interpretation improve when visual inputs are displayed close in space.¹⁸

Principle 4: Annotate images with labels and provide descriptive text.

Verbal interpretation of salient features and meaning of images can help learners construct a visual schema (such as a prototypical image) or cognitive schema (a network of knowledge) that can guide feature selection, configuration, and other perceptual organization.¹⁹

Principle 5: Accentuate pathology shown in images with distinct borders in color.

Boundaries around critical visual attributes facilitate a learner’s efficient orientation to visual stimuli.^16,19

Principle 6: Allow learners to zoom images.

Zooming of an image to different levels of magnification and resolution helps minimize perceptual ambiguity.^6,19

Principle 7: Use animations for appropriate learning purposes.

Animations can best support the following learning aspects: 1) sequential actions in procedural skills; 2) cause and effect of complex behaviors; 3) revealing system functions, such as a progression of oral pathology; 4) analogy or guidance involving motion; and 5) illustrating a task that is difficult to describe accurately with text.^4,20

Principle 8: Add narration, text, and visual cues to supplement animations.

Providing an animated depiction of a phenomenon alone may not be sufficient for learners to acquire coherent and comprehensive knowledge structures.²¹ Learning can be better facilitated if animations are accompanied by narration, text, and visual cues. In addition, the ability to pause, repeat, and change the speed of animations is a helpful learning tool.

The following two principles, though not related to visual learning principles, are added because of their importance to the design of web resources:

Principle 9: Match assessment format with didactic format.

If visual resources, such as images, videos, and animations, are used in the didactic materials, then comparable resources should be used to assess learners’ knowledge. There is a need for development and validation of methods to test learning outcomes that are fundamentally visual in nature and difficult to access via verbal mode.^4,6,22

Principle 10: Annotate content of the external sites linked from a site.

When making content available from external links to web resources, provide a brief annotation of the external site to orient users to the content before they visit the site.

	Conclusions

Top Abstract Methods Results Discussion Conclusions References

 
Our review of existing web resources has yielded a number of design principles that can be applied to the development of future online educational modules in oral health. The proliferation of web resources in dental and medical education makes a comprehensive review challenging to conduct. However, it also allows educators an opportunity to critically evaluate the scope of content and format of interface design in existing resources before developing comparable online resources in their expert domains. As web-based educational modalities serve as an increasingly important vehicle for delivering dental and medical educational content, both educators and web developers will benefit from an expanding knowledge base of designing educationally effective web resources.

	Acknowledgments

 
This work was partially supported by funding from the U.S. Health Resources and Services Administration’s Bureau of Health Professions (cooperative agreement #8 U7 HP 00026-01).

	REFERENCES

Top Abstract Methods Results Discussion Conclusions References

 

1. Oral health in America: a report of the surgeon general—executive summary. Rockville, MD: U.S. Department of Health and Human Services, National Institute of Dental and Craniofacial Research, National Institutes of Health, 2000. At: www.nidcr.nih.gov/sgr/sgr.htm. Accessed: May 7, 2004.
2. Mouradian WE, Schaad DC, Kim S, Leggott PJ, Domoto PS, Maier R, et al. Addressing disparities in children’s oral health: a dental-medical partnership to train family practice residents. J Dent Educ 2003;67(8):886–95.[Abstract]
3. Rozier RG, Sutton BK, Bawden JW, Haupt K, Slade GD, King RS. Prevention of early childhood caries in North Carolina medical practices: implications for research and practice. J Dent Educ 2003;67(8):876–85.[Abstract]
4. American Dental Association’s Guidelines for the Design of Educational Software. At: www.dental.pitt.edu/informatics/edswstd/. Accessed: May 7, 2004.
5. Health on the Net. At: www.hon.ch/HONcode/Conduct.html. Accessed: May 7, 2004.
6. The Quality Criteria for Electronic Publications in Medicine. At: www.imbi.uni-freiburg.de/medinf/gmdsqc/e.htm. Accessed: May 7, 2004.
7. Johnson LA, Schleyer T. Development of standards for the design of educational software. Quintessence Int 1999;30(11):763–8.[Medline]
8. Mayer RE, Moreno R. Aids to computer-based multimedia learning. Learning and Instruction 2003;12:107–19.
9. Eysenbach G, Powell JP, Juss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the World Wide Web. JAMA 2002;287:2691–700.[Abstract/Free Full Text]
10. Gagliardi A, Jadad AR. Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. BMJ 2002;324(7337):569–73.[Abstract/Free Full Text]
11. Frasca D, Malezieux R, Mertens P, Neidhardt JPH, Voiglio EJ. Review and evaluation of anatomy sites on the Internet (updated 1999). Surg Radiol Anat 2000;22:107–10.[Medline]
12. Kim S, Brinkley JF, Ross C. A profile of online anatomy information resources: design and instructional implications. Clin Anat 2003;16(1):55–71.[Medline]
13. Anglin GJ, Towers RL, Levie WH. Visual message design and learning: the role of static and dynamic illustrations. In: Jonassen DH, ed. Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan, 1996.
14. West CK, Farmer JA, Wolff PM. Instructional design: implication from cognitive science. Englewood Cliffs, NJ: Prentice Hall, 1991.
15. Schleyer T, Johnson LA, Pham T. Instructional characteristics of online continuing education courses. Quintessence Int 1999;30(11):755–62.[Medline]
16. Park I, Hannafin MJ. Empirically based guidelines for the design of interactive multimedia. Educ Tech Res Development 1993;41(3):63–85.
17. Sweller J, Chandler P. Why some material is difficult to learn. Cognition and Instruction 1994;12(3):185–233.
18. Wickens CD, Andre AD. Proximate compatibility and information display: effects of color, space, and objectness on information integration. Hum Factors 1990;32(1):61–71.[Medline]
19. Kundel HL, Nodine CF. A visual concept shapes image perception. Radiology 1983;146:363–8.[Abstract/Free Full Text]
20. Park O, Hopkins R. Instructional conditions for using dynamic displays: a review. Instructional Science 1993;21:427–49.
21. Lowe RK. Animation and learning: selective processing of information in dynamic graphics. Learning and Instruction 2003;13:157–76.
22. Friedman CP, McNutt RA, Bliek RD, Evans AT, Wallsten TS, Martz JM. Visual information and the diagnosis of chest pain. Acad Med 1994;69(10):S28–S30.[Medline]

This article has been cited by other articles:

				W. Hendricson, E. Eisenberg, G. Guest, P. Jones, L. Johnson, F. Panagakos, J. McDonald, and L. Cintron What Do Dental Students Think About Mandatory Laptop Programs? J Dent Educ., May 1, 2006; 70(5): 480 - 499. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Kim, S. Articles by Shaul, C. Search for Related Content PubMed PubMed Citation Articles by Kim, S. Articles by Shaul, C.