P121 - Through a glass darkly: evaluation of HIV/AIDS Website reach; usage and impact
The Canadian AIDS Treatment Information Exchange (CATIE)'s bilingual Website (CATIE.ca) is a leading source of HIV/AIDS treatment information, having won international recognition from Médisite, Britannica's Internet Guide, the Lancet, and JAMA. With over 10,000 pages of content, CATIE.ca is Canada's largest online archive of HIV/AIDS treatment information. In order to assess Website usage, reach and impact, CATIE conducted evaluation research using three methods: 1) aggregate server log analysis (1996-2003); 2) Website user survey (2003, n=263); and 3) key informant interviews (2002, n=41). Server log analysis provided indicators of site usage (24,000 user sessions/month), content access (most popular content: fact sheets and news), and historical usage trends. Challenges included defining 'page', excluding non-person visits, and interpreting ISP address labels. The Web survey provided indicators of who uses the site (36% people living with HIV/AIDS, 39% community caregivers, 14% health care providers), the ir geographical origin, how information is used (44% managing their health, 29% sharing with family/friends, 28% professional development, 23% research projects, 16% giving to clients), and site helpfulness ( & gt;86% found the site helpful or very helpful). Extracting meaningful information involved sampling challenges. Comparing survey results to log file analysis on overlapping indicators allowed independent verification. Both the user survey and key informant interviews found that users want: quality information, interactivity, regular updates, easy navigation and targeted information. Due to inherent limitations in each methodology, multiple methods are needed to fully grasp reach, usage and impact of a Website. Mixed methods also give opportunities for triangulating methods in order to increase confidence. CATIE’s experience has shown the ability of mixed evaluation methods to inform future Website development, and also offers useful benchmarks for other Websites.