International Journal of Medical Informatics
Volume 77, Issue 6 , Pages 377-385 , June 2008

Investigating evaluation frameworks for health information systems

  • Maryati Mohd. Yusof

      Affiliations

    • Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia
    • Corresponding Author InformationCorresponding author. Tel.: +603 89216649.
    • ,
  • Anastasia Papazafeiropoulou

      Affiliations

    • School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK
    • ,
  • Ray J. Paul

      Affiliations

    • School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK
    • ,
  • Lampros K. Stergioulas

      Affiliations

    • School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK

Received 15 December 2005 ,Revised 12 August 2007 ,Accepted 12 August 2007.

References 

  1. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: trends in evaluation research 1982–2002. Methods Inf. Med. 2005;44:44–56
  2. Ammenwerth E, Brender J, Nykanen P, Prokosch H-U, Rigby M, Talmon J. HIS-EVAL workshop participants. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck.. Int. J. Med. Inf. 2004;73(6):479–491
  3. Moehr JR. Evaluation of health information systems: beyond efficiency and effectiveness. Comput. Biol. Med. 2002;32(3):111–112
  4. Friedman CP, Wyatt JC. Evaluation Methods in Medical Informatics. New York: Springer-Verlag; 1997;
  5. Wyatt JC, Wyatt SM. When and how to evaluate health information systems?. Int. J. Med. Inf. 2003;69(2–3):251–259
  6. Ammenwerth E, Graber S, Hermann G, Burkle T, Konig J. Evaluation of health information systems—problems and challenges. Int. J. Med. Inf. 2003;71:125–135
  7. Yusof M Mohd, Paul RJ, Stergioulas L. Health information systems evaluation: a focus on clinical decision supports system. In: Proceedings of the XIX International Congress of the European Federation for Medical Informatics (MIE2005). Geneva, Switzerland. 2005;p. 855–860
  8. Gremy F, Fessler JM, Bonnin M. Information systems evaluation and subjectivity. Int. J. Med. Inf. 1999;56:13–23
  9. Van der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J. Am. Med. Inform. Assoc. 2003;10(3):235–243
  10. Lederer AL, Salmela H. Towards a theory of strategic information systems planning. J. Strategic Inf. Syst. 1996;5:237–253
  11. Lucas HC. The Analysis, Design, and Implementation of Information Systems. Watsonville: Mitchell McGraw Hill; 1992;
  12. Martin C, Powell P. Information Systems: A Management Review. Berkshire: Mc Graw Hill Book Company; 1992;
  13. A. Olerup, On a Contingency Framework of Computerised Information System, In: J. Lucas H.C., F.F. Land, T.J. Lincoln, K. Supper (Eds.), The Information Systems Environment, Amsterdam, 1980.
  14. Wetherbe JC. Executive 's Guide to Computer Based Information Systems. New Jersey: Prentice Hall, Inc.; 1983;
  15. Smith J. Health Management Information Systems: A Handbook for Decision Makers. Buckingham: Open University Press; 2000;
  16. In:  Van Bemmel JH,  Musen MA editor. Handbook of Medical Informatics. Heidelberg: Springer-Verlag; 1997;
  17. Gross PA, Bates DW. A pragmatic approach to implementing best practices for clinical decision support systems in computerized provider order entry systems. J. Am. Med. Assoc. 2007;14(1):25–28
  18. Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al. Medication-related clinical decision support in computerized order entry systems: a review. J. Am. Med. Inform. Assoc. 2007;14(1):29–40
  19. Kuhn KA, Giuse DA, Talmon JL. The Heidelberg Conference: setting an agenda for the IMIA working group on health information systems. Int. J. Med. Inf. 2003;69(2–3):77–82
  20. M. Williamson, How do I evaluate user acceptance of an electronic decision support system? Australian Health Information Council, available from http://www.ahic.org.au/evaluation/user.htm, 2004 (cited 23 December 2004).
  21. Beyer HR, Holtzblatt K. Apprenticing with the customer. Commun. ACM. 1995;38:45–52
  22. Goodhue DL. Understanding user evaluations of information systems. Manag. Sci. 1995;41:1827–1844
  23. Willcocks L. Managing technology evaluation—techniques and processes. In:  Galliers RD,  Baker BSH editor. Strategic Information Management: Challenges and Strategies in Managing Information Systems. Oxford: Butterworth Heinemann; 1994;p. 365–381
  24. Clarke K, O’Moore R, Smeets R, Talmon J, Brender J, McNair P, et al. A methodology for evaluation of knowledge-based systems in medicine. Artif. Intell. Med. 1994;6(2):107–121
  25. Brender J, Talmon J, Nykanen P, McNair P, Demeester M, Beuscart R. In:  van Gennip EMSJ,  Talmon J editor. Assessment and Evaluation of Information Technologies in Medicine. vol. 17:Amsterdam: IOS Press; 1995;p. 189–208
  26. Brender J. Methodology for Assessment of Medical IT-Based Systems, vol. 42. Amsterdsam: IOS Press; 1997;
  27. Kaplan B. Organizational evaluation of medical information reources. Evaluation Methods in Medical Informatics. New York: Springer-Verlag; 1997;pp. 255–280
  28. Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information systems evaluation: guidance for the organizational decision maker. Methods Inf. Med. 2003;42(4):458–462
  29. Jorgensen T. In:  van Gennip EMSJ,  Talmon J editor. Assessment and Evaluation of Information Technologies in Medicine. vol. 17:Amsterdam: IOS Press; 1995;p. 99–110
  30. Kaplan B, Shaw N. People, organizational and social issues: evaluation as an Exemplar. In:  Haux R,  Kulikowski C editor. Yearbook of Medical Infomatics. 2002;pp. 91–102
  31. Coiera E. Guide to Health Informatics. Hodder Arnold; 2003;
  32. Currie LM. Evaluation frameworks for nursing informatics. Int. J. Med. Inf. 2005;74:908–916
  33. Kaplan B. Evaluating informatics applications—clinical decision support systems literature review. Int. J. Med. Inf. 2001;64(1):15–37
  34. In:  van Gennip EMSJ,  Talmon J editor. Assessment and Evaluation of Information Technologies in Medicine. Amsterdam: IOS Press; 1995;
  35. Yusof M Mohd, Paul RJ, Stergioulas LK. Towards a framework for health information systems evaluation. In: Proceedings of the 39th Hawaii International Conferences on System Sciences. Kauai, Hawaii, USA. 2006;
  36. Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library—resource projects to increase what is learned. J. Am. Med. Inform. Assoc. 1994;1(1):28–33
  37. Shaw NT. CHEATS: a generic information communication technology (ICT) evaluation framework. Comput. Biol. Med. 2002;32(3):209–220
  38. Grant I, Plante F, Leblanc . The TEAM methodology for the evaluation of information systems in biomedicine. Comput. Biol. Med. 2002;32(3):195–207
  39. Westbrook JI, Braithwaite J, Iedema R, Coiera EW. Evaluating the impact of information communication technologies on complex organizational systems: a multi-disciplinary, multi-method framework. In: Proceedings of the Medinfo’04 Congress. San Francisco, CA, USA. 2004;p. 1323–1327
  40. Dixon DR. The behavioral side of information technology. Int. J. Med. Inf. 1999;56:117–123
  41. Kazanjian A, Green CJ. Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework. Comput. Biol. Med. 2002;32:165–177
  42. Anderson JG. Evaluation in health informatics: social network analysis. Comput. Biol. Med. 2002;32:179–193
  43. Berg M. Patient care information systems and health care work: a sociotechnical approach. Int. J. Med. Inf. 1999;55(2):87–101
  44. G.L. Glandon, T.L. Buck, Cost-benefit analysis of medical information systems: a critique, in: J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.). Evaluating Health Care Information Systems: Methods and Applications, Newbury Park, 1994, pp. 164–89.
  45. Jiang JJ, Muhanna WA, Klenin G. User resistance and strategies for promoting acceptance across system types. Inf. Manag. 2000;37(1):25–36
  46. R.M. Gardner, M.M. Shabot, Patient-monitoring Systems, in: E.H. Shortliffe, L.E. Perreault, G. Wiederhold, L.M. Fagan (Eds.). Medical Informatics: Computer Applications in Health care and Biomedicine, New York, 2001, pp. 443–484.
  47. Parrino T. Information technology and primary care at the VA: making good things better. Health Serv. Res. Dev. 2003;
  48. Gawande AA, Bates DW. The use of information technology in improving medical performance—part II. Physician-support tools. Med. Gen. Med. 2000;2(1):
  49. Randolph AG, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH. Users’ guides to the medical literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. JAMA. 1999;282(1):67–74
  50. Hunt D, Haynes RB, Hanna S, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA. 1988;280(15):1339–1346
  51. Kaplan B. Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int. J. Med. Inf. 2001;64(1):39–56
  52. Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf. Med. 2004;43(3):215–231
  53. W.H. DeLone, McLean, E.R., Information systems success: the quest for the dependent variable information systems research 3 (1)(1992) 60–95.

PII: S1386-5056(07)00159-1

doi: 10.1016/j.ijmedinf.2007.08.004

International Journal of Medical Informatics
Volume 77, Issue 6 , Pages 377-385 , June 2008

Article Tools