Volume 77, Issue 6, Pages 377-385 (June 2008)

2 of 7

ABSTRACT

FULL TEXT

FULL-TEXT PDF (190 KB)

Investigating evaluation frameworks for health information systems

Received 15 December 2005; received in revised form 12 August 2007; accepted 12 August 2007. published online 30 January 2009.

Abstract

Background and purpose

Evaluation of health information systems (HIS) enables the assessment of the extent to which HIS are fulfilling their objectives in supporting the services of healthcare delivery. This paper presents an overview of evaluation in health informatics and information systems.

Methods

Literature review on discourses, dimensions and methods of HIS and IS evaluation. A critical appraisal of selected HIS and IS evaluation frameworks is undertaken in order to identify HIS evaluation dimensions and measures. The frameworks are compared based on their inclusion of human, organizational and technological factors.

Results

We found that an increasing number of evaluation studies deal with two distinct trends of HIS: one considers human and organizational issues and the other is concerned with the employment of a subjectivist approach. Our review indicates that current evaluation methods complement each other in that they evaluate different aspects of HIS and they can be improved upon.

Conclusions

Evaluation is complex; it is easy to measure many things but not necessarily the right ones. Nevertheless, it is possible to consider, a HIS evaluation framework with more comprehensive and specific measures that would incorporate technological, human and organizational issues to facilitate HIS evaluation.

Keywords: Information systems, Health information systems, Evaluation, Human factors, Organizational factors

Article Outline

• Abstract

• 1. Introduction

• 2. An overview of health information systems

• 3. Health information systems evaluation

• 3.1. Early approaches to health information system evaluation

• 3.1.1. The who

• 3.1.2. The when

• 3.1.3. The what

• 3.1.4. The how

• 4. HIS evaluation frameworks

• 4.1. Locating evaluation frameworks

• 4.2. Critical appraisal of evaluation frameworks

• 4.2.1. Generic evaluation frameworks

• 4.2.2. System development life cycle based evaluation frameworks

• 4.2.3. Socio-technical based evaluation frameworks

• 5. Discussion and conclusions

• Acknowledgment

• References

• Copyright

1. Introduction

Researchers and practitioners in the health informatics field recognize the importance of the evaluation of HIS. The focus of such evaluations is moving from technical issues to human and organizational issues (trend 1); and from an objectivist to a subjectivist approach (trend 2) [1].

The extent to which HIS fulfil their role and support the services of healthcare delivery is obviously important. Ammenwerth et al. [2] defined HIS evaluation as “the act of measuring or exploring attributes of a HIS (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context”. This definition outlines three key issues: measuring, attributes of HIS and the support of decision making. Undertaking the evaluation is challenging as the decision making in design, development, purchase or management in HIS all requires evaluation [3].

It is claimed that HIS evaluation is not straightforward and a number of problems pose challenges to its evaluators, which are partly due to HIS complexity [4], [5], [6]. HIS evaluation seeks to answer the why, who, when, what and how questions relating to technological, human and organizational issues surrounding it [7]. In addition, HIS evaluation is unclear and confusing [8] and it is argued that an existing strong foundation for good evaluation theory and practice is yet to be disseminated in an understandable form [2]. This may explain why despite an increasing number of HIS being developed, the number of published evaluations is very limited [2], [9]. Evaluation of HIS is also difficult to perform, particularly in selecting a framework to be applied and methods to be used [2]. However, there are a number of proposed approaches that can be adopted/adapted to overcome these problems [2], [4] as well as deriving some more improved methods and extensions [3].

This paper reviews the discourses, dimensions and methods of HIS evaluation described in the wider health informatics and information systems literature. The review covered HIS evaluation studies as well as some related IS studies, studies of human and organizational factors in HIS and existing HIS evaluation frameworks. In order to present this review, the paper is organized as follows. The following section provides an overview of HIS and their classification. Section 3 presents a classification of early evaluation studies that highlights the main evaluation themes and concerns. This discussion is followed by a critical analysis section (Section 4) examining the existing frameworks for HIS evaluation. These frameworks have been selected for their intent to evaluate HIS from different perspectives. Some preliminary discussion, together with conclusions, closes this paper, and sets the scene for another paper (part II–current issue), which introduces a framework, that arose naturally from follow up research to the present work.

2. An overview of health information systems

In order to explain what an information systems (IS) is, the paper adopts the following understanding of what constitutes a system and information. According to Lederer and Salmela [10], input, processing and output are elements that form a system. A system also comprises a combination of variables or components that are interrelated, organized and depends on each other [11]. According to Martin and Powell [12], information is the outcome of data processing and is used to aid in decision making. Lucas [11] defines IS as a number of procedures organized to facilitate decision making, communication and control in organizations. Hence, IS assists organizations to gather, process and disseminate information within the organization and their environment [13]. Wetherbe [14] regards IS as a physical process, which supports system objects in achieving organizational goal. Based on the above definitions, in this paper, an information system is defined as a group of interrelated processes implemented to aid in enhancing the efficiency and effectiveness of an organization in performing its functions and attaining its objectives.

For the purpose of this review, the term health information systems is used to refer to computer based information systems used in healthcare settings. HIS are used extensively in healthcare organizations to support various conventional data processing tasks including patient billing, accounting, inventory control, statistics calculation and patient history maintenance (see Table 1). They are also used for scheduling, automating nurse stations, monitoring intensive care patients and providing preliminary diagnoses [15].

Table 1.

Classification of health information systems


Information systems	Descriptions	Characteristics	Author(s)
Patient centered information systems	They are the electronic version of patients’ information. Different terms are used to refer to these systems including electronic patient record (EPR), electronic medical record (EMR) and computer based patient record (CPR)	• Manage comprehensive patient care information such as medical records, appointment scheduling, theatre management and ward reporting	Smith [15]
		• Have entry and retrieval functions for medical records and clinical procedures

Administrative information systems	Record the main business processes and routine transactions of organizations such as patient admission, discharge and transfer, bill processing, reporting and other management purposes.	• May constitute accounting subsystems, financial subsystems, inventory subsystems, equipment subsystems and general management subsystems tailored to the clinical environment	Smith [15]; Glandon and Buck [44]; Jiang et al. [45]

Clinical information systems (CIS)	Represent separate systems in specialized service of clinical departments. Examples of CIS include patient monitoring systems and anesthesia documentation system	• Perform specific tasks including collection of specific data for patient care, research, management, planning and maintenance of national data repositories	Ammenwerth and de Keizer [1]; Smith [15]; Van Bemmel and Musen [16]; Gardner and Shabot [46]
		• Specific tasks operate in departments such as internal medicine, cardiology, neurology, obstetrics, surgery and psychiatry
		• CIS are used for administrative support, patient data collection, decision support, picture archiving, image analysis, monitoring, reporting, assessment and research

Radiology information systems	Support the acquisition and analysis of radiological images as well as administrative functions of radiology department. Example: picture archiving and communication systems (PACS)	• May be stand alone or integrated in hospital information systems	Van Bemmel and Musen [16]

Laboratory information systems	Perform data validation, administration, electronic transmission and computer storage	• In high demand when a large number of tests generate large data. Samples are analyzed fully automatically, and the results are computer generated	Van Bemmel and Musen [16]
		• Support clinician to analyze trends to assess treatment effects

Pharmacy information systems	Maintain medication information	• Include functions such as keeping patients’ medication records, checking prescriptions, and providing drug prescriptions and administration to physicians and nurses	Van Bemmel and Musen [16]

Telemedicine	Telemedicine provides and supports healthcare services and education across distances via electronic communications and IT	• Facilitates exchange between primary care physicians and specialists as well as patients from disperse locations	Smith [15]; Parrino [47]; Gawande and Bates [48]
		• “Allows physicians to practice medicine at a distance”

Clinical decision support systems	Designed specifically to aid clinical decision making	• Common functions: alerting, reminding, critiquing, interpreting, predicting, diagnosing, assisting and suggesting	Randolph et al. [49]; Hunt et al. [50]

Hospital information systems	Consist of integrated hospital information processing systems. Examples: computerized physician order entry (CPOE) (which are also referred to as computerized provider order entry), patient care information systems, nursing (bedside) documentation systems, nursing IS, general practitioner IS	• Support healthcare activities at the operational, tactical and strategic levels	Ammenwerth and de Keizer [1]; Van der Meijden et al. [9]; Smith [15]; Van Bemmel and Musen [16]
		• Encompass patient management, administration, facilities management and medical applications
		• Contain database systems, data communication facilities and terminal or workstations

HIS range from simple systems, such as transaction processing systems, to complex systems, such as clinical decision support systems (CDSS). The health informatics literature defines in various terms different types of HIS and these terms are applied inconsistently. Hence, the classification of different types of HIS is offered in Table 1.

Patient centered information systems are the core system in healthcare organizations; they are usually linked to other HIS to provide patients’ information and their medical history. Clinical information systems are designed uniquely according to each clinical department. A number of systems are identified as clinical support information systems including radiology information systems, laboratory information systems and pharmacy information systems [16]. Hospital information systems is a general term that spans a variety of hospital information processing system types. For example, clinical physician order entry (CPOE) systems are gaining popularity and are commonly integrated with CDSS to support basic decision making (drug-allergy checking, basic dosing guidance, formulary decision support, duplicate therapy checking and drug–drug interaction checking), as well as advanced decision making (such as guidance for medication-related laboratory testing, drug-pregnancy checking and drug-disease contraindication checking) [17], [18]. Whatever type of HIS, an effective evaluation technique to assess their appropriateness to their organization could be useful. In the next section, we present a number of evaluation frameworks that have been used for this purpose.

3. Health information systems evaluation

Evaluation serves a number of purposes. Given the unpredictable characteristics of IS in general and the aim of improving clinical performance and patient outcomes in particular, evaluation is undertaken to understand system performance [16]. Potentially, the evaluation of health informatics application can help improve the quality of care and its costs and to determine the safety and effectiveness of HIS [16], [19]. Evaluation can be used to improve HIS through using past experience to identify more effective techniques or methods, investigate failure and learn from previous mistakes [4].

3.1. Early approaches to health information system evaluation

As mentioned above, evaluation seeks to answer the why (objective of evaluation), who (which stakeholders’ perspective is going to be evaluated), when (which phase in the system development life cycle), what (aspects or focus of evaluation) and how (methods of evaluation) questions. A discussion of early studies on HIS evaluation is presented based on these evaluation questions and a summary presented in Table 2. Due to its relative popularity, there are a large number of evaluation studies on clinical decision support systems (CDSS).

Table 2.

Early studies on HIS evaluation


Author(s)	Theme	Findings/conclusions
Ammenwerth et al. [2] Ammenwerth et al. [6]	Problems and challenges of HIS evaluation	Research in health informatics evaluation is still at its infancy and what constitutes ‘good’ HIS is still unclear. It seems desirable to have a broadly accepted, detail evaluation framework that could guide researcher to undertake evaluation studies.
Moehr [3]	Comparison between objectivist and subjectivist approach	Subjectivist approach has advantages over the limitations of objectivist approach.
Kaplan [51]	Critiques for randomized controlled clinical trials (RCT) and experimental approaches	The limitations of RCT/experimental approaches to evaluation call for alternative approaches that address contextual issues such as social interactionist.
Clarke et al. [24]; Brender [25]; Brender [26]	Methodology for the evaluation of CDSS	Four iterative phase development evaluation cycle for CDSS have been identified. CDSS evaluation should start at the system conception while its integration in system development should ensure a more comprehensive evaluation, alert possible causes for failure, and thereby avoid wasted time and effort.
Hunt et al. [50]	Review the effects of CDSS on physician performance and patient outcome based on the assessment of RCT	There is a rapid increase in published CDSS studies with improved quality. The benefits of CDSS in enhancing clinical performances can be seen in drug dosing, preventive care and other aspects of health care but not convincingly in diagnosis. The studies on CDSS effects on patient outcomes are limited.
Kaplan [33]	Review CDSS literature concerning evaluation	Although CDSS is acknowledged for its potential to improve care, evidence is unclear in its diagnostic function. There is a general consensus on limited use of CDSS despite its proven or potential benefits. Most studies use experimental or RCT approach but very few studies involve field tests and almost none take place in actual clinical settings. Most studies focus on physicians and exclude other clinicians. Studies in understanding issues surrounding development, implementation and use of CDSS are lacking.
Gremy et al. [8]	The importance of human factors in HIS evaluation	Human factors are central to HIS evaluation. Systems that involve human interaction have the greatest failures in contrast with systems that work independently of the user. When it comes to the evaluation of most HIS, it appears that we are still in a “blind alley”.
Kaplan [27]; Kaplan and Shaw [30]; Kaplan and Shaw [52]	Review on human, organizational and social issues in HIS evaluation	Human, organizational and social issues are important to address during system design, implementation and use. Newer evaluation trends are focusing more on these non-technical issues. An evaluation framework based on social interactionist theory is proposed. It is known as 4Cs (communication, care, control, context).
Van der Meijden et al. [9]	Review on success factors of inpatient patient care information systems using DeLone and McLean IS success model	A wide range of attributes could be categorized according to IS success model [53] but some attributes related to IS failure did not match any of the categories. IS success model is applicable in the evaluation of inpatient patient care information systems. More thorough evaluations of patient care information systems can be performed to address factors that contribute to systems’ success and failures.
Currie [32]	Review on evaluation frameworks of health informatics based on user-, context- and functionality-centric, SDLC recognition, theory based and qualitative approach employed	Quantitative and qualitative methods are both rigorous in their own way. The use of qualitative approach is increasing in the evaluation of health informatics. The use of qualitative approach can potentially enhanced user acceptance and ideally avoid system failure.

3.1.1. The who

Evaluation involves many stakeholders who have different views on the systems. Popular types of stakeholders of HIS include developer, user, patient and purchaser. The perceived usefulness of the evaluation results varies for different individual. The potential of HIS to improve patient care and the performance of clinicians is often thwarted by the users’ reluctance to accept and adopt it [20]. Therefore, the usefulness of HIS depends largely on users (customers), because they are the experts in their work, not the developers (designers) [21].

Goodhue [22] employed user evaluation of task-technology fit as a measure of success. His study resulted in two important findings. First, the usefulness of a technology seems to relate to the tasks of the user. It appears that users regard their system as tools, which aid or hinder them in performing their tasks. Users respond positively to system's features that realize task demands. Second, users seem capable of performing the evaluation of the task-technology fit of a particular technology that they have been using. Goodhue's [22] study also indicated that user evaluations could be beneficial to the practitioner as they provide fair, detailed diagnostics of information systems and services.

3.1.2. The when

In general, apart from the feasibility study, IS evaluation can be carried out during the four main phases when using the classical system development life cycle (SDLC)—pre-implementation (development), during implementation, post-implementation or routine operation [23]. In the health informatics domain, four evaluation phases have been identified, which are also based on the SDLC: preliminary, validity, functionality and impact [24], [25], [26]. Each phase addresses specific evaluation aspects. Depending when in the system development life cycle it is done, evaluation can be formative or summative. The aim of formative evaluation is to improve the system under development or during implementation; thus, problems can be identified as they emerge and the system can be improved as it is being developed. Evaluation should start with the system conception while its integration into system development should ensure a more comprehensive evaluation, alert possible causes for failure, and thereby avoid wasted time and effort [24].

On the other hand, the aim of summative evaluation is to assess a system in operation and overall system effectiveness, and to provide information for determining system continuation [27]. Most studies focus on summative aspects; thus, there is limited support by methods and guidelines for constructive (formative) evaluation in system implementation or installation [2].

3.1.3. The what

Many aspects of HIS can be evaluated. Evaluation involves human, technology, organizations and interaction between them [1], [5]. Hence, evaluation can cover technical, professional, organizational, economic, ethical and legal domains [28], [29]. Brender [26] compares evaluation studies in the field of health informatics and in the information systems field and observes that the focus of empirical evaluation studies in the health informatics literature was the aspects of correctness, while in the IS literature, the emphasis was on theoretical and practical technological aspects. The evaluation trend of health informatics has been increasingly shifting towards the human and organizational factors. For example, Kaplan and Shaw [30] presented a number of evaluation studies which concentrated on the following human and organizational issues: organizational readiness, diffusion of innovation, workflow, change management, and human factors, clinical context, cognitive factors, and methods of development and dissemination in determining the system success. Clearly evaluation should address not just how well a system works, but also how well the system works with particular users in a particular setting. Coiera [31] agrees with this view; stressing that the evaluation emphasis is commonly on technical issues or clinical processes although CDSS are frequently justified based on clinical benefit grounds.

3.1.4. The how

Evaluation can be conveniently classified into objectivist and subjectivist approaches [4], [16]. The objectivist approach assumes that everyone agrees, or can be brought to consensus, on what is good and right about important system properties. Numerical measurement, which is preferably derived from experiments such as randomized controlled clinical trials (RCT), is superior in the objectivist approach, compared to verbal description. On the other hand, the subjectivist approach assumes that, “when phenomena involve people and become complex, there is no a single truth about them” [4], leading to various perspectives on what is good and right about different systems and context, among individuals and groups. In contrast with the objectivist approach, verbal description is vital to illustrating these differing perspectives. In objectivist studies, objective assessment of subjects, variables and data collection methods are selected while in subjectivist studies, research are conducted based on the judgements of expert evaluators or system stakeholders in the natural environment of the subjects, without manipulating it, and themes of interview emerge during the study [4], [9].

The subjectivist approach is viewed as being holistic, thorough, rigorous, economical and time efficient as opposed to the objectivist approach, which is viewed as being expensive, time consuming and labor intensive. In addition, “difficulties in conducting objectivist studies … make it difficult to conduct such studies in the first place” [3]. The limitations of the objectivist approach suggest that the subjectivist approach is a better alternative [3], [28], [32]. While objectivist approaches are excellent for examining system performance or particular changes in the behaviors of clinical practice, they are less suitable for investigating why and how a system works with a specific user in a specific setting [33]. Evaluation can be performed using quantitative and qualitative methods or ideally, a combination of both methods; the latter is strongly advocated as it provides a more comprehensive view of the evaluation studies.

Early efforts addressing evaluation methods and their applications have been published in 1990s to guide researchers and practitioners in evaluating IT healthcare applications, particularly imaging systems and knowledge based and decision support systems [34]. In addition to methodology guidelines, [34] also highlights the importance of human and organizational issues, alongside other pragmatic issues like background issues and approaches in different phases.

4. HIS evaluation frameworks

The approaches to HIS evaluation that have been developed are based on one or more domains such as technical, sociological, economic, human and organizational. In the next section, a number of frameworks are reviewed to identify the evaluation dimensions and measures used to evaluate systems in a healthcare setting (see Table 3).

Table 3.

Selected HIS evaluation frameworks


Framework/author(s)	Evaluation aspects
	Technology	Human	Organization
Generic evaluation frameworks
House's multiple approaches to evaluation (Friedman and Wyatt [4])	Information resources	Designers	Clinical environment
	Archetypes	Developers
	Software	Administrators
		Users

System development stage (Stead et al. [36])	System	Users	Social, cultural and functional environment
	Hardware and software infrastructure
	System development stages

CHEATS (Shaw [37])	Technical	Human	Clinical
		Education	Organization
		Social	Administration

System development life cycle (SDLC) based evaluation frameworks
Evaluation methodology for knowledge based systems (Clarke et al. [24]; Brender et al. [25]; Brender [26])	Technical verification	Human-computer interaction	Effect on patient care, health care service, social relations among professionals, organizational structure, legal cases
	Functions completeness and correctness	Stakeholders
	Functionality
	Transferability

Five step evaluation process (Gremy et al. [8])	HIS development stages	Human	General impact
	Machine	Ethics
	Program
	Model
	Aim
	Meaning
	Software
	Data definition and entry
	Output interpretation

TEAM (Grant et al. [38])	IS based on management level	Role	Structure

MEM (Westbrook et al. [39])	Point of care clinical systems	Staff attitude, perception	Organizational structure
	IT/ ICT		Work practices
	Information exchange		Communication

Socio-technical based evaluation frameworks
ITAM (Dixon [40])	IT adoption	Individual user

HTA (Kazanijian and Green [41])	Technology assessment activity	Population at	Population impact
		Risk	Economic concerns
			Social context

Social network analysis (Anderson [42])	Computer use	Network	Network
	System files	Relation	Relation
	HIS	Physician role
	Information dissemination

Socio-technical approach (Berg [43])	Patient Care IS	Network	Health care practices
Socio-technical approach (Berg [43])		Role and task	Workflow

4Cs (Kaplan [27])	HIS and its development impact	Communication	Control, care, context

4.1. Locating evaluation frameworks

In order to do the review, we searched health informatics, information systems, computer science and engineering databases including PubMed, Science Direct, Web of Knowledge, Engineering Village 2 and the ACM digital library. Other sources include textbooks, web search using Google and Yahoo search engines, and citation searching and chaining. From the results retrieved, twelve frameworks have been identified based on the following criterion, which we have defined from our literature review: The featured evaluation frameworks are explicitly designed for the assessment of human interaction with IS in a healthcare setting including human and organizational factors.

4.2. Critical appraisal of evaluation frameworks

Building on previous studies on the evaluation approach [35], this critical appraisal is presented based on the adaptation of the classification HIS evaluation frameworks used by Currie [32]. We compare the strength and limitations of the following frameworks in the light of evaluation measures from the human, organizational and technological domains (see Table 3).

4.2.1. Generic evaluation frameworks

The first two frameworks emphasis more on methods to be used according to different system development stages. In Friedman and Wyatt [4], evaluation is classified into eight approaches of subjectivist and objectivist evaluation types. Similarly, Stead et al. [36] built a matrix of the relationship of system development stage to the level of evaluation. Ideally, a system should play a part in the overall technological infrastructure within the organization as well as in the overall social, cultural and functional environment of its intended users. However, the criteria as to how this can be evaluated would benefit from further clarification.

CHEATS is a generic framework for evaluating IT in healthcare that has six evaluation aspects: clinical, human and organizational, educational, administrative, technical and social [37]. CHEATS attempts to provide a more comprehensive evaluation and some more specific measures, especially in the clinical aspect. However, the dimensions within some of the aspects, such as technical, human and organizational could benefit from further development.

4.2.2. System development life cycle based evaluation frameworks

Brender [26] proposed an evaluation methodology based on the system development process, namely exploration, validity, functionality and impact phase based on [24], [25]. The evaluation aspects of this methodology are classified according to the evaluation phases shown in Table 3 of this chapter. This methodology includes a comprehensive scope of technology and organizational issues, but it could benefit more from further clarification of human issues and overall evaluation dimensions and measures.

Gremy et al. [8] presented a five step HIS evaluation process; each step is associated with system development stages, problems at stake and the role of human as actors alongside machines at work. While the human is central to this framework, organizational issues can also be given similar emphasis. Moreover, the evaluation criteria of this framework can be specified in more detail.

A global framework known as total evaluation and acceptance methodology (TEAM) was developed based on systemic and model theories [38]. It has three dimensions: role, time (evaluation phase) and structure (strategic, tactical, operational management level). The 3D structure of this model illustrates clearly the components of system evaluation. However, apart from the role and time aspects, the structure aspect can be challenging as the selection of evaluation measures can be categorized into more than one management level. As a whole, this framework is quite broad for a specific type of IS evaluation.

Westbrook et al. [39] outlined a number of methods throughout pre, during and post implementation. This multi-methods evaluation model (MEM), which uses a multi-disciplinary approach, provides a useful, specific guide to methodology selection. The evaluation criteria however, are disbursed among the methods presented.

4.2.3. Socio-technical based evaluation frameworks

An IT implementation and evaluation framework for individual users known as the IT adoption model (ITAM) was constructed to study the individual user perspective and potential IT adoption [40]. From the individual user perspective, this framework includes comprehensive evaluation criteria and relationships among them. This framework is clearly insufficient for a wider scope of evaluation, which involves the organizational aspect.

Aiming for a comprehensive framework, Kazanjian and Green [41] proposed a multi-disciplinary model for supporting decision making of health technology assessment (HTA). The main evaluation dimensions are population at risk, population impact, economic concerns, social context (including ethical, legal and political concerns) and technology assessment information. The framework provides useful guidelines on three key questions in the decision making process, namely stakeholders types, purpose and value of a new technology and benefit of technology adoption. Moreover, this framework includes broader, comprehensive view of technology assessment. Unlike the rest of the frameworks, the economic and impact dimensions are described in detail. The application of the framework is, however, limited by unspecified evaluation measures of technology and human factors.

Focusing on more specific framework, a social network analysis is proposed [42]. The framework is used to study the pattern of relations among a group of individual, departments and organizations relevant to HIS. The framework focuses on the relationships and communications among individuals and organization; however, the balance can be kept by considering more technical issues in the assessment aspect. In a similar vein, Berg [43] used the socio-technical approach where work practices are seen as networks of various related elements such as people, tools, organizational processes, machines and documents. This framework highlights the importance of an integrated network embracing technology, humans and the organization. The specific evaluation criteria could be more clearly stated. Kaplan [27] developed 4Cs from the Social Interactionist Theory, which stands for communication (interaction within department), care (medical care delivery), control (control in the organization) and context (clinical setting). Again the evaluation measures of this framework would benefit from more clarification and the control aspect needs further explanation.

We argue that the combination of different evaluation measures and their subsequent classification into structured dimensions and factors can contribute to inform decision making and guide almost every HIS development throughout the entire system life cycle. We have identified a number of evaluation measures based on generic, system development-based and socio-technical contexts of HIS evaluation. Emphasis may be given on specific evaluation measures during the evaluation process, by measuring these factors according to the specific context of study, to inform decision for the future and further development of HIS.

5. Discussion and conclusions

Overall, the evaluation frameworks complement each other in that they each evaluate different aspects of HIS pertinent to human, organizational and technological factors. As illustrated in Table 3, these frameworks differs in terms of generality and specificity, timing based on the system development phases and theoretical underpinning. In addition, these frameworks do not provide explicit evaluation categories to the evaluator.

We suggest that these different aspects can be combined in a single framework to enable comprehensive evaluation studies, and then more specific measures within the dimensions of each aspect can be defined to facilitate HIS evaluation. An attempt to include comprehensive, specific evaluation measures of HIS is presented in the proposal for a framework that is featured in the second part of our papers that is included in the same issue of this journal. Consequently the details of our proposal are left for the reader to pursue in our second paper, rather than duplicate them here.

To conclude this paper, we observe that evaluation is not easy. It is easy to measure many things but not necessarily the right ones. The more comprehensive framework in our follow up paper might encompass the evaluation needs because of its breadth, but care needs to be taken to identify the relevant and the not-so-relevant parts. Based on the specific study context, the evaluation can selectively focus on specific evaluation measures, to ensure that the evaluation outcomes will have an impact on informing decisions regarding further system development.

Summary points

What was known before the study:

•Heath information systems (HIS) evaluation has a number of problems and barriers that pose challenges to its evaluators.

•There is a large number HIS evaluation frameworks looking at different aspects of these systems.

•The existing evaluation methods do not provide explicit evaluation categories.

What the study has added to the knowledge:

•A classification of HIS based on their particular descriptions and characteristics.

•A review of findings of both health informatics and information systems evaluation.

•A critical appraisal of existing evaluation frameworks of HIS.

Acknowledgements

We gratefully acknowledge the funding received from both the Public Service Department of Malaysia and Universiti Kebangsaan Malaysia (National University of Malaysia) that helped sponsor this study.

References

[1]. [1]Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care: trends in evaluation research 1982–2002. Methods Inf. Med. 2005;44:44–56. MEDLINE

[2]. [2]Ammenwerth E, Brender J, Nykanen P, Prokosch H-U, Rigby M, Talmon J. HIS-EVAL workshop participants. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck.. Int. J. Med. Inf. 2004;73(6):479–491.

[3]. [3]Moehr JR. Evaluation of health information systems: beyond efficiency and effectiveness. Comput. Biol. Med. 2002;32(3):111–112. Full Text | Full-Text PDF (29 KB) | CrossRef

[4]. [4]Friedman CP, Wyatt JC. Evaluation Methods in Medical Informatics. New York: Springer-Verlag; 1997;.

[5]. [5]Wyatt JC, Wyatt SM. When and how to evaluate health information systems?. Int. J. Med. Inf. 2003;69(2–3):251–259.

[6]. [6]Ammenwerth E, Graber S, Hermann G, Burkle T, Konig J. Evaluation of health information systems—problems and challenges. Int. J. Med. Inf. 2003;71:125–135.

[7]. [7]Yusof M Mohd, Paul RJ, Stergioulas L. Health information systems evaluation: a focus on clinical decision supports system. In: Proceedings of the XIX International Congress of the European Federation for Medical Informatics (MIE2005). Geneva, Switzerland. 2005;p. 855–860.

[8]. [8]Gremy F, Fessler JM, Bonnin M. Information systems evaluation and subjectivity. Int. J. Med. Inf. 1999;56:13–23.

[9]. [9]Van der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J. Am. Med. Inform. Assoc. 2003;10(3):235–243. MEDLINE | CrossRef

[10]. [10]Lederer AL, Salmela H. Towards a theory of strategic information systems planning. J. Strategic Inf. Syst. 1996;5:237–253.

[11]. [11]Lucas HC. The Analysis, Design, and Implementation of Information Systems. Watsonville: Mitchell McGraw Hill; 1992;.

[12]. [12]Martin C, Powell P. Information Systems: A Management Review. Berkshire: Mc Graw Hill Book Company; 1992;.

[13]. [13]A. Olerup, On a Contingency Framework of Computerised Information System, In: J. Lucas H.C., F.F. Land, T.J. Lincoln, K. Supper (Eds.), The Information Systems Environment, Amsterdam, 1980.

[14]. [14]Wetherbe JC. Executive 's Guide to Computer Based Information Systems. New Jersey: Prentice Hall, Inc.; 1983;.

[15]. [15]Smith J. Health Management Information Systems: A Handbook for Decision Makers. Buckingham: Open University Press; 2000;.

[16]. [16]In: Van Bemmel JH, Musen MA editor. Handbook of Medical Informatics. Heidelberg: Springer-Verlag; 1997;.

[17]. [17]Gross PA, Bates DW. A pragmatic approach to implementing best practices for clinical decision support systems in computerized provider order entry systems. J. Am. Med. Assoc. 2007;14(1):25–28.

[18]. [18]Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al. Medication-related clinical decision support in computerized order entry systems: a review. J. Am. Med. Inform. Assoc. 2007;14(1):29–40. MEDLINE | CrossRef

[19]. [19]Kuhn KA, Giuse DA, Talmon JL. The Heidelberg Conference: setting an agenda for the IMIA working group on health information systems. Int. J. Med. Inf. 2003;69(2–3):77–82.

[20]. [20]M. Williamson, How do I evaluate user acceptance of an electronic decision support system? Australian Health Information Council, available from http://www.ahic.org.au/evaluation/user.htm, 2004 (cited 23 December 2004).

[21]. [21]Beyer HR, Holtzblatt K. Apprenticing with the customer. Commun. ACM. 1995;38:45–52.

[22]. [22]Goodhue DL. Understanding user evaluations of information systems. Manag. Sci. 1995;41:1827–1844.

[23]. [23]Willcocks L. Managing technology evaluation—techniques and processes. In: Galliers RD, Baker BSH editor. Strategic Information Management: Challenges and Strategies in Managing Information Systems. Oxford: Butterworth Heinemann; 1994;p. 365–381.

[24]. [24]Clarke K, O’Moore R, Smeets R, Talmon J, Brender J, McNair P, et al. A methodology for evaluation of knowledge-based systems in medicine. Artif. Intell. Med. 1994;6(2):107–121. MEDLINE | CrossRef

[25]. [25]Brender J, Talmon J, Nykanen P, McNair P, Demeester M, Beuscart R. In: van Gennip EMSJ, Talmon J editor. Assessment and Evaluation of Information Technologies in Medicine. vol. 17:Amsterdam: IOS Press; 1995;p. 189–208.

[26]. [26]Brender J. Methodology for Assessment of Medical IT-Based Systems, vol. 42. Amsterdsam: IOS Press; 1997;.

[27]. [27]Kaplan B. Organizational evaluation of medical information reources. Evaluation Methods in Medical Informatics. New York: Springer-Verlag; 1997;pp. 255–280.

[28]. [28]Stoop AP, Berg M. Integrating quantitative and qualitative methods in patient care information systems evaluation: guidance for the organizational decision maker. Methods Inf. Med. 2003;42(4):458–462. MEDLINE

[29]. [29]Jorgensen T. In: van Gennip EMSJ, Talmon J editor. Assessment and Evaluation of Information Technologies in Medicine. vol. 17:Amsterdam: IOS Press; 1995;p. 99–110.

[30]. [30]Kaplan B, Shaw N. People, organizational and social issues: evaluation as an Exemplar. In: Haux R, Kulikowski C editor. Yearbook of Medical Infomatics. 2002;pp. 91–102.

[31]. [31]Coiera E. Guide to Health Informatics. Hodder Arnold; 2003;.

[32]. [32]Currie LM. Evaluation frameworks for nursing informatics. Int. J. Med. Inf. 2005;74:908–916.

[33]. [33]Kaplan B. Evaluating informatics applications—clinical decision support systems literature review. Int. J. Med. Inf. 2001;64(1):15–37.

[34]. [34]In: van Gennip EMSJ, Talmon J editor. Assessment and Evaluation of Information Technologies in Medicine. Amsterdam: IOS Press; 1995;.

[35]. [35]Yusof M Mohd, Paul RJ, Stergioulas LK. Towards a framework for health information systems evaluation. In: Proceedings of the 39th Hawaii International Conferences on System Sciences. Kauai, Hawaii, USA. 2006;.

[36]. [36]Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library—resource projects to increase what is learned. J. Am. Med. Inform. Assoc. 1994;1(1):28–33. MEDLINE

[37]. [37]Shaw NT. CHEATS: a generic information communication technology (ICT) evaluation framework. Comput. Biol. Med. 2002;32(3):209–220. Abstract | Full Text | Full-Text PDF (93 KB) | CrossRef

[38]. [38]Grant I, Plante F, Leblanc . The TEAM methodology for the evaluation of information systems in biomedicine. Comput. Biol. Med. 2002;32(3):195–207. Abstract | Full Text | Full-Text PDF (139 KB) | CrossRef

[39]. [39]Westbrook JI, Braithwaite J, Iedema R, Coiera EW. Evaluating the impact of information communication technologies on complex organizational systems: a multi-disciplinary, multi-method framework. In: Proceedings of the Medinfo’04 Congress. San Francisco, CA, USA. 2004;p. 1323–1327.

[40]. [40]Dixon DR. The behavioral side of information technology. Int. J. Med. Inf. 1999;56:117–123.

[41]. [41]Kazanjian A, Green CJ. Beyond effectiveness: the evaluation of information systems using a comprehensive health technology assessment framework. Comput. Biol. Med. 2002;32:165–177. Abstract | Full Text | Full-Text PDF (162 KB) | CrossRef

[42]. [42]Anderson JG. Evaluation in health informatics: social network analysis. Comput. Biol. Med. 2002;32:179–193. Abstract | Full Text | Full-Text PDF (286 KB) | CrossRef

[43]. [43]Berg M. Patient care information systems and health care work: a sociotechnical approach. Int. J. Med. Inf. 1999;55(2):87–101.

[44]. [44]G.L. Glandon, T.L. Buck, Cost-benefit analysis of medical information systems: a critique, in: J.G. Anderson, C.E. Aydin, S.J. Jay (Eds.). Evaluating Health Care Information Systems: Methods and Applications, Newbury Park, 1994, pp. 164–89.

[45]. [45]Jiang JJ, Muhanna WA, Klenin G. User resistance and strategies for promoting acceptance across system types. Inf. Manag. 2000;37(1):25–36.

[46]. [46]R.M. Gardner, M.M. Shabot, Patient-monitoring Systems, in: E.H. Shortliffe, L.E. Perreault, G. Wiederhold, L.M. Fagan (Eds.). Medical Informatics: Computer Applications in Health care and Biomedicine, New York, 2001, pp. 443–484.

[47]. [47]Parrino T. Information technology and primary care at the VA: making good things better. Health Serv. Res. Dev. 2003;.

[48]. [48]Gawande AA, Bates DW. The use of information technology in improving medical performance—part II. Physician-support tools. Med. Gen. Med. 2000;2(1):.

[49]. [49]Randolph AG, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH. Users’ guides to the medical literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. JAMA. 1999;282(1):67–74. MEDLINE | CrossRef

[50]. [50]Hunt D, Haynes RB, Hanna S, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA. 1988;280(15):1339–1346. MEDLINE | CrossRef

[51]. [51]Kaplan B. Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int. J. Med. Inf. 2001;64(1):39–56.

[52]. [52]Kaplan B, Shaw NT. Future directions in evaluation research: people, organizational, and social issues. Methods Inf. Med. 2004;43(3):215–231. MEDLINE

[53]. [53]W.H. DeLone, McLean, E.R., Information systems success: the quest for the dependent variable information systems research 3 (1)(1992) 60–95.

a Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia

b School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK

Corresponding author. Tel.: +603 89216649.

PII: S1386-5056(07)00159-1

doi:10.1016/j.ijmedinf.2007.08.004

2 of 7