international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 journal homepage: www.ijmijournal.com Review Impacts of structuring the electronic health record: A systematic review protocol and results of previous reviews Hannele Hyppönen a,∗, Kaija Saranto b, Riikka Vuokko a, Päivi Mäkelä-Bengs a, Persephone Doupi a, Minna Lindqvist a, Marjukka Mäkelä c a Institute for Health and Welfare (THL), Department of Information, PB 30, 00271 Helsinki, Finland b University of Eastern Finland, Finland c Institute for Health and Welfare (THL), Service System Department, Finland a r t i c l e i n f o Article history: Received 12 February 2013 Received in revised form 27 November 2013 Accepted 28 November 2013 Keywords: Electronic health records Data structure Systematic literature review protocol Information quality Process quality a b s t r a c t Purpose: This paper (1) presents the protocol of an on-going systematic literature review on the methods of structuring electronic health record (EHR) data and studying the impacts of implemented structures, thus laying basis for the analysis of the empirical articles (2) describes previous reviews published on the subject and retrieved during the search of bibliographic databases, and (3) presents a summary of the results of previous reviews. Methods: Cochrane instructions were exploited to outline the review protocol – phases and search elements. Test searches were conducted to refine the search. The abstracts and/or full texts of review papers captured by the search were read by two of the team members independently, with disagreements first negotiated between them and if necessary eventually resolved in the team meetings. Additional review articles were picked from the reference lists of the reviews included in our search results. The elements defined in the search strategy and analytic framework were converted to a data extraction tool, which was tested by extracting data from the reviews captured by the search. Descriptive analysis of the extracted data was conducted. Results: The 12-stage review protocol that we developed includes definition of the problem, the search strategy and search terms, testing the strategy, conducting the search, updating search from references found, removing duplicates, defining the inclusion and exclusion criteria, exclusion and inclusion of papers, definition of the analytic framework to extract data, extracting data and reporting results. Our searches in fifteen electronic bibliographic databases retrieved 27 reviews, of which 14 were included for full text analysis. Of these, 11 focused on medical and three on nursing record structures. The data structures included forms, ontologies, classifications and terminologies. Some evidence was found on data structure impact on information quality, process quality and efficiency, but not on patients or professionals. Conclusions: The 12 step review protocol resulted in a variety of reviews of different ways to structure EHR data. None of them compared outcomes of different structuring methods; all had a narrower definition of the Intervention (a specific EHR structure) and Outcome (a specific impact category). Several reviews missed a clear connection between the data ∗ Corresponding author. Tel.: +358 50 3751858. E-mail address: [email protected] (H. Hyppönen). 1386-5056 © 2013 The Authors. Published by Elsevier Ireland Ltd. http://dx.doi.org/10.1016/j.ijmedinf.2013.11.006 Open access under CC BY-NC-SA license.160 international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 structures (interventions) and outcomes, indicating that the methods and applications for structuring patient data have rarely been viewed as independent variables. The review protocol should be defined in a manner that allows replication of the review. There are different ways of structuring patient data with varying impacts, which should be distinguished in further empirical studies, as well as reviews. © 2013 The Authors. Published by Elsevier Ireland Ltd. Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 2. Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 3.1. The protocol for the systematic review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 3.2. Description of previous reviews captured and analysed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 3.3. Summary of the impacts of the structures reviewed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 4.1. The review protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 4.2. PICO-elements of previous reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 4.3. The reviewed (and missing) impacts of structuring EHR data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 4.4. Strengths and weaknesses of the study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Author contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Conflict of interest statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Appendix A. Supplementary data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 1. Introduction The primary purpose of electronic health record systems (EHR systems, see annex 1 for abbreviations used in this article) is to support efficient, high-quality integrated health care, independent of the place and time of health care delivery. It is estimated that information and communication technology (ICT) implementation can result in care that is safer, and more responsive to patients’ needs and, at the same time, more efficient [1]. The range of possible ICT applications in the health care sector has increased exponentially, with a number of countries progressing from local towards regional or national level patient/health information exchange [2–7]. In many eHealth implementation strategies, the importance of defining standard structures for core patient information is crucial [7,8]. Structuring patient data is perceived to support clinical care processes, facilitate new technologies for increasing patient safety and care quality, enable quality monitoring of the health service processes and evidence-based management locally, regionally and nationally by enhancing collection of statistical information [7,9,10]. It is also assumed to enable easier participation of citizens in their care process. Evidence to support these assumptions is, however, yet scarce [11,12] while the balance between risks and benefits of free text vs. structured data in EHR documentation has long been identified as a fragile one [13–15]. In Finland, one of the leading countries in global eHealth [16,17], the national health information archive (KanTa) is being implemented step by step from 2009 to 2016. In addition to the document archiving service, the architecture supports National Health Information Exchange Services for both professionals and citizens. Both implemented and planned solutions depend heavily on the use of various classifications, the adoption of which has progressed rapidly [18,19]. The systematic review protocol and review of reviews outlined in this paper are part of a project intending to inform the evidencebased planning of the Finnish national health information system’s evaluation and monitoring. The aims of this paper are: (1) To present a protocol for a systematic literature review on methods of structuring electronic health record (EHR) data and studying their impacts, thus laying basis for search and analysis of the empirical articles, (2) To describe previous reviews published on the subject and retrieved during the search of bibliographic databases, and (3) To present a summary, using the analytical framework proposed for this review, of the results of the reviews analysed for this paper. In accordance to good research practices, we describe in this paper the stages and rationale of the study protocol developed and applied for the systematic review. We present and discuss the results of analysing earlier reviews on the subject identified through our search of bibliographic databases and we report on the current state of progress in our review of empirical studies, the final results of which will be offered in forthcoming publications. Open access under CC BY-NC-SA license.international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 161 We started the analysis with a review of existing reviews because we needed to know what evidence was already available, and ideally (in case of finding a review with thorough methodology) we could have updated an existing review. Secondly, the empirical studies referred to in the reviews were used as test cases for making sure our search strategies were sensitive enough. We compared the references of the empirical articles to our original search results in order to test the sensitivity of our search strategies and to update our search result. In addition, we wanted to test our data extraction model for the review of primary empirical studies. A secondary benefit of this order of proceeding was also that we could describe the review methodology properly and refer to it in the followup articles. 2. Methods A systematic literature review is a comprehensive, protocolbased review and a synthesis of research focusing on a chosen topic or on defined research questions. A review team was brought together that consisted of an informatician, a systematic review specialist, two medical informatics specialists and an evaluation specialist. The team met regularly to outline the review protocol. Cochrane instructions on the review protocol [20] were modified to fit our purpose: We outlined the questions that we were addressing for the basis of our search strategy as advised in the protocol, and used the PICOelements to define the search terms and their combinations. PICO refers to defining the Population (Participants), Intervention (or Exposure for observational studies), Comparators (main alternative interventions) and Outcomes. PICO allows taking a systematic approach to the literature search from bibliographic databases [20]. Following the Cochrane protocol, we then defined the criteria against which the search results would be assessed for exclusion and inclusion in the review. Instead of defining the health problem and the interventions to tackle it, as suggested by the Cochrane instructions, we focussed on various anticipated outcomes of different methods of structuring patient data. We used existing frameworks to define indicators for measuring benefits and potential harms, but did not use study design as one of the inclusion criteria as advised by the Cochrane protocol. Following the Cochrane protocol we outlined the process for assessing, and summarizing studies in the review. To limit the number of hits, “news” & “letters to editors”- categories were excluded already in the search phase. Review articles were processed separately from other results of our search strategies. Before proceeding to the exclusion round, we updated our material by searching for relevant publications in the reference lists of the reviews retrieved. The review articles were also used to test the analysis framework and to produce a summary of earlier findings as background reference. We drew a flow chart of the reviewing progress and results at each phase of the systematic review protocol. By documenting all these phases, the protocol forms a detailed record of how we will answer the research questions, making the process repeatable and transparent for scientific debate. The Table 1 – The review protocol. 1. Defining the research problem. 2. Defining the databases and search strategies using the PICO method 3. Conducting test searches, updating the search strategy 4. Conducting searches, saving results to RefWorks reference management system 5. Removing duplicates 6. Updating search results from reference lists of previous reviews 7. Defining exclusion and inclusion criteria 8. Exclusion using article heading and/or abstract (two independent reviewers) plus consensus round 9. Inclusion round based on full text of remaining articles (as above) 10. Generating information collection and reporting templates with help of an analytical framework, testing and refining with a sample of 22 articles 11. Extracting data from articles using the template (2 independent reviewers), data analysis 12. Generation of the review report results-section in this paper describes our protocol in detail, the resulting reviews and their findings. 3. Results 3.1. The protocol for the systematic review The protocol we developed includes 12 phases as depicted in Table 1. The research problem was defined as three research questions. (1) What methods have been used to structure patient information? (2) How have the resulting interventions been evaluated? (3) What impacts have the different structuring methods produced and for whom? The PICO-elements for our search strategy were defined as follows: Population was specified as the different professional groups involved in documenting and utilizing EHR data, with the addition of the term “patient access to records”. Intervention was specified as structured documentation in the EHR. Comparison was specified as free text or narrative EHR documentation. Outcomes were specified as evaluation or assessment studies in order to cover a broad range of outcomes. The elements were modified to search terms according to each database’s terminology, supplemented with text search. Since there is a long history of structuring the EHR, the search spanned a period from 1975 to November 2011. Database searches were conducted and duplicates removed in November 2011. Annex 2 depicts, as an example, the search strategy used in the Medline search. The databases selected, the number of references found and remaining after removal of the duplicates are depicted in Table 2. The exclusion criteria used in the next step of the study protocol (the review of headings and abstracts) are depicted in Table 3. We used two additional criteria for Population: The study needed to be conducted in Upper middle and High Income countries [21], and the reporting language needed to be Finnish, Swedish or English. For the inclusion round (full text review), our inclusion criteria were the positive expressions of the exclusion criteria combined with several generic criteria, to ensure compliance with the repeatability requirement for systematic reviews: the162 international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 Table 2 – Databases searched number of references per database, duplicates in each new database search compared with previous ones, and references left per database after removal of duplicates. Database References Duplicates New references Medline (OVID) 335 335 Cinahl 84 21 63 ProQuest Health Management 55 6 49 Science Direct 37 0 37 Linda 32 0 32 Medic 31 7 24 Cochrane Database of Controlled Trials 20 7 13 DARE 19 0 19 NHS Economic Evaluation Database 16 0 16 Academic Search Elite (ASE) 14 2 12 Arto 8 1 7 Cochrane Database of Systematic Reviews 8 0 8 PubMeda 8 0 8 HTA 1 1 0 Web of Science 1 40 10 30 Web of Science 2b 35 8 27 Sum 743 63 680 a The PubMed search was a targeted update search on selected computer science publications (of ACM and IEEE). b A test search (Topic = (electronic patient record) AND Topic = (structured data) AND Topic = (impact)). Table 3 – Applied exclusion criteria. PICO-elements Exclusion criteria for headings and abstracts Population Not upper middle and high income countries [21] Reporting language not Finnish, Swedish or English Primary users not clinicians, nursing staff, patients’ access to records, health care management or researchers. Intervention Not focusing on EMR or Nursing record structuring or impacts of structures on clinical work, decision support, management or research Comparison No specific exclusion criteria, Free text as search term Outcome No evaluation of outcomes of implementation/exploitation of structures article is available, it is an original reference (as opposed to duplicate), it is a scientific journal publication (not a dissertation or a book), it is empirical (and not a review, since we analysed reviews separately for this article), and there is a named author. Each PICO-element was extended by sub-elements for the data extraction form: “Population” with e.g. users of structures and context of use, “Intervention” with e.g. type, phase of development and method of application, and “Comparison” with free text or narrative. The Outcomes, in particular, were extended with the assistance of an analytic framework to extract information on the various types of impacts reported in the articles. The framework is based on a number of earlier published models [3,22–25] that are shown in the columns of Fig. 1. In addition, the data extraction form included a category for extracting information on study design, indicators and data collection methods used. The draft version of the data extraction form was tested and refined based on a sample of 22 empirical articles. Fig. 2 depicts the flow chart on how the review process progressed at each stage of the systematic review protocol, with the resulting reviews to be analysed for this article. 3.2. Description of previous reviews captured and analysed Of the original search of databases (680 unique articles) we identified 27 reviews. One of the reviews [26] was located separately on the basis of the published protocol [27], which had been captured by our search. Of the 27 reviews 13 were excluded based on the abstract, 8 due to unsuitable intervention, 3 due to not presenting a full text review and 2 as duplicates (see Fig. 2). All the rest were included after initial reading of the full texts. Of the five reviews focusing on EHR structures, only one [28] clearly identified a single type of structuring method (ontologies) and a single impact category (auditing conformity to guidelines). One review [29] focused on a single impact category (information quality), searching for methods to measure it from structured and textual record data. One review [26] focused mainly on one type of structuring method (templates) of patient history with a variety of impacts, mainly information quality. Two reviews [3,30] focused on impacts of the entire record system, with some empirical studies included where structure specific impacts could be identified. Of the two nursing reviews focusing on data structures one focused on nursing record systems [31] and described the change during two decades (data quality, process, efficiency). The remaining review [32] assessed the nursing documentation structures. Altogether 174 empirical studies had been analysed in the 14 reviews included in our review of reviews. There were 11 identical references with our original search result of 680 studies: Four of the studies analysed by Urquhart [31] and four byinternational journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 163 Health IT evaluaon studies (22) IS success model (23) Clinical adopon framework (3) Model of acvity system (24) EUnetHTA (25) Actors Division of work Funding and incenves Legislaon, Policy, Governance Legal aspects Standards Societal, polical, economic trends Ethical analysis Organizaon Organizaon Organisaonal aspects Intenon to use Use Objecves Health problem, current use of technology Care quality Clinical effectiveness Safety Access Social aspects (e.g. access) Social aspects Outputs, outcomes Organisaonal aspects (ulizaon) Rules Tools, resources [Care and informaon] Processes Descripon and technical characteriscs of technology Costs and economic evaluaon Time Impacts on Health care system outcomes Outcome quality Organisaonal impacts (net benefits) People Impacts on Health care system structures / inputs Process impacts Individual impacts (net benefits) Structural quality, Informaon quality Process quality Producvity, efficiency (Ulizaon, amount of) use Informaon system quality Informaon quality Support Service quality User sasfacon Fig. 1 – The analytic framework for grouping outcomes of the reviewed studies. Müller-Staub [32]; one analysed by Fernando [26], one by Lau [3] and one by White [28]. Thiuru [29] and Boyle [30] had analysed no identical references. Possible reasons and implications for this “mismatch” in search results are assessed in the discussion section. There were We added 10 empirical studies from the 174 studies reviewed in previous reviews to our search result, as they seemed to fit within the scope of our study. Results of testing the data extraction tool (including elements from our analytical framework) are presented in Table 4. In the detailed data extraction phase, only 7 of the 14 reviews were found to focus on EHR data structures in a manner that the categories defined in our data extraction form could be extracted from them (see Table 4 for their main details). The other 7 reviews discussed exploitation of EHR data and EHR from a general perspective that did not associate observed impacts with utilized structures. In the view of the seven remaining review articles, the impacts fell into three main categories: (1) impacts on health care system structures/inputs, (2) impacts on health care processes and (3) impacts on health care system outcomes (Fig. 1). Thiuru et al. [29] focused on data quality for structured diagnostic data including codes, classifications, and nomenclatures. Data structures and precise codes were not identified clearly in the referred articles. The review includes the following named structures or classifications: Read, OXMIS, ICPC, and ICD. According to the review, data reliability was usually measured with rate comparisons. Data sensitivity or positive predictive values were the two most common measures of validity in relation to completeness of data. The scope of reference standards varied broadly. The ability to link prescriptions to diagnosis was the favoured means of identifying patients and establishing predictive validity of diagnostic codes. Diseases with clear diagnostic criteria were generally better recorded, as well as data on specific procedures. Record linkage and automated utilization of structured data, for example, in investigations and test results, were reported as positive outcomes. Müller-Staub et al. [32] examined the effects of nursing diagnostics on the quality of patient assessments; the frequency of documented nursing diagnoses; the accuracy of nursing diagnoses, including related signs and symptoms (defining characteristics) and aetiologies (related factors) as well as coherence among diagnoses, interventions and outcomes. Based on their findings, the use of nursing diagnoses improved documentation. Pain was the most frequently diagnosed symptom across the reported study settings. The accuracy of reported nursing diagnoses and the inclusion related signs/symptoms and aetiologies were weak. Studies also reported deficiencies to state diagnoses based on aetiological factors. The review concluded that although standardized nursing diagnoses led to better documentation, better documentation did not necessarily lead to better patient care outcomes. In the last update of their review, Urquhart et al. [31] reported the impact of nursing record systems on nursing practice and patient outcomes. The review included both paper based and computerized patient record systems. The referred studies provided no evidence of any measurable difference, in nursing practice or patient outcomes in relation to the use of any kind of nursing record systems. The review concludes that nursing record systems may be successful in specific issues, such as reducing lost notes or decreasing the time required for data entry and the amount of paper files. However, it is uncertain whether changing an entire system of recording nursing care may improve how nurses practice or how well a patient is cared for. Boyle et al. [30] focused on clinical decision support systems, forms, and management reports or conformity to care guideline concerning smoking status. The referred studies of forms as a means of intervention were observational. Boyle164 international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 Table 4 – Reviews with impacts of specific structures (interventions) on health care inputs, processes, outputs and outcomes. Intervention: structures Intervention: domains of implementation Population: context of implementation Outcomes: impact categories Methods: study selection criteria Methods: databases reviewed Nr of reviewed studies Reference Codes, classifications ontologies, and textual data Patient diagnostic data Primary care Structural/input (info quality) Own criteria based on reference standard, study objectives and data types. All major bibliographic and several specialist databases 52 Thiuru et al. [29] NANDA-I, NIC and NOC Nursing diagnoses, interventions and outcomes Hospital, secondary hospital, Differing specialties, home and school nursing Input (assessment quality), processes, output (efficiency) RCT, quasi-experimental, chart review, correlational, descriptive, pre–post-test, observational, explorative, qualitative interviews PubMed, Medline, Cinahl, Cochrane 36 Müller-Staub et al. [32] Forms, nursing nomenclature Nursing care documentation Hospital, community, primary care Input (info quality), processes, output (efficiency), outcomes RCT, controlled before and after studies, interrupted time series Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register; MEDLINE, EMBASE, CINAHL, BNI, ISI Web of Knowledge, and ASLIB Index of Theses 9 Urquhart et al. [31] Forms Clinical interventions for tobacco use Primary ambulatory secondary hospital Processes, outcomes RCT, observational Cochrane 11 Boyle et al. [30] Ontologies Patient data for diabetes, cancer waiting times, cardiovascular disease, cholesterol, preventive services, hypertension, maternal-child health, endoscopy – Process (compliance to guidelines) – PubMed, Medline, Cinahl, Cochrane, WOS 13 White et al. [28] Forms Generic and disease-specific patient history (cardiac, stroke, paediatrics) Primary and specialized ambulatory and hospital (mainly emergency) care Structural/input (info quality), processes, outputs (efficiency) No methodological exclusion criteria 9 different databases 10 Fernando et al. [26] Forms Patient data Ambulatory care Structural/input (system quality) – Medline, Cinahl 43 Lau et al. [3]international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 165 Fig. 2 – The application of the systematic review protocol as a flow chart. The darker box indicates materials analysed for this paper (review articles); the dotted line indicates the three on going analyses of empirical articles. et al. describe qualitatively the documentation before and after implementing the structures related to smoking status and cessation. The authors conclude that more well designed RCT’s are needed that can assess the promise of EMRs to enhance clinical treatment of smoking cessation. White et al. [28] reviewed the use of clinical ontologies for monitoring compliance with clinical practice guidelines on smoking session. The ontologies and other referred structures included Medicine Clinical Terms Nomenclature, UMLS (Unified Medical Language System), ICD, ICPC; Current procedural terminology, national drug code, ATC (Anatomical Therapeutic Chemical Classification System), Omaha Nursing, and Minimal Standard Terminology for Gastrointestinal endoscopy. The review argues that there are three types of problems in using ontologies for guideline compliance monitoring. (1) Scalability issues when ontology-based screening focuses on just one stage or element of guideline audit, or when patient selection is biased or missing. (2) Vocabulary issues when limited granularity is used to code guidelines and used in free text. (3) Guideline issues when treatment decisions are not covered by the guideline, guidelines are outdated or erroneous, and when there is interpretation discrepancy. The review concludes that some decision systems can integrate guideline monitoring into the workflow, while there are rare studies on ontology use to audit EMRs for compliance to guidelines. Fernando et al. [26] reviewed interventions that included forms for preoperative care, asthma assessment, head injury and other condition specific forms. Implemented structures were not a primary focus in the included studies. However,166 international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 increase in completeness of information and in diagnostic accuracy was demonstrated in the studies, when utilizing a structured form, but no attempt to confirm if the additional information based on the completeness of documentation was clinically useful. No evidence on risks of structuring was found. In most of the reviewed studies, the structured patient data was a means of validating another technical intervention, or means of obtaining data to improve a clinical hypothesis or care pathways. Lau et al. [3] applied a modified IS success model to report studies that described availability of EMR and its effect on physician practice. Two of the included studies defined EMR structures (forms). The authors concluded that currently limited positive EMR impact in the physician office has been observed. To improve EMR success in future, lessons learned from previous studies are to be accounted for. 3.3. Summary of the impacts of the structures reviewed Based on the analytical framework (Fig. 1), the outcomes in the review articles were classified according to the main impact categories (Fig. 3). As Fig. 3 shows, the review articles covered all three main impact categories (health care inputs, health care processes, and health care outcomes) from our framework. The reviews associated codes, classifications and ontologies with improved information quality (Thiuru et al. [29], Fernando et al. [26], Lau et al. [3], Müller-Staub et al. [32], Urquhart et al. [31]), ontologies and forms with improved care and information processes (White et all [28], Fernando et al. [26], Boyle et al. [30], Urquhart et al. [31]) and NANDA-I, NIC, NOC, and nursing nomenclature and forms with improved productivity and efficiency (Fernando et al. [26], Müller-Staub et al. [32], Urquhart et al. [31]). 4. Discussion 4.1. The review protocol The research problem that triggered the systematic review project as a whole and the review of reviews presented in this paper, focused on uncovering the different ways to structure patient information and identifying the impacts of these interventions. The scope was thus much wider than e.g. in a standard Cochrane review that defines a health problem and proceeds to compare different interventions to tackle it. The wide intervention and outcome definition resulted in a less focused search result, though capturing a variety of relevant interventions, which had not been in the focus of previous reviews. The 15 different databases required modifications of the search strategy to match the database-specific search properties. The importance of covering a broad scope of bibliographic resources became evident in the phase of duplicate removal: each of the included databases provided new references, which we would have otherwise missed. The impact (or outcome) classification presented in Fig. 1 and tested with the review articles seemed too precise to analyse outcomes of the reviews. However, it offered a common understanding for bracketing the outcomes, when individual researchers were analysing the contents of the articles. It also helped in detecting types of outcomes that have previously not been in focus. Further analysis will indicate how well the framework is suited to the analysis of a much larger number of empirical articles. 4.2. PICO-elements of previous reviews The diversity of definitions of the review protocols and PICO-elements in the earlier reviews [c.f. [33]] limited our possibilities to use them to test sensitivity of our review protocol. Moreover, it made it impossible for us to build on earlier work by updating an existing review. Reporting the results of the earlier reviews did however serve the purposes for which the review of reviews was conducted: presenting what was discovered previously, determining if an update of an earlier review could have been done, and testing the analytical framework proposed for this review. The analysis of relevant earlier review articles according to our analytic framework underlined how reviews on EHR structures are yet scarce, as we found a paucity of reviews with structuring as an independent variable. We concluded that no previous protocol to search and review studies concerning different EHR data structures, their quality and various outcomes had been sufficiently systematic and comprehensive. As such, our study protocol offers a tool to researchers aiming e.g. to identify empirical articles for more focused reviews of varying impacts of different EHR data structures. 4.3. The reviewed (and missing) impacts of structuring EHR data All the three main outcome categories (Fig. 3) were covered in the reviews, but many of the individual elements in each category were not. Compared to the expected outcomes of structuring patient data [7,9,10], there was evidence of improved information quality in the Input-catogory, but no evidence that this would support clinicians’ care processes. Impacts on actors were scarce (e.g. on user skills related to the implemented new method of structuring patient data, usability or usefulness of the structured data). There was evidence to support the administrative viewpoint of increasing adherence to documentation and care guidelines. In the Outputand Outcome-categories, impacts focused on productivity and secondary use of structured data (for automatic monitoring of care guideline compliance). There was little or no evidence found of expected benefits of structuring for “patient safety”, “care quality” or “easier participation of citizens in their care process”. The review papers had varied concluding remarks. Thiuru et al. [29] discussed utilization of structured data, for example, in coding diagnostic criteria and validating diagnostic codes by linking them to prescriptions data. Automated utilization of structured data was perceived as a positive outcome. Similarly, White et al. [28] concluded that structured patient data can be utilized, for example, in clinical decision support systems, although such have not been widely tested or audited. Boyle et al. [30] added that structured patient data can improve the identification of risk patients. Both Müller-Staub et al. [32] and Fernando et al. [26] noted the increases in quality patientinternational journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 167 Health IT evaluaon studies (22) IS success model (23) Clin frame ical a wo dopon rk (3) Model of acvity system (24) EUnetHTA (25) Actors Division of work Funding and incenves Legislaon, Policy, Governance Legal aspects Standards Societal, polical, economic trends Ethical analysis Organizaon Organizaon Organisaonal aspects Objecves Health problem, current use of technology Care quality Clinical effecveness Safety Access Social aspects (e.g. access) Impacts on Health care system outcomes Outcome quality Organisaonal impacts (net benefits) People Impacts on Health care system structures / inputs Process impacts Individual impacts (net benefits) Structural quality, Informaon quality Process quality Producvity, efficiency: Fernando et al (26), MüllerStaub et al (32), Urquhart et al (31) Informaon system quality Informaon quality: Thiuru et al (29), Fernando et al (26), Lau et al (3), Müller-Staub et al (32), Urquhart et al (31) Support Service quality User sasfacon (Ulizaon, amount of) use Social aspects Outputs, outcomes Organisaonal aspects (ulizaon) Rules Tools, resources Care and informaon processes: White et al (28), Fernando et al (26), Boyle et al (30), Urquhart et al (31) Descripon and technical characteriscs of technology Costs and economic evaluaon Fig. 3 – Impact categories covered by the previous reviews. information. However, both argued that increases in clinical information does not necessarily lead to better patient outcome as there is little evidence to confirm the usefulness and usability of increases in clinical information. Urquhart et al. [31] add that new documentation structures do not imply changes in practices or in process outcomes. 4.4. Strengths and weaknesses of the study The strengths of the systematic review include a clear aim and object, which lead to our study questions. A transparent 12-step protocol for a systematic review was developed and followed by a research team with methodological and content expertise. An exhaustive search was conducted on 15 bibliographic databases with an extensive search strategy. A framework for analysing the results was juxtaposed against previous frameworks for extracting and grouping data from the reviews and the empirical articles. The framework proved useful as a systematic documenting tool of study data and as such, a basis for further analysis. The main weakness of our systematic review protocol is related to the difficulty in defining search terms for the intervention and outcomes: We needed to maintain an extensive definition of both in the search protocol to answer the questions set for the review. As methods or actual impacts of patient information structuring are not well defined in literature, we could not define the search protocol in a more detailed manner to leave out articles where methods of patient information structuring are not adequately described for comparison. Previous reviews had much narrower definition of the PICO elements, but their research problems were also different, not aiming to compare outcomes of different interventions. The mismatch between our search and previous searches (our search found only 11 of the 174 previously analysed empirical articles – or vice versa: of the empirical articles we found, previous reviews only found 11) may be explained by the fact that previous reviews have focused on a single method of structuring, not compared different methods, and many also focused on one clearly defined outcome category. In addition, earlier reviews had detected a similar problem to ours: not being able to find studies, were a method for structuring had been regarded as an independent variable, thus including many studies, where impacts of methods of structuring EHR contents could not be singled out. We did not want to include empirical articles without a clear connection between a168 international journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 structuring method and outcomes. In the light of these two issues, it is not surprising that there were only 11 common hits in our searches compared to previous ones. We added 10 new references from previous reviews, but only a thorough analysis will show, how many of these will actually be included in the results section (of the 14 analysed reviews only 7 were included in the results). The earlier reviews did not describe their review protocol in a detailed enough manner that would have allowed us to compare the protocols to find similarities and differences in more detail. 5. Conclusions Diverse foci on various EHR contents to be structured, structuring methods and impact measures induce difficulties in grouping and summarizing the results of previous reviews. The positive outcomes of different structuring methods seem to cluster on information quality and process quality from the administrative viewpoint, but not necessarily leading to better patient outcomes. A more systematic reporting of the review protocols as well as of the variety of benefits connected to the diverse ways of structuring patient data would contribute to a coherent evidence base for decision making. Author contributions All the authors, Hannele Hyppönen, Kaija Saranto, Riikka Vuokko, Persephone Doupi, Päivi Mäkelä-Bengs ja Marjukka Mäkelä contributed to the study design, search protocol, selection and analysis of the reviews as well as revisions and approval of the article. Hannele Hyppönen had main responsibility of the article. She was responsible for the study questions, the study outline, the analysis framework and EMR-results-sections of the report. Kaija Saranto had main responsibility for the Nursing informatics section of the report (results and conclusions). She also contributed actively in all the other parts of the article. Riikka Vuokko and Persephone Doupi had the main responsibility of the terminology used in the article. They contributed to editing the article from terminological viewpoint. Marjukka Mäkelä had the main responsibility for the methodology section of the article. She steered the authoring team in reporting of a systematic review protocol with her strong expertise in systematic reviews in the HTA-context. Conflict of interest statement No reported conflicts of interest. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/ j.ijmedinf.2013.11.006. Summary points What was already known on this topic • Emphasis on structuring electronic health records (EHRs) is increasing globally. • There are arguments presented for and against structuring of the EHR data. • Many of the benefits expected from EHR implementations rely on the use of structured documentation. What this study added to our knowledge • EHR data structures have rarely been viewed as the intervention, and hence there is a paucity of reviews on the diverse impacts brought about by different methods of structuring EHR content. • EHR data structures have so far been mainly associated with increases in the information quality and process quality/efficiency. • We propose a protocol for systematically reviewing the literature on structured EHR data and their impacts that can serve as the blueprint for further research in the field. r e f e r e n c e s [1] OECD, Improving Health Sector Efficiency: The Role of Information and Communication Technologies. OECD Health Policy Studies, OECD, 2010. [2] J. Hendy, B. Reeves, N. Fulop, A. Hutchings, C. Masseria, Challenges to implementing the national programme for information technology (NPfIT): a qualitative study, BMJ (August) (2005) 331–336. [3] F. Lau, M. Price, J. Boyd, C. Partridge, H. Bell, R. Raworth, Impact of electronic medical record on physician practice in office settings: a systematic review, BMC Med. Inf. Decis. Making 12 (10) (2012) 1–10. [4] National eHealth Strategy, 2012, Available from: http://www.health.gov.au/internet/main/publishing.nsf/ Content/National+Ehealth+Strategy (Internet). [5] STM, eHealth Roadmap Finland, Ministry of Social Affairs and Health, 2007, Report No.: 15. [6] Meaningful Use Definition & Objectives, Office of the National Coordinator for Health Information Technology, 2013, http://www.healthit.gov/providers-professionals/ meaningful-use-definition-objectives (accessed 16.07.13) (Internet). [7] EHR Interoperability: The Structured Data Capture Initiative, 2013, Available at: http://www.healthit.gov/buzz-blog/ electronic-health-and-medical-records/ehr-interoperabilitystructured-data-capture-initiative/ (cited 16.07.13) (Internet). [8] epSOS—European Patient Smart Open Services Project, Semantic Issues, 2013, Available at: http://www.epsos.eu/ technical-background/semantic-issues.html (cited 16.07.13) (Internet). [9] P.G. Shekelle, S.C. Morton, E.B. Keeler, Costs and Benefits of Health Information Technology (Structured Abstract). Health Technology Assessment Database, John Wiley & Sons, Ltd., Chichester, UK, 2011, Issue 4, Dson: ST. [10] European Commission e-Health-Making Healthcare Better for European Citizens: An Action Plan for European eHealthinternational journal of medical informatics 8 3 ( 2 0 1 4 ) 159–169 169 Area, European Commission, Brussels, 2004, Report No.: COM.356 final. [11] Knowledge to Action Research Programme. Evidence Summary: Electronic Health Records (EHRs), Ottawa Hospital Research Institute, 2013, Available at: http://www.ohri.ca/kta/docs/KTA-EHR-evidence-review.pdf (retrieved 16.07.13). [12] J. Car, A. Black, C. Anandan, et al., The Impact of eHealth on the Quality & Safety of Healthcare, A Systemic Overview & Synthesis of the Literature. Report for the NHS Connecting for Health Evaluation Programme, March 2008. [13] A.M. van Ginneken, The computerized patient record: balancing effort and benefit, International Journal of Medical Informatics 65 (2) (2002) 97–119. [14] C. Lovis, R. Baud, P. Planche, Power of expression in the electronic patient record: structured data or narrative text? Int. J. Med. Inf. 58 (September) (2000) 101–110. [15] H.J. Tange, A. Hasman, R. de Vries, H.C. Schouten, Medical narratives in electronic medical records, Int. J. Med. Inf. 46 (August (1)) (1997) 7–29. [16] S. Schug, D. Whitehouse, eHealth Strategy and Action Plan of Finland in a European Context. Workshop Report, Ministry of Social Affairs and Health, Finland, 2013, Available at: http://www.stm.fi/c/document library/get file?folderId= 6556944&name=DLFE-26602.pdf [17] Estonia Tops the List of Best E-health implementors in Europe, Htcluster.eu, 2013, Available at: https://htcluster.eu/?q=node/2259 (Internet) (cited 16.07.13). [18] I. Winblad, J. Reponen, P. Hämäläinen, Tieto- ja viestintäteknologian käyttö terveydenhuollossa vuonna 2011. Tilanne ja Kehityksen Suunta, Terveyden ja hyvinvoinnin laitos, Helsinki, 2012, Report No.: 3/2012. [19] P. Hämäläinen, J. Reponen, I. Winblad, J. Kärki, M. Laaksonen, H. Hyppönen, et al., eHealth and eWelfare of Finland. Checkpoint 2011, National Institute for Health and Welfare, Helsinki, Finland, 2013, Report No.: 5/2013. [20] J.P.T. Higgins, S. Green, Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0, 2011, Available from: http://www.cochrane-handbook.org (Internet) (cited 03.09.12). [21] Country Income Groups (World Bank Classification), World Bank, 2013, Available from: http://www.grida.no/graphicslib/ detail/country-income-groups-world-bank-classification 1394 (Internet) (cited 22.01.13). [22] E. Ammenwerth, N. de Keizer, An inventory of evaluation studies of information technology in health care: trends in evaluation research 1982–2002, Methods Inf. Med. (44) (2005) 44–56. [23] W. DeLone, E.R. McLean, The DeLone and McLean model of information systems success: a ten-year update, J. Manage. Inf. Syst. 19 (Spring (4)) (2003) 9–30. [24] H. Hyppönen, eHealth services and technologies: challenges for co-development, Hum. Technol. 3 (May (2)) (2007) 188–213. [25] K. Lampe, M. Mäkelä (Eds.), HTA Core Model for Medical and Surgical Interventions. First Public Draft (revised) ed., EUnetHTA: Work Package 4, 2007. [26] B. Fernando, D. Kalra, Z. Morrison, E. Byrne, A. Sheikh, Benefits and risks of structuring and/or coding the presenting patient history in the electronic health record: systematic review, BMJ Qual. Saf. (21) (2012) 337–346. [27] E. Byrne, B. Fernando, D. Kalra, A. Sheikh, The benefits and risks of structuring and coding of patient histories in the electronic clinical record: protocol for a systematic review, Inform. Prim. Care 18 (3) (2010) 197–203. [28] P. White, A. Roudsari, Use of ontologies for monitoring electronic health records for compliance with clinical practice guidelines, Stud. Health Technol. Inf. 164 (2011) 103–109. [29] K. Thiuru, A. Hassey, F. Sullivan, Systematic review of scope and quality of electronic patient record data in primary care, BMJ 326 (2003) 1070–1072. [30] R. Boyle, L. Solberg, M. Fiore, Use of electronic health records to support smoking cessation, Cochrane Database Syst. Rev. (2010) 10. [31] C. Urquhart, R. Currell, M.J. Grant, N.R. Hardiker, Nursing record systems: effects on nursing practice and healthcare outcomes, Cochrane Database Syst. Rev. 21 (January (1)) (2009) CD002099. [32] M. Müller-Staub, M.A. Lavin, I. Needham, van AchterbergF T., Nursing diagnoses, interventions and outcomes – application and impact on nursing practice: systematic review, J Adv Nurs 56 (2006) 514–531. [33] J. Talmon, E. Ammenwerth, J. Brender, N. deKeizer, P. Nykänen, M. Rigby, STARE-HI – statement on reporting of evaluation studies in health informatics, Int. J. Med. Inf. (79) (2009) 1–9. [34] H. Hyppönen, A. Faxvaag, H. Gilstad, G.A. Hardardottir, L. Jerlvall, M. Kangas, et al., Nordic eHealth Indicators: Organisation of Research, First Results and Plan for the Future (TemaNord), Nordic Council of Ministers, Oslo, 2013.