The State of Evidence in Patient Portals: Umbrella Review (2024)

PMCID: PMC7688386

Marcy G Antonio, BSc, MPH,^#1 Olga Petrovskaya, RN, PhD,² and Francis Lau, PhD, FCAHS, CTSS¹

Author information Article notes Copyright and License information PMC Disclaimer

This article has been corrected. See J Med Internet Res. 2021 August; 23(8): e32421.

Background

Patient portals have emerged as a recognized digital health strategy. To date, research on patient portals has grown rapidly. However, there has been limited evaluation of the growing body of evidence on portal availability, use, clinical or health behavior and outcomes, and portal adoption over time.

Objective

This paper aims to comprehensively consolidate the current state of evidence on patient portals using the umbrella review methodology, introduce our approach for evaluating evidence for quantitative and qualitative findings presented in included systematic reviews, and present a knowledge translation tool that can be used to inform all stages of patient portal adoption.

Methods

For this study, a modified version of the Joanna Briggs Institute umbrella review method was used. Multiple databases were searched for systematic reviews focused on patient portals, and the final sample included 14 reviews. We conducted a meta-level synthesis of findings from quantitative, qualitative, and mixed methods primary studies reported in systematic reviews. We organized the umbrella review findings according to the Clinical Adoption Meta-Model (CAMM). Vote-counting, GRADE (Grading of Recommendations, Assessment, Development, and Evaluations), and CERQual (Confidence in the Evidence from Review of Qualitative Research) were used to assess the umbrella review evidence.

Results

Our knowledge translation tool summarizes the findings in the form of an evidence map. Organized by the CAMM categories, the map describes the following factors that influence portal adoption and effects over time: patient contexts, patient's interest and satisfaction, portal design, facilitators and barriers, providers' attitudes, service utilization, behavioral effects, clinical outcomes, and patient-reported outcomes. The map lists the theories and mechanisms recognized in the included portal research while identifying the need for business models and organizational theories that can inform all stages of portal adoption. Our GRADE and CERQual umbrella review evaluation resulted in the majority of evidence being rated as moderate to low, which reflects methodological issues in portal research, insufficient number of studies, or mixed results in specific focus areas. The 2 findings with a high rating of evidence were patients' interest in using portals for communication and the importance of a simple display of information in the portals. Over 40 portal features were identified in the umbrella review, with communication through secure messaging and appointment booking mentioned in all systematic reviews.

Conclusions

Our umbrella review provides a meta-level synthesis to make sense of the evidence on patient portals from published systematic reviews. Unsystematic and variable reporting of portal features undermines the ability to evaluate and compare portal effects and overlooks the specific context of portal use. Research designs sensitive to the social, organizational, policy, and temporal dimensions are needed to better understand the underlying mechanisms and context that leverage the identified factors to improve portal use and effects.

Search Strategy and Inclusion or Exclusion Criteria

In April 2018, the original search was conducted in 9 databases: MEDLINE, EMBASE, CINAHL Plus with Full Text, Web of Science Core Collection, Scopus, the Cochrane Database of Systematic Reviews, the PROSPERO registry, the JBI Database of Systematic Reviews and Implementation Reports, and Proquest Dissertations and Theses. An updated search in MEDLINE was conducted in November 2019 to identify the SRs published since the initial search. The complete search strategy is provided in Multimedia Appendix 1. The key inclusion criterion was specific to SRs focused on patient portals (irrespective of population groups and study designs) and published since 1990 in English. The SRs that were excluded were those with multiple eHealth technologies, standalone (ie, not tethered) personal health records, those focused on low- and medium-resource countries and thus contextually unique, reviews of reviews, scoping and integrative literature reviews, and reviews that do not provide a complete list of included PSs or designs.

Review Selection and Critical Appraisal

Citations were imported to Covidence. Two researchers independently screened titles and abstracts and then full-text articles against the inclusion and exclusion criteria. Discrepancies were resolved by consensus between 2 researchers and/or by a third researcher. The methodological quality of each SR was independently assessed by at least two researchers using the JBI critical appraisal checklist for SRs consisting of 11 questions [26]. Upon reaching a consensus among all researchers, low-quality SRs with a cutoff point <6 were eliminated.

Our initial database search for SRs yielded 158 citations. After eliminating duplicates and screening for relevant titles and abstracts, we retained 40 SRs for full-text assessment. By applying the inclusion or exclusion criteria, we identified 16 SRs for critical appraisal. An updated search in November 2019 identified 108 citations, yielding 6 SRs for full-text assessment, 3 of which underwent critical appraisal (the list of excluded reviews from the full-text assessment is provided in Multimedia Appendix 2 [14-18,24,26,29-36]). The 19 SRs included in critical appraisal had scores that ranged from 3 to 11, with 6 reviews scoring 10 or 11 (Multimedia Appendix 2). Five SRs with scores <6 were excluded, leaving 14 reviews as the final selection for synthesis [14-18,24,29-36]. Figure 1 shows the Preferred Reporting Items for SRs and Meta-Analyses flow diagram for the selection process.

Analysis and Synthesis

In this paper, we refer to the findings at 3 different levels. PS findings are the individual findings reported by the SR authors in the results section and appendices of the SRs. In turn, SR findings represent the synthesis by SR authors reflecting how they combined PSs. The umbrella review findings are our synthesis of the PS findings reported in included SRs and of any conceptualizations or models advanced as the outcome in the included SRs.

Initially, we categorized the included SRs according to the logic underpinning their approach to synthesis [25,37]: reviews following the aggregation logic (n=13) in which the SR authors reported outcomes based on the summary of individual PS findings and reviews following the configuration logic (n=1) where the SR authors' analysis combined findings from PSs to articulate a theory. Figure 2 shows our categorization of the included SRs for data extraction and synthesis. We grouped all included aggregation SRs into (1) group A: purely quantitative reviews (n=3) where the SR authors only reported on findings from quantitative PSs, and (2) group B: reviews with a mix of quantitative, qualitative, or mixed methods PSs (n=10). There were no purely qualitative reviews (ie, those with exclusively qualitative PS design).

One researcher extracted all the data, and 100% of the outputs were validated by at least one other researcher, with discrepancies resolved by discussions among the 3 researchers. All relevant PSs, their design, sample size, and all findings, as reported by the SR authors were extracted into Excel tables. Duplicate findings from duplicate PSs were removed to manage an overlap among the reviews. Data were synthesized into themes and then a smaller number of domains and statements. No statistical meta-analyses or subgroup analyses were performed due to heterogeneity of the study design.

As a precursor for rating the strength of umbrella review quantitative evidence and the confidence in umbrella review qualitative evidence, our 2 main data extraction documents were quantitative and qualitative Excel tables, respectively. The headings and/or introductory sentences in the paragraphs describing findings or results in each SR were used for initial domain categorization. Domains were further developed through an iterative process of cross-checking with the Clinical Adoption Meta-Model (CAMM) framework, sorting within Excel documents, and weekly analysis meetings among the researchers.

Our quantitative data extraction table included those PSs from SRs (both group A and group B), where the SR authors referenced statistical significance (P values, significance, and confidence intervals) of PS findings. Our qualitative data extraction table included those PSs from SRs in group B, where the SR authors provided sufficient information about the study design for the reader to classify those PSs as qualitative. After removing duplicates, the quantitative table included 360 unique individual findings from 10 SRs [14-17,30-35], and the qualitative table included 246 unique findings from 10 SRs [14,15,17,24,29-31,33,35,36]. Three SRs that were part of our umbrella review included papers that were a mix of PSs and reviews [17,24,35]. As an additional strategy to manage duplicates, we only extracted data from the PSs included in these SRs.

In addition, to extract all other data not fitting the 2 aforementioned tables, we created mixed methods and quantitative descriptive Excel data extraction documents, similar to the designs of PSs as reported in SRs. Our mixed methods table included 76 unique findings from mixed methods PSs from SRs in group B. The quantitative descriptive table included 393 unique findings from quantitative PSs from SRs in both groups A and B for which no statistical data or significance were provided by the SR authors, and which were unsuitable for evaluating the strength of evidence.

Rating the Umbrella Review Evidence

As the concluding step, 2 researchers independently assessed the strength of evidence for quantitative umbrella review finding statements and the confidence in the evidence for qualitative umbrella review finding statements. For this purpose, we developed meta-level umbrella review tools, GRADE-UR (Grading of Strength of Evidence for Quantitative Research at the Level of an Umbrella Review) and CERQual-UR (Grading of Confidence in the Evidence of Qualitative Research at the Level of an Umbrella Review), by applying a voting-counting method [38] and adapting GRADE [39-41] and CERQual [42-44] SR evaluation tools. The methodological approach was conceptualized independent from the GRADE working group. The GRADE-UR and CERQual-UR acronym was created by the authors of this paper to reflect an adaptation of GRADE and CERQual.

The GRADE-UR tool rates the strength of evidence as high, moderate, low, or insufficient [39-41]. Briefly, GRADE-UR evaluation of the quantitative finding statements is based on the following information: the SR authors' critical appraisal of PSs, the PS sample sizes, the number of RCTs, reporting of statistical significance, the number of PS findings that agree with the umbrella review finding statement, and the outcome measures used in PSs. The CERQual-UR tool rates confidence in the evidence as high, moderate, low, or very low. CERQual-UR evaluation of the qualitative finding statements is based on the following information: specific questions from our critical appraisal using the JBI checklist, SR methodological features, and their presentation of the results. A further description of GRADE-UR and CERQual-UR can be found in Multimedia Appendix 3 [26,38-44].

To the best of our knowledge, no tools are currently available for rating the evidence in reviews that synthesize mixed methods and findings from quantitative studies reported in SRs without any reference to statistical analysis; thus, we were unable to assess umbrella review evidence synthesized from our respective Excel tables. However, we compared umbrella review findings from mixed methods or quantitative descriptive PSs with the umbrella review findings from quantitative and qualitative PSs. We looked for differences and correspondences with the goal of formulating any additional umbrella review finding statements arising from mixed methods or quantitative descriptive findings and not subject to GRADE-UR or CERQual-UR evaluation. This information can be found in Multimedia Appendix 4 [14-18,24,29-36].

Output of the Umbrella Review

The output of our umbrella review consists of 2 summary of findings and evidence profile tables, a narrative synthesis, and a knowledge translation tool in the form of an evidence map on patient portal use and impact. CAMM [45] underpins the categorization of these outputs. CAMM is a maturity model used to understand, describe, and explain the adoption of digital health technologies through the preadoption, early adoption, and mature adoption stages [45]. It is a temporal adoption model with 5 dimensions: availability, system use, clinical/health behavior, outcomes, and time. In this review, adoption refers to the planning, implementation, utilization, and support of a patient portal. Availability refers to making the patient portal accessible to users. Use refers to patterns of user interaction and experience with the portal. Clinical/health behavior refers to changes in user behaviors from interacting with the portal. Outcomes refers to the health impact from portal use. Time refers to the transition periods across the 4 dimensions [45].

Population

A total of 6 reviews focused on specific populations: people with diabetes [14,35] or other chronic conditions [17] and pediatric [15], vulnerable [32], and hospitalized [33] patients. Study participants included patients, family members, parents or guardians, and health care providers. A total of 7 reviews reported sample sizes of their included PSs, with sample sizes ranging from 5 participants in a qualitative study [33] to 529,605 in a cross-sectional study [31].

Objectives

Included SRs examined patient engagement [29], facilitators and barriers of portals use [36], meaningful use [17], health literacy [30], mechanisms for achieving portal outcomes [18], effects [17,24,31], and the impact of portals [16,35]. The findings from quantitative PSs reported in SRs pertained to portal enrollment or use levels by sociodemographic factors, the role of portal training, patient satisfaction and empowerment, clinical outcomes including screening rates and treatment adherence, and portal impact on health service utilization [14-17,29,31,32,34]. The findings from qualitative PSs reported in SRs tended to focus on barriers and facilitators; portal design; communication between providers and patients; perceived care quality; providers' concerns; and patient safety, empowerment, engagement, and satisfaction [14,15,17,24,29-31,35,36].

Study Design

Across the 13 aggregation reviews, the number of PSs ranged from 5 [16] to 143 [24], yielding 280 unique PSs of varied designs. The quantitative PSs were classified as 32 RCTs, 18 cohort studies, 8 time series and 16 cross-sectional studies, 3 surveys, 8 pre-post, 2 post only, 1 retro-audit, and 1 quasi-experimental (further information on the design of the PSs for each of the UR findings is provided in Multimedia Appendix 4). In these quantitative studies, data were generated through structured self-report questionnaires, system logs, administrative data sets, and patient medical records. None of the reviews included meta-analyses due to the presence of heterogeneous population groups and diverse measures reported. The designs or methods of data collection used in qualitative and mixed methods PSs encompassed interviews, questionnaires, focus groups, usability studies including observations, and case reports. Critical appraisal tools were used in 7 reviews [14-16,24,31,32,34,35]. Heterogeneity was explicitly discussed as an issue in 4 reviews [16,24,32,34]. Limitations recognized by SR authors included the variability in portal utilization measures [14], portal types and population [31], and portal definitions [33], low quality of reporting [35], limited variety of study designs [31,33], and the potential of underpowered studies [32].

Across the 13 aggregation reviews, we found limited application of models, theories, and frameworks as the conceptual foundation for evaluating patient portal use and outcomes. None of the included SRs applied models, theories, or frameworks as a guide for their review. Some authors briefly referenced models, theories, and frameworks as justification for the review or in their discussion, whereas others summarized the use of models and theories in the included PSs [29,31,36]. Goldzweig et al [31] found that 5 out of 21 PSs used a theory or model, and Irizarry et al [29] found that 11 of 120 PSs used a theoretical framework, but the SR authors did not elaborate on how these theoretical underpinnings related to the outcomes. One SR observed that the chronic care model was cited most often within PSs [29], and 2 SRs commented on how the model’s concept of self-management influences outcomes [14,24]. The following theories were referenced across the SRs: Roger's Diffusion of Innovation [29], activation theory [36], theory of coping and self-determination [36], and grounded theory [sic] [36]. Review authors recommended reconsidering the business model of patient care [34] and developing a framework to identify appropriate outcome measures for long-term portal use [36].

Context

A total of 10 reviews named the countries where the PSs were conducted; most were in the United States, with Europe, Australia, and Asia referenced [14-16,18,24,30,31,34-36]. A total of 8 reviews reported study settings, which varied from hospitals, clinics, group practices, and primary care, or a combination of settings [15,16,24,29,31,33,34,36].

Table 1

Summary of quantitative umbrella review findings and Grading of Strength of Evidence for Quantitative Research at the Level of an Umbrella Review evaluation of quantitative evidence.

Table 2

Summary of qualitative umbrella review findings and Grading of Confidence in the Evidence of Qualitative Research at the Level of an Umbrella Review evaluation of qualitative evidence.

Who Adopts Patient Portals

Many SRs examined the characteristics of patients who use patient portals. Portal users are more likely to be middle aged (≤65 years) [14,17,31,32,34]; White (compared to Black, Hispanic and Asian) [14,15,31,32,34]; and have higher income [14,15,31,35], private insurance [14,15,31,34,35], higher education [14,31,35], or higher illnesses burden or service needs [14,15,31,34]. Females are more active adopters of patient portals when compared to males [17,31,34], except for those with lower socioeconomic status [34]. Among diabetes patients, males are more likely to enroll or use portals [14,35]. Patients with lower health literacy and numeracy skills are less likely to use portals [14,17,30].

Patients' Interest and Satisfaction in Portal Design

Patients are interested and satisfied in using patient portals if they are designed to be easy to use and useful [15,24,33]. The information in patient portals should be transparent and easy to understand, written in nonmedical language, and have a simple display [29,33]. To make portals more patient-friendly and attuned to variations in health literacy, there should be minimal navigation steps and educational information on specific laboratory results, medications, and allergies [15,29,33].

Factors to Consider in Increasing Portal Adoption by Patients

There are many barriers that impede the adoption of patient portals. Patients may forget their password, not have internet access, or lack the necessary interface or literacy and computer skills [14,17,24,30,31,33,35,36]. Portal use may also be discouraged when patients receive unreliable messages, encounter complex language, or have difficulty locating information within the portal [14,29-31]. Patients may also be unaware about portals, or have limited belief in portal benefits [14,17,29,36] and have privacy and security concerns [14,15,24,29,30,33,36]. Facilitation of portal adoption included encouragement from providers and family members [14,29,36] and training and education on portals [32].

System-Related Factors for Portal Implementation

A limited number of reviews addressed these factors. The lack of incentive or reimbursem*nt may dissuade providers from promoting portal use to their patients [14,31]. Two SRs referenced the need for the development of guidelines and governance frameworks for integration of patient portals into organizational processes and to support compliance with regulations [24,33].

Health Care Providers’ Concerns on Portal Implementation

Multiple qualitative studies reported on providers’ attitudes toward portals [24,29,33]. The accuracy of patient-entered information and how portal information can increase patient anxiety or overwhelm, overload, and offend patients were repeated concerns of providers [24,29,33]. Providers preferred control over information to offset concerns about patient safety, privacy, and confidentiality [24,33]. Providers were also concerned about liability; change in workload; and the lack of training, skills, and resources for using patient portals [24,29,33].

Impact on Communication and Patient-Provider Relationships

Communication and influence on patient-provider relationships became a common thread throughout our evidence map. Some SRs found that patients were interested in the potential of using portals for communication with providers [17,30,33] and preferred the convenience, asynchronous aspect, and timeliness of communication afforded within patient portals [15,29,33,35]. Patients who used portals reported satisfaction with communication and the resulting treatment, medications, and care supported through portals [16,31,34,35]. However, some patients lack interest in portals, as they are satisfied with their current mode of communication [14,17].

Three SRs reported on providers’ interest in portals for encouraging patient engagement and communication of complex information with recognition of the potential impact on patient-provider relationships [24,29,33]. In addition, the evidence that portal use improved communication, information sharing, and patient-provider relationships [14,16,31,34] was rated as low.

Portal Use and Impact on Patient Outcomes and Behavioral Effects

Although patient portals may improve patient safety by having patients note and correct errors [17,24,33,35], safe and effective portal use may be compromised because of patients’ limited health and computer literacy [29-31]. Portal use may also reduce caregiver burden and improve the quality of preventive or follow-up care [17,33]. For clinical outcomes, evidence is limited and inconsistent. We found low strength of evidence for changes in blood pressure [16,31,35] and metabolic measures [15,16,31,35] with portal use. We found moderate strength of evidence for improvement in hemoglobin A_1c [15-17,31], preventive care [31,34,35], and medication adherence [16,17,31,33,34].

Impact on Service Utilization and Provider Workload

Limited number of studies have examined changes in the utilization of preventive or testing services [31,33], rate of hospitalizations [16,31,34] and emergency department visits [16,31,34], and no-show rates [17,34], all of which demonstrated low strength of evidence. Although a significant number of quantitative and qualitative PSs examined portal impact on provider workload, the evidence was inconsistent as to how patient portals may impact the number of contacts and face-to-face consults, phone volume, emails or messaging, and office visits [15-17,24,31,34,35]. We found very low confidence in the evidence that portals change provider workload and moderate strength of evidence for no change in workload. A similar pattern of variability was found within quantitative descriptive PSs not included in the aforementioned description: 10 found no change, 12 reported a decrease, and 20 reported an increase in provider workload.

Mechanisms

The review by Otte-Trojel et al [18] included in our analysis hypothesized 4 mechanisms for patient portals to achieve outcomes: patient insight, convenience, continuity of care, and patient activation [18]. Insights that patients gain from their online health information and EHR can improve communication, empowerment, understanding of one’s health condition, and adherence to treatment. Convenience is the time saved when patients have online access to providers and services. Care continuity improves patient-provider communication. Activation leads to empowerment through power balance and self-identity and to better self-care through improved relationships, trust, and availability of educational resources [18]. When we examined how similar concepts were referenced in the other included SRs, we found the following: low strength of evidence for association between portal use and changes to patient empowerment and self-efficacy scores [16,31]; moderate confidence in the evidence that portals could empower patients for self-care and shared decision making [17,24,35]; and moderate confidence in the evidence that portal information could provide patients and parents a greater sense of control, involvement, understanding, and security in care planning [15,33,35].

Time Dimension in Portal Use

The study by Grossman et al [32] was the only SR that reported in detail how portal use changed over time as a result of portal training and education. Bush et al [15] observed that longitudinal studies did not track portal usage over time. Other SRs did not seem to extract information on changes in portal use over time. Similarly, longitudinal changes in clinical outcomes were not highly represented in SRs, with 2 reviews referencing changes in outcomes over multiple time periods [31,35].

Click here to view.^{(20K, pdf)}

Click here to view.^{(40K, xlsx)}

Click here to view.^{(127K, pdf)}

Click here to view.^{(271K, pdf)}

Contributed by

Authors' Contributions: MA and FL developed the intellectual idea for the review. OP and FL drafted the protocol and its various components, with MA commenting on drafts. OP led the submission of the protocol and PROSPERO registration. During early stages, FL led the development of the study, with MA and OP providing suggestions. OP collaborated with a librarian to confirm the original search strategy and managed citations in Covidence. MA and OP updated the search. FL, MA, and OP independently conducted critical appraisal. FL extracted most of the data, with MA and OP validating and/or completing data extraction. FL synthesized quantitative findings, with MA independently validating and/or completing the synthesis. FL led vote-counting and application of GRADE, with MA independently validating and/or completing the evaluation of quantitative evidence. FL drafted the early versions of the manuscript and the knowledge translation tool. MA led all subsequent steps of this study. MA synthesized qualitative, mixed methods, and quantitative descriptive findings, with OP independently validating the synthesis. MA led the adaptation of CERQual tool, development of CERQual-UR method, and synthesis of qualitative statements, with OP contributing throughout the process. MA and OP independently applied CERQual-UR to evaluate the confidence in qualitative evidence. MA developed the final knowledge translation tool, with OP providing feedback. MA led the manuscript development and authored several parts, with OP writing specific parts and contributing throughout. MA, OP, and FL approved the final manuscript and helped to resolve disagreement and reach consensus throughout the study.

3. Lindsay JA, Hogan JB, Ecker AH, Day SC, Chen P, Helm A. The importance of video visits in the time of COVID-19. J Rural Health. 2020 Jun 7;:-. doi:10.1111/jrh.12480. epub ahead of print http://europepmc.org/abstract/MED/32506751. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

5. Using MyHealth Records. Government of Alberta. -. [2020-05-01]. https://myhealth.alberta.ca/myhealthrecords#:~:text=You%20need%20to%20sign%20up,License%20or%20Alberta%20Identification%20card.

6. Killen V. Introducing: MyHealth Records Patient Portal! Alberta Medical Association. 2019. Jan 17, [2020-05-01]. https://www.albertadoctors.org/leaders-partners/ehealth/myhealth-records/introducing-mhr-patient-portal.

7. Online Health and Prescription Services. National Health Service. 2019. Oct 15, [2020-07-21]. https://www.nhs.uk/using-the-nhs/nhs-services/gps/gp-online-services/

8. Citizen Health Portal Improves Access and Helps Patients Better Manage Their Care. Canada Health Infoway. 2020. [2020-07-20]. https://www.infoway-inforoute.ca/en/component/edocman/resources/videos/3546-citizen-health-portal-improves-access-and-helps-patients-better-manage-their-care?Itemid=101.

9. Patient Access Information for Providers. HealthIT. 2019. Nov 6, [2020-07-20]. https://www.healthit.gov/topic/patient-access-health-records/providers.

10. What is a Patient Portal? HealthIT. 2017. Sep 29, [2020-08-01]. www.healthit.gov/faq/what-patient-portal.

11. Are There Different Types of Personal Health Records (PHRs)? HealthIT. 2019. Jul 30, [2020-08-06]. https://www.healthit.gov/faq/are-there-different-types-personal-health-records-phrs.

12. Han H, Gleason KT, Sun C, Miller HN, Kang SJ, Chow S, Anderson R, Nagy P, Bauer T. Using patient portals to improve patient outcomes: systematic review. JMIR Hum Factors. 2019 Dec 19;6(4):e15038. doi:10.2196/15038. https://humanfactors.jmir.org/2019/4/e15038/ [PMC free article] [PubMed] [CrossRef] [Google Scholar]

15. Bush RA, Connelly CD, Fuller M, Pérez A. Implementation of the integrated electronic patient portal in the pediatric population: a systematic review. Telemed J E Health. 2016 Feb;22(2):144–52. doi:10.1089/tmj.2015.0033. http://europepmc.org/abstract/MED/26258289. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

16. Ammenwerth E, Schnell-Inderst P, ho*rbst A. The impact of electronic patient portals on patient care: a systematic review of controlled trials. J Med Internet Res. 2012 Nov 26;14(6):e162. doi:10.2196/jmir.2238. https://www.jmir.org/2012/6/e162/ [PMC free article] [PubMed] [CrossRef] [Google Scholar]

17. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res. 2015 Feb 10;17(2):e44. doi:10.2196/jmir.3171. https://www.jmir.org/2015/2/e44/ [PMC free article] [PubMed] [CrossRef] [Google Scholar]

18. Otte-Trojel T, de Bont A, Rundall TG, van de Klundert J. How outcomes are achieved through patient portals: a realist review. J Am Med Inform Assoc. 2014;21(4):751–7. doi:10.1136/amiajnl-2013-002501. http://europepmc.org/abstract/MED/24503882. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

27. Pollock A, Campbell P, Brunton G, Hunt H, Estcourt L. Selecting and implementing overview methods: implications from five exemplar overviews. Syst Rev. 2017 Jul 18;6(1):145. doi:10.1186/s13643-017-0534-3. https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-017-0534-3. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

28. Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011 Feb 3;11(1):15. doi:10.1186/1471-2288-11-15. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-11-15. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

29. Irizarry T, deVito AD, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res. 2015 Jun 23;17(6):e148. doi:10.2196/jmir.4255. https://www.jmir.org/2015/6/e148/ [PMC free article] [PubMed] [CrossRef] [Google Scholar]

32. Grossman LV, Creber RM, Benda NC, Wright D, Vawdrey DK, Ancker JS. Interventions to increase patient portal use in vulnerable populations: a systematic review. J Am Med Inform Assoc. 2019 Aug 1;26(8-9):855–70. doi:10.1093/jamia/ocz023. http://europepmc.org/abstract/MED/30958532. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

33. Kelly MM, Coller RJ, Hoonakker PL. Inpatient portals for hospitalized patients and caregivers: a systematic review. J Hosp Med. 2018 Jun 1;13(6):405–12. doi:10.12788/jhm.2894. http://europepmc.org/abstract/MED/29261819. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

37. Sandelowski M, Voils CI, Leeman J, Crandell JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012 Oct;6(4):317–31. doi:10.1177/1558689811427913. http://europepmc.org/abstract/MED/23066379. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

38. Lau F, Kuziemsky C, Price M, Gardner J. A review on systematic reviews of health information system studies. J Am Med Inform Assoc. 2010;17(6):637–45. doi:10.1136/jamia.2010.004838. http://europepmc.org/abstract/MED/20962125. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

42. Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, Bohren MA, Tunçalp O, Colvin CJ, Garside R, Carlsen B, Langlois EV, Noyes J. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018 Jan 25;13(Suppl 1):2. doi:10.1186/s13012-017-0688-3. https://implementationscience.biomedcentral.com/articles/10.1186/s13012-017-0688-3. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

45. Price M, Lau F. The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems. BMC Med Inform Decis Mak. 2014 May 29;14:43. doi:10.1186/1472-6947-14-43. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-14-43. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

46. Otte-Trojel T, de Bont A, Rundall TG, van de Klundert J. What do we know about developing patient portals? A systematic literature review. J Am Med Inform Assoc. 2016 Apr;23(e1):e162–8. doi:10.1093/jamia/ocv114. http://europepmc.org/abstract/MED/26335985. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

49. Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice--adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol. 2013 Aug;66(8 Suppl):S12–20. doi:10.1016/j.jclinepi.2013.04.005. https://linkinghub.elsevier.com/retrieve/pii/S0895-4356(13)00154-6. [PMC free article] [PubMed] [CrossRef] [Google Scholar]

50. Ammenwerth E, Lannig S, Hörbst A, Muller G, Schnell-Inderst P. Adult patient access to electronic health records. Cochrane Database Syst Rev. 2017;6:CD012707. doi:10.1002/14651858.CD012707. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6481623/ [PMC free article] [PubMed] [CrossRef] [Google Scholar]

51. Sun R, Korytkowski MT, Sereika SM, Saul MI, Li D, Burke LE. Patient portal use in diabetes management: literature review. JMIR Diabetes. 2018 Nov 6;3(4):e11199. doi:10.2196/11199. https://diabetes.jmir.org/2018/4/e11199/ [PMC free article] [PubMed] [CrossRef] [Google Scholar]