2023 Real World Testing Results

REAL WORLD TESTING RESULTS for MD LOGIC EHR

General Information

Developer Name: MD Logic, Inc. Product Name(s): MD Logic EHR Version Number(s): 7.2

Certified Health IT Product List (CHPL) ID(s): 15.04.04.2785.MDLo.07.02.1.221207

Developer Real World Testing Page URL: www.mdlogic.com/solutions/real-world-testing

Summary of Testing Methods and Key Findings

Case management logs, system logs, and secure email logs were reviewed to determine the frequency of incoming data by type as well as the ability of the module to successfully read and incorporate the contained data. Log files obtained during Real World Testing were de-identified and used for analysis in several areas. Using live client environments, we established test APIs according to publicly available instructions and sent requests for patient data. The ability to receive requests and the accuracy of responses was catalogued and error rates tracked. Various test data was imported to and exported from these cloned systems and CQMs calculated and compared to expected results. Error rates were tracked and reviewed to offer additional explanations for possible data gaps.

Multiple certification criteria were tested simultaneously in this testing plan across two use cases. These metrics and use cases represent the specific interoperability scenarios relevant to the specialty care settings listed in our plan: provider to provider, provider to patient, patient to third party and provider to third party. Real World Testing demonstrated that MD Logic EHR is conformant to the required criteria

§ 170.315(b)(1) Transitions of Care,
§ 170.315(b)(2) Clinical Information Reconciliation and ,
§ 170.315(b)(3) Electronic Prescribing,
§ 170.315(b)(6) Data Export,
§ 170.315(c)(1) Record and Export,
§ 170.315(c)(2) Import and Calculate,
§ 170.315(c)(3) Report,
§ 170.315(e)(1) View, Download, and Transmit to 3rd party,
§ 170.315§(g)(7) Application Access – Patient Selection,
§ 170.315§(g)(8) Application Access – Data Category Request and
§ 170.315(g)(9) Application Access –All Data Request.
§ 170.315(g)(10) Standardized API for Patient and Population Services

Our overall findings are that our systems function as intended. Any errors were the result of the unpredictability of the real-world use of these features and are not encountered while using standardized testing tools.

Standards Updates

Standard (and version)	All standards versions are those specified in USCDI v1
Date of ONC-ACB notification (SVAP of USCDI)	Not applicable
Date of Customer Notification (SVAP only)	Not applicable
USCDI-updated criteria	None

Care Settings

The MD Logic EHR supports the documentation, tracking and sharing of interoperability data within and outside of specialty care settings. For the purposes of testing, our specialty care settings included: General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, Podiatry and Neurosurgery.

Metrics and Outcomes

Use Case 1 (Single Patient) Metrics:

Measure 1: Sharing EHI

Catalogue mechanisms were used to share transitions of care documents, EHI and to track usage of the various transport mechanisms. These include transitions of care for patient referrals, transmitting data to be viewed by the patient via a patient portal and patient transmission of EHI data to a 3rd party.

Associated Criterion:

Certification Criteria

Requirement

§ 170.315(b)(1) Transitions of care

(i)(A) - send transitions of care

(ii)(B) – display a human-readable C-CDA

(iii)(A) – (F) - C-CDA includes the USCDI, Encounter diagnoses according to either ICD-10-CM or SNOMED CT® codes, Cognitive status, and Functional status, reason for referral, and the referring or transitioning provider’s name and office contact information.

(iii)(G) – includes data for patient matching

§ 170.315(e)(1) View Download Transmit

(i)(A) – View transition of care/referral summaries

(i)(B)(2) Download ambulatory summary or inpatient summary using CCD Template

(i)(C) – Transmit to 3^rd party

Relied Upon Software: Surescripts and ViewMyHealthRecords.com

Outcomes:

As the care settings in which this measure was tested consisted of specialty care providers, we expected that the number of outbound referrals sent via secure email would be low. In fact, among 12 client settings over the six month testing period, only 17 outgoing referrals were sent using this method. This is an increase over 2022, but low nonetheless. All were shared successfully, resulting in a 100% success rate. There were no errors.

The more frequent method of sharing EHI was by means of the patient portal (ViewMyHealthRecords.com). The tracked clients provided 20,741 patients access to their health information via patient portal across the six month testing period. No issues were encountered related to the ability of patients to access their data or the readability of the data on ViewMyHealthRecords.com. During these same six months, 4210 patients viewed their health data. Thirteen patients then either downloaded their data or transmitted it to a 3^rd party. No errors were logged or reported. As expected, transmission and readability were successful in all cases.

Challenges Encountered: N/A

Measure 2: Receiving and Incorporating EHI

Track the frequency of incoming CCDA documents via Direct protocol and manual import, and the success rate of incorporating information into the patient chart.

Associated Criterion:

Certification Criteria

Requirement

§ 170.315(b)(1) Transitions of care

(i)(B) - receive transitions of care

(ii)(A) – detect valid and invalid ToC

§ 170.315(b)2 – Clinical information reconciliation and incorporation

(ii) - Properly match a received ToC to the correct patient.

(iii)(B) - (D) - review, validate, and incorporate a patient’s medication list, allergies and intolerances list, and problem list.

Relied Upon Software: Surescripts Direct protocol

Outcomes:

A set of client systems were tracked over a six month testing period. Over that period, 1039 CCDA documents were received via Direct protocol. None were received via manual import. 361 CCDA documents were successfully incorporated into patients’ charts. We attribute the low number of filed CCDAs to several factors:

Clients discarded the CCDA messages before the patient showed up.
Three clients experienced a display error when to incorporating CCDAs into the patient’s chart claiming the CCDAs were not properly filed. We fixed the error and updated each client’s system.
We encountered an issue parsing dates on medications from inbound CCDAs where other changes in the program would not allow blank dates. This issue does not arise when performing standard testing with the EDGE test tool, only in certain real world situations with uncontrolled import data. We fixed the error and distributed updated software to the clients.
Clients received duplicate CCDAs.
CCDAs were received for patients who were never seen by the clients.
There were no attempts to reconcile a number of CCDA messages.

Challenges Encountered: N/A

Measure 3: Electronic Prescribing

Track the frequency of nonscheduled medications prescribed in the specialty care setting and the success rate of timely electronic transmission of those prescriptions.

Associated Criterion:

Certification Criteria

Requirement

§ 170.315(b)3 – Electronic Prescribing

(ii)(A) – send and receive the specified prescription transactions electronically

(ii)(C) – send and receive the reason for the prescription

Relied Upon Software: Surescripts

Outcomes:

Our expectation was that 90% of electronic prescriptions would be successfully transmitted in a timely manner to NewCrop via Surescripts. Over the six month testing period, 12 client systems generated 10,462 electronic prescriptions, 9558 of which were successfully transmitted as expected, for a rate of 91%. While this number is as expected, we were correct in our expectation that the gap between the actual success rate and a perfect success rate was caused exclusively by user error rather than system error. We identified the following to be the primary causes of issues with submitting electronic prescriptions:

Accidentally tried to prescribe a duplicate script to the same patient
A user without permission to process ePrescriptions attempted to send
Prescribed a medication who’s NDC had recently expired and when the replacement was sent, original prescription was never deleted
Attempted to send a controlled substance without have the appropriate subscription to send controlled substances electronically
Prescriptions were documented in the system, but the intended facilities were unable to receive electronic prescriptions (i.e. nursing homes or other long-term care facilities)

Challenges Encountered: N/A

Measure 4: Application Programming Interfaces

Assess the ability of an API to receive and respond to calls in accordance with the publicly available documentation.

Associated Criterion:

Certification Criteria	Requirement
§ 170.315 (g)7 – Application access – patient selection	(i) – receive a request with sufficient information to uniquely identify a patient and return an ID or token
§ 170.315 (g)8 – Application access – data category request	(i)(A) – respond to requests for patient data for each of the individual categories listed in the Common Clinical Data Set and return the full set of data for that category
§ 170.315 (g)9 – Application access – all data request	(i)(A) – respond to requests for patient data for all of the data categories at one time (i)(B) – respond to requests for patient data associated with a specific date and/or specific date range.
§ 170.315(g)10 – Standardized API for patient and population services	(i)(A) Respond to requests for a single patient’s data according to the standard (i)(B) Respond to requests for multiple patients’ data as a group according to the standard (ii)(A) Respond to search requests for a single patient’s data consistent with the search criteria included in the implementation specification (ii)(B) Respond to search requests for multiple patients' data consistent with the search criteria included in the implementation specification

Relied Upon Software: Inferno Test Tool

Outcomes:

We have received zero requests for API access. So internal testing was on select client systems with limited access to fictitious patients.

§ 170.315 (g)7-(g)9 – Application Access- Patient Selection, Data Category Request & All Data Request

On test client systems, we created a test user and gave access to particular patients. The test systems, 24 individual data calls were made for individual and all category requests, and in all cases the expected result was achieved as follows:

Login With Good credentials
- Result: 200: OK
Data Category with invalid category
- Result: 400: Invalid Request
Data Category with invalid access token
- Result: 401: Unauthorized
Data Category: Encounter
- Result: 200: OK
Data Category: Problem
- Result: 200: OK
Data Category: Procedures
- Result: 200: OK
Data Category: CareTeam
- Result: 200: OK
Data Category: Medications
- Result: 200: OK
Data Category: MedicationAllergies
- Result: 200: OK
Data Category: Immunizations
- Result: 200: OK
Data Category: Goals
- Result: 200: OK
Data Category: CarePlan
- Result: 200: OK
Data Category: VitalSigns
- Result: 200: OK
Data Category: SmokingStatus
- Result: 200: OK
Data Category: LabTests
- Result: 200: OK
Data Category: LabResults
- Result: 200: OK
Data Category: UniqueDeviceIds
- Result: 200: OK
Data Category: Assessment
- Result: 200: OK
Data Category: ReasonforReferral
- Result: 200: OK
Data Category: HealthConcerns
- Result: 200: OK
Data Category: FunctionalStatus
- Result: 200: OK
Data Category: CognitiveStatus
- Result: 200: OK
Data Category: All Data
- Result: 200: OK
- Result also includes CCDA document.

§ 170.315(g)10 – Standardized API for Patient and Population Services

The Inferno Test tool was used. All tests were run successfully.

Challenges Encountered: Due to lack of real-world usage of APIs by our clients, this measure was tested via simulations on live client systems. This was anticipated in our Real World Testing plan.

Use Case 2: Population Services

Measure 1: Data Export

Assess the ability of the Health IT to create an export summary for a set of patients with specified parameters as well as the usage of the feature.

Associated Criterion:

Certification Criteria

Requirement

§ 170.315(b)6 – Data Export

(i)(A) – set configuration options for data elements when creating export summaries

(ii) – create export summaries using the Continuity of Care Document template

(iii)(A) – set/enter date and time period

(iv) – set storage location

Relied Upon Software: N/A

Outcomes:

We did not have any reported attempts by our test clients to export data. This was expected from our test plan.

Challenges Encountered: N/A

Measure 2: CQMs: Import

Assess the ability to import a QRDA I file and use the associated data to calculate CQMs.

Associated Criterion:

Certification Criteria

Requirement

§ 170.315(c)2 – Clinical Quality Measures - import and calculate

(i) – import QRDA Category I data file

(ii) – calculate each CQM presented for certification.

Relied Upon Software: Cypress Test Tool

Outcomes:

As expected, we did not have any requests

Clones were made of client systems to test this functionality. Test data was created with the Cypress Test Tool. This test data was imported into the test systems without issue.

Challenges Encountered:

As with the previous year, due to lack of actual real-world test cases of clients needing to import QRDA1 files, we were forced to run simulations using a copy of real client data. A copy was used as to not contaminate a real working environment. This was expected and noted as a likely scenario in our Real World Testing plan, as we have not had any clients that have used the QRDA I import to date.

Measure 3: CQMs: Export

Assess the ability to accurately record data necessary for the calculation of CQMs, as well as the successful creation and export of QRDA I and QRDA III files utilizing that data.

Associated Criterion:

Certification Criteria

Requirement

§ 170.315(c)1 – Clinical Quality Measures - record and export

(i) – record all data necessary to calculate CQMs

(ii) – export QRDA Category I data file

§ 170.315(c)3 – Clinical Quality Measures - report

Create QRDA Category III file for reporting

Relied Upon Software: Cypress Test Tool, Client Submissions

Outcomes:

§ 170.315(c)3 – Clinical Quality Measures - Report
Few of our Real World Testing client partners (and clients in general), are actively using eCQMs to submit their MIPS data. However, for the clients who are participating, data was exclusively gathered via documentation in MD Logic rather than via QRDA I import. In all client systems, CQMs were successfully calculated at the end of the measurement period and accurately reflected the data captured. In addition, QRDA III files were created successfully in all cases and were successfully exported and submitted for reporting MIPS. Any errors encountered were due to missing client data such as NPI or TaxId numbers.

§ 170.315(c)1 – Clinical Quality Measures - Record and Export
Because none of these clients actively used the QRDA I export, we combined testing of this measure with our CQM Import testing using clones of live client systems. Multiple sets of test data were imported into these systems and the resulting CQM calculations effectively incorporated this information. CQMs included in this testing were 22, 50, 69, 117, 123, 131, 1134, 138, 139, 147, and 155. Updated QRDA I data was then exported from the clone environment without error in all cases.

Challenges Encountered:

As with the previous year. Due to lack of actual real-world test cases of clients needing to export QRDA1 files, we were forced to run simulations using a copy of real client data. A copy was used as to not contaminate a real working environment. This was expected and noted as a likely scenario in our Real World Testing plan, as we have not had any clients that have used the QRDA I export to date.

Schedule of Key Milestones:

Key Milestone	Care Setting	Date/Timeframe
Released documentation for Real World Testing and provided to authorized representatives and providers running MD Logic	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Completed prior to Dec, 2022
Selected clients who would capture data	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Completed from Jan - Feb 2023
Began data and information collection as laid out by the plan	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Completed from Jul 2023
Followed-up with providers and authorized representatives on a regular basis to understand any issues that arose with the data collection.	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Completed from, Jul.-Dec. 2023
Ended data collection for the testing period	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Dec. 31, 2023
Analyzed data and generated reports	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Completed from Jan-Feb. 2024
Submitted Real World Testing report to Drummond Group	Podiatry, General Surgery, Plastic and Hand Surgery, Colon Rectal Surgery, ENT, Orthopedic Surgery, & Neurosurgery	Feb 1 2024