AUTHORS:
Timothy Bennett, Director Strategic Healthcare Initiatives
Todd Margo, Director Business Development
Self-testing has been shown to be an essential testing practice when it comes to establishing a proving ground for standards and implementation guide development. However, with the increased use and support of Health Level Seven (HL7®) Fast Healthcare Interoperability Resources (FHIR®), the question remains “Is self-testing enough to prepare vendors and users for rapid and meaningful adoption of the upcoming complex multi-party FHIR based use cases like Payer to Payer (P2P) data exchange or electronic Prior Authorization (ePA)?”
Drummond’s 25 years of experience in supply chain software interoperability standards testing has demonstrated more than self-testing alone is needed, otherwise adoption of FHIR based process innovation in healthcare will be needlessly slowed and frustrating for all stakeholders. Which leads to the question, “Why do FHIR use cases demand more robust testing practices to maximize their efficacy?”
The answer lies in the healthcare sector’s rapidly increasing support of the FHIR standard.
High Stakes for FHIR
Healthcare providers, payers, the health IT industry, regulators, and other key stakeholders have committed to the FHIR standard for health IT interoperability. Already the industry has experienced the influence of FHIR based data exchange through the Office of the National Coordinator for Health Information Technology’s (ONC) 21st Century Cures Final Rule in 2020, which mandated provider facing health information technology (health IT) applications to support FHIR application programming interfaces (APIs) for patient access to their health data.
Similarly, the Patient Access and Interoperability Final Rule from the Centers for Medicare & Medicaid Services (CMS) in 2020 also mandated that payers support FHIR APIs for their members to access their health and claims data. As a result, hundreds of consumer and provider FHIR based apps have been introduced to the market. While the uptick and deployment have been slow, the foundation for major advances in exchanging clinical data has been set in motion.
Now, buoyed by the CMS Advancing Interoperability and Prior Authorization Process final rule and the Congressional No Surprises Act, another wave of more complex collaborative multi-party bi-directional FHIR use cases will provide a challenging proving ground for FHIR adoption to support further advances towards value-based care.
These initiatives establish that payers support FHIR APIs for P2P data exchange, provider access to patient clinical data from their payer, and an overhaul of ePA. It is further expected that in the forthcoming HTI-2 rule —Health Data, Technology, and Interoperability —the ONC will initiate rulemaking of their own to require provider EHR systems to implement FHIR API support for ePA. These FHIR use cases will require a myriad of health IT systems deployed at thousands of provider and payer locations to interoperate and exchange health data to save billions in needless expenses and improve the care experience. This means upcoming FHIR use cases will require more complex bi-directional data sharing across multiple parties and involve numerous processing steps, with a goal of business process automation. Configurations will include direct connections between organizations or involve intermediary third-party value-add network service(s).
For example, to achieve the benefits of ePA, providers and payers must work together to ensure accurate automation occurs, and eventually, all ePA implementation requirements are supported. Both sides will use FHIR technology built to adhere to standards for data sharing and process orchestration and management, following one or more FHIR Implementation Guides (IGs).
However, the healthcare ecosystem will need to demonstrate success at scale in real production environments involving real healthcare practitioners, real health plans, and real patients to achieve these goals.
But how can the healthcare ecosystem quickly reach high levels of adoption and, as an outcome, more seamless sharing of clinical data? A critical success factor for these upcoming use cases will be to minimize technical interoperability issues, as organizations roll out these new processes in production.
This will require prioritizing rigorous pre-production testing of systems to ensure early and ongoing successful deployments; to avoid disappointment and delay change. Now at this point one may be thinking: what exact problems could develop if one doesn’t prioritize rigorous testing of pre-production systems before going live?
The Dangers of Playing With FHIR
Since FHIR use case processes span multiple organizations, when problems are uncovered after going live (i.e. in production), they will almost always be exceptionally difficult to diagnose and disruptive due to the moving parts and single point of control or governance.
In some cases, when an interoperability problem is found, the connected organizations will resort to a temporary but potentially painful manual workaround. Fixing a defect in a production multi-party collaborative electronic process is an order of magnitude more complex and time-critical than resolving a defect in a self-contained software application used by a single organization. As a result, stakeholders in the FHIR community must minimize the occurrences of post-production FHIR interoperability issues for these upcoming FHIR use cases if they intend to avoid having the improved efficiency of FHIR based data exchange be impacted by the repeated occurrence of interoperability problems. Stakeholders should make it a top priority to ensure participating software systems are robustly verified for interoperability prior to entering production . Once in production, providers and payers will be focused on building their collaborative networks to fully realize the benefits of the new automation for their patient and member populations. In short, the solution to minimizing occurrences of post-production FHIR interoperability issues is rigorous testing.
But the question is, what kind of testing?
And does the nature of these new use cases as multi-party collaborative processes dictate special testing needs?
A Review of Current Testing Practices
Let’s consider common practice today in the FHIR community. Testing, while widely discussed and recognized as important, consists of the following common approaches:
- Developer self-testing,
- Connectathon events,
- Third-party conformance testing (ONC testing for Patient Access as well as the unique Drummond Payer and Patient Access FHIR® API Certification Program and the FHIR® Client App Certification Program)
Let’s consider each of these within the context of minimizing post-production issues for the upcoming collaborative FHIR use cases.
Developer Self-Testing
First, developer self-testing typically tests a single system against a reference server or client, depending on the use case, using testing tools. It could also include a pilot test with another organization (or a few, but this is rare due to added costs and scheduling complexity, etc.).
While Developer Self-Testing has proven to be a very effective method for assessing FHIR interoperability efficacy, it does not attempt to test an implementation against a representative collection of other real-world implementations likely to be encountered in real production environments, which is imperative for exhaustive cross-party FHIR interoperability testing.
As a result, self-testing rarely performs direct interoperability testing with other real-world implementations to confirm the interoperability of pre-production systems.
Connecathons
Let’s now consider Connectathons.
These events are great for bringing together organizations and their respective FHIR technologies to test over two to five days. During this time, testing is rigorous and elevated by a face-to-face collaboration framework that allows participants to communicate in real time. Traditionally, developer attendees test with other assigned attendees to prove a particular use case, profile, or portion of functionality.
That being said, while Connectathons are extremely effective exercises, they don’t lend themselves as well to full interoperability testing of a multi-party collaborative process. This is because testing time for Connectathons are constrained to some pre-event testing through the internet (where possible) and then only a few days of testing during the event, whether on-premises or virtually. As a result, this limits the set of test participants, the time to interact, and the number of test cases, which may range from a few to only one.
Additionally, when issues are found during a Connectathon (inevitable), there is often not a lot of time available for debugging, fixing, and retesting— as Connectathons are designed to flag larger problems with possible solutions, not to develop the required fixes during the event.
3rd Party Conformance Testing
Moving on to 3rd party conformance testing, we will use the ONC (g)(10) example for discussion. For these test criteria for FHIR based Patient Access use cases, EHR products are typically conformance tested against an approved testing method (tool), either the Inferno Framework, AEGIS Touchstone, or other testing tools.
Conformance testing is a valuable supplement to both developer self-testing and Connectathons because it introduces the notion of an independent third-party test and certification by an authorized ACB. However, the ONC test criteria to date have been limited to patient records request use cases. In addition, ONC testing by an ACB tests a single commercial product (an EHR) using a specific testing tool.
This is like developer self-testing but with the benefit of independent third-party oversight and provides the tester with proof of adherence to compliance or the specified standard. However, it doesn’t rigorously test a software system against a variety of expected interacting systems [(g)(10), for example, a collection of FHIR Client Apps likely to be encountered in the real world.
Interoperability Testing–What is Really Needed?
While essential and valuable, existing testing practices and programs are not enough to adequately prepare organizations for production use of FHIR at scale for the upcoming multi-party use cases being discussed. It’s for this reason more rigorous testing is needed— one that simulates the real-world many-to-many (for example, many payers interacting with many providers and vice versa) nature of the upcoming FHIR use cases when they are adopted at scale.
At first, the concept of doing something different than is typically required could be hard to grasp. Organizations may believe that they already spend enough time and resources on testing, and doing something different or new that would offer much greater test coverage and depth would be cost-prohibitive or perhaps not even possible. Fortunately, this is not true, as there is a proven interoperability testing solution—based on its original business in supply chain standards software testing—that’s now been adapted to FHIR.
Full-matrix interoperability testing is an automated testing platform that to date, has been used by supply chain software developers (marketed to large user organizations Walmart, Home Depot, and manufacturers like 3M). It provides a cost-effective and efficient way for software developers and end-user organizations to perform interoperability testing directly and automatically with each other, thereby confirming interoperability before production cutover, and minimizing post-production interoperability issues in the field.
Learn more about the origin and effectiveness of Drummond’s full-matrix interoperability testing—The History of Drummond’s Full-Matrix Interoperability
An Introduction to Full-Matrix Interoperability Testing for the FHIR Community
Drummond has enhanced our proprietary full-matrix interoperability testing software to support FHIR-based testing, which uses native FHIR TestScript and TestPlan resources for test case specification. We refer to the adapted Drummond full-matrix interoperability testing platform as FHIRplace, which is Drummond’s expert-led and automated full-matrix interoperability testing program.
Many may assume that a full-matrix interoperability testing event is similar to Connectathons as full-matrix testing events occur regularly, and organizations voluntarily participate in testing events. But the similarities end there. A FHIRplace full-matrix testing event is fully-remote, highly-automated, rigorous and exhaustive. A test event requires the use of pre-production or production systems, typically lasts several weeks, and provides a safe space for all testing participants. These characteristics combine to support the overarching goal of confirming the interoperability of all participants’ production-ready software systems—an all-or-nothing outcome.
FHIRplace testing events begin with an announcement to the market of a test event and an invitation for participant registration. A test event specifies a set of interoperability test cases that participants must support to pass the testing event process and receive the Drummond Certified seal. Each participant must test with every other participant, as dictated by their role in the test case. For example, in a Drummond supply chain AS2 testing event, each participant tests with every other participant because AS2 is a peer-to-peer protocol (all participants must exercise their roles as both sender and receiver). This would be true for a FHIR use case like P2P Data Exchange. However, for a more complex multiple collaborative process like ePA, where participant roles will differ, each payer participant would test with each provider participant, and vice versa. Since the overarching goal is the success of all participants, collaboration is at the heart of a full-matrix test.
A testing event follows multiple phases designed to weed out all known interoperability issues among participants, leaving a final run phase as mostly a formality prior to the issuance of certification. Diagram 1 below illustrates the general concept of participation for a full-matrix interoperability test, just to show the core idea of each participant testing with every other participant. The illustration includes eight participants (in reality, a testing event would include many more).
Each participant tests against every other participant in their use case, as dictated by a participant’s role in a test case. The testing process proceeds at a timely pace, allocating sufficient time to accomplish the overarching goal of every-to-every direct interoperability testing, which inevitably involves debugging and resolving interoperability issues that arise from the unique full-matrix testing approach. This testing process can be applied to common bi-directional FHIR use cases like P2P Data Exchange and ePA Testing.
It is vital to note that the testing process, while lengthy, is highly automated. For example, this includes test case generation to achieve full-matrix coverage, which can result in thousands of executed tests. This enables test participants to focus on resolving interoperability issues and not administration, making the testing of a variety of bi-directional FHIR uses cases like ePA testing and P2P data exchange extremely efficient. While the steps for each will be different, both exercises will leverage automated testing mechanics and peer-to-peer communication to facilitate the interoperability assessment process.
Discover the intricate details of P2P data exchange and ePA FHIRplace testing by delving into the Unveiling FHIRplace: Optimizing Healthcare Data Exchange blog.
The distinct phases of a test event are Connectivity/QA, Debugging, Dry Run, Final Dry Run, and Certification. By the time testing reaches the Final Dry Run phase, all interoperability issues are expected to be resolved so the Certification Run is more of a formality. In addition, code changes are disallowed once testing reaches the Certification Run. This is obviously another notable advance on Connectathons, whereby the Drummond process ensures only stable software code bases and configurations are allowed to participate in the Certification run. This safeguard protects users and buyers of participating software products—helping them be confident that what they are using or buying is the actual code that passed interoperability testing.
Ongoing Full-Matrix Testing for Maintaining Interoperability
A test participant must integrate with the Drummond FHIRplace platform by developing a FHIRplace Client module. This is a one-time effort and, once completed, enables participation in ongoing test events to maintain ongoing certification. Given FHIR is such a dynamic technology standard (as are emerging and evolving use cases and IG’s, etc.) it’s important to have an ongoing relationship with full-matrix interoperability testing events so that organizations can keep up with the increasing complexity of FHIR use-cases and ensure that their real-world systems continue to interoperate.
Drummond envisions two types of participants, along with — third-party network service providers — routinely participating in full-matrix testing events. The first type is commercial software developers offering products for FHIR use cases or self-developers of such software.
The second group of full-matrix testing participants are end-user organizations, such as provider or payer organizations, that would bring their pre-production implementations to a round of interoperability testing.
If you’re curious about the nuances of full-matrix interoperability testing and its impact on both types of participants check out FHIR Adoption Roadmap: From Self-Testing to FHIRplace and Beyond to learn more.
Summary
To help ensure transformative success of upcoming complex multi-party FHIR use cases, testing must go beyond developer self-testing and Connectathons. While essential, these practices will not prepare systems for rapid production onboarding and scaling. The FHIR community should supplement current practice with a stated goal to minimize the occurrences of post-production software issues since fixing problems in production when multiple parties are involved will be complex and disruptive.
How will the market address the challenges rapid FHIR adoption brings?
With full-matrix fully-automated interoperable testing, Drummond has a proven technology platform and formal testing process adapted specifically to support FHIR-based testing events. Drummond has 25 years of experience using its FHIRplace testing platform, managing 40 full-matrix testing events for supply chain standards software testing. This experience and technology are readily applicable to the needs of the FHIR community. A summary of the benefits of the FHIRplace full-matrix test concept include:
- Interoperability issues not otherwise detectable by other means have repeatedly been uncovered in software systems that have been pre-certified as conformant to the standard. This has helped minimize issues in production-level mission-critical supply chain processes, such as avoiding unplanned outages, costly manual workarounds, rework, and most importantly, avoiding disruption of critical workflows.
- Questions of standards interpretation have arisen frequently during full-matrix testing events, which has encouraged test participants to come together to agree on clarifying an interpretation of some part of the specification. These important conversations create confidence in the implementation but also in the standard overall. These decisions are faithfully documented by Drummond, who acts as an independent mediator among test participants. The clarifications are fed back into a standards specification revision process for improvement.
- Drummond’s FHIRplace platform fully automates full-matrix testing to minimize the test administration burden on participants while maximizing the efficiency of its issue resolution. Without this level of automation, the full-matrix testing concept would not be possible, and the discovery of otherwise undetected interoperability issues would not occur. The automation makes it not only possible, but highly efficient.
- Finally, because the overarching goal of full-matrix testing is the successful testing of each participant with every other participant, a spirit of collaboration is woven into the process, providing a safe space for developers to test with other each other. This has proven vital as it promotes rapid resolution of issues, and effective discussion of issues that require reinterpretation of a standard. Moreover, it carries over to post-testing event real-world situations where two organizations (or developers) might need to quickly resolve an issue (which might have nothing to do with interoperability) and can leverage personal relationships developed during the Drummond testing experience.
Drummond’s FHIRplace full-matrix testing platform is an ideal model for interoperability testing, which includes essential current practice but adds the final touch of real-world interoperability assurance.