VistA is Going Places, and Also Staying Put

Andy OramThe Veterans Health Administration's hospital software, VistA, is a computing legend. Few pieces of software have become the subject of a popular book (Best Care Anywhere), won repeated awards for their usability, or been credited with a 180-degree turn-around in an organization's quality. But VistA is getting long in the tooth, and many--including now the US Department of Veterans Affacirs (VA) itself--are questioning whether it's time for something new.

The speculations aren't just about VistA. They extend to all health care software of that generation, including the industry's leading electroinc health record (EHR) system--Epic--and the venerable Intermountain Healthcare. All these systems were built on MUMPS, a 1960's-era programming language that survives as a digression from mainstream computing and that I described in my review of Best Care Anywhere. Open source advocate Roger A. Maduro, publisher and Editor-in-Chief of Open Health News, points out that, in addition to its dominant role in EHR software, MUMPS is extensively used in banking, so it certainly has proven its stability.

In itself, a programming language shouldn't be a barrier--for instance, the VistA vendor Medsphere built a Java wrapper around VistA that lets programmers do new extensions with no knowledge of MUMPS (also known by M, by which it became an ANSI standard). 

However, VistA and the other systems mentioned are also tied to a unique hierarchical database (usually embodied in a proprietary product called Caché, although an open source alternative exists, as GT.M), that predates the relational era. In many ways the database resembles the modern crop of “NoSQL” databases.

Like other NoSQL solutions, VistA’s database offers a lot of flexibility and speed, but is ill-suited to business intelligence (BI), reporting, and other analytics workloads. That’s not because of any insurmountable technical limitations of the MUMPS database, but because there’s no industry-wide standard querying language, like SQL, that allows for the creation of a large tooling ecosystem. The current computing environment, where NoSQL databases of various flavors have been widely celebrated, leads programmers to bring a more open mind toward the eccentricities of the MUMPS database. But the database represents an extra learning hurdle for programmers and administrators who might show interest in entering the health care field by working with one of these legacy systems.

Shahid N. ShahControlled Experiment: DoD and VA

VistA has also come away with wounds from recent encounters with the Department of Defense (DoD). Some 30 years ago DoD adopted VistA as their EHR, renaming it the Composite Health Care System (CHCS). Unfortunately DoD decided to continue the development of their version of VistA separate from the VA, and followed the approach of paying government contractors to update and enhance it, instead of leveraging DoD personnel in the ways the VA leveraged its own staff in collaborative development effort.

The DoD then added a graphical user Interface called the Armed Forces Health Longitudinal Technology application (AHLTA), which among other things used a Windows graphical interface coded in Visual Basic (I just got done saying that programming language shouldn't be a barrier, but not all languages are equal). The obvious solution was to converge on a single record for the DoD and the VA, given that--in a best case scenario--the same person will transition from one institution to the other. But the two agencies never agreed on how to come together.

While VistA grew mainly through bottoms-up input from clinician users of the system who were working in an agile fashion to “scratch their own itches,” DoD’s capabilities were usually introduced through top-down traditional systems analysis methods. This difference in development approaches cannot be understated. The development history has created an environment in which the DoD’s users are looking forward to a replacement, while users of the VistA are quite happy with existing functionality but are struggling for ways to innovate on top of it.

A second blow came in 2015 when the DoD turned down a Vista-based product and threw its 11-billion dollar contract instead to Cerner (partnering with Leidos, a systems integration contractor familiar to the DoD). Although the DoD did not cite an official reason for rejecting the VistA-based product, we can see the appeal of choosing a solution in use by many other major health systems instead of one created specifically for the DoD. The argument for Cerner is, essentially, that innovation is slower inside of a huge bureaucratic system (DoD) building a system for its own needs than in a vendor with a broad user base, competing with other vendors and moving with the market. Just like DoD doesn’t build its own jets, tanks, and ships, goes the argument, it shouldn’t build its own EHR system. However, the suitability of a market-driven system depends on whether the DoD’s requirements match those of other customers in the market. Time will tell whether the unique requirements of battlefield medicine and clinical requirements of an unusually young patient population (soldiers “age out” of DoD’s system into VistA) can be handled in the new system.

Now the VHA itself is having a similar debate: keep the home-grown system or migrate to a solution used by others in industry? It seems to be breaking with tradition and appears open to retiring its “legacy” systems because some senior officials believe that VistA cannot innovate fast enough to keep up with new requirements in digital health, mobile device support, IoT, security, privacy, etc. So far, there’s no sign that VistA is unable to flex to new requirements, but the DoD decision was a powerful example to witness and has led some to dream of creating solutions faster and more cheaply through a commercial solution with wide industry adoption.

It is too soon, though, to expect the VA to cast VistA to the winds. After talking recently with several people who have worked on VistA projects, we heard an interpretation of the VA's request for comments that is more nuanced than the dire one presented in the afore-mentioned article.

Seong K. MunA Facelift for VistA

The robustness and usefulness of a computer system has to be measured at two levels, at least: how users view it and how developers view it. So far as users go, VistA's interface is already well-liked, but it's going to get better. Currently, the main interface is the legacy Computerized Patient Record System (CPRS), although the Indian Health Service (IHS) has opted for another interface for their VistA-derived EHR, the Resource and Patient Management System (RPMS).

Last year, work started inside the VA on a web-based interface and data integration platform called Enterprise Health Management Platform (eHMP). Although the move to the Web is one benefit of eHMP, it should really prove its worth as a data integration platform--and here is where it makes developers happy. Any EHR with an API (which is most of them nowadays) can furnish data to the new platform, and the user will not have to know what the underlying organization of the data is.

This easy integration will not lead to the end of VistA, but will allow commercial modules (of the type encouraged during the federal Meaningful Use program) to be integrated. According to Seong K. Mun, president and CEO of OSEHRA, the emerging FHIR standard is integral to eHMP, showing that the VA is staying astride modern trends in health records.

One key factor to consider with APIs, including FHIR, is whether they are read-only (you can get most data out of the system but not put anything in), read-mostly (small amounts of write-back are allowed to the EHR), or read-write APIs (reading and writing data to and from the system is equally supported). While many EHRs now offer read-only or read-mostly APIs, very few allow full workflow integration in a read-write style that’s going to be truly useful in complex systems integration projects. Similarly, the FHIR developers have given significant attention to the reading and sending of data out of a system like VistA, but haven’t prioritized complex workflows that require writing data back into the system.

Roger Baker, the former Assistant Secretary and CIO for the VA, said that the VA has experimented with connecting the MUMPS database and relational databases for years, using different solutions for the many different VistA packages (lab, scheduling, pharmacy, etc.). Their analytics currently run on patient data in a relational database. Baker says that eHMP, six years in the making, will make data available to different databases, so that the clinicians may still access it from the traditional hierarchical database while analytics get it from a relational one. The lab, scheduling, and pharmacy applications are examples of non-clinical parts of the VA system that have their own terminal-based interfaces and use neither CPRS nor the new eHMP.Roger Baker

Baker also points to an earlier attempt, some fifteen years ago, to rip and replace VistA with a new system called Healthy Vet. Baker characterizes that effort as a failure. He expects that, with more than 150 major hospitals using VistA, it will not be replaced anytime soon. Based on our own knowledge of the situation, it could easily take more than a decade.

The VA is certainly watching what the Department of Defense is doing, and Baker thinks the success or failure of the Cerner project will leave lessons for future VA work. Within two or three years, we'll either see Cerner implementations in several DoD facilities, or "DoD leaders will be testifying every month before Congress," Baker predicted. Success may lead the VA to be bolder in pursuing solutions outside VistA. Mun understands that the VA has many enterprise-level issues that might require outside products, which may end up being proprietary, but that VistA will continue to meet its clinical needs. The VA is expected to occasionally review its strategy, which may include both proprietary and open source licenses, for major software programs. Mun believes such reviews should consider the long-term governance strategy suitable for the enterprise IT solution critical for the VA’s unique mission.

VistA Inside and Out

Proprietary and open source versions of VistA have been on the market for many years, but in 2011 a major turn took place in the community: the VA set up a foundation called the Open Source Electronic Health Record Alliance (OSEHRA) and rallied the motley elements of the VistA community around it. A major milestone was the consolidation of all the outside code bases around a single repository shepherded by OSEHRA. Companies that had been used to competition discovered they could all benefit by supporting the consolidated project.

The code base within the VA itself, however, remained separate. Although participants can bring up reasons both technical and historical to leave two separate code bases, we consider it the result of lingering trust and control issues. Plenty of major companies release code to an open repository and use the same version as the rest of the community. But there may be some precedents for the VA aloofness. For use in Android, at one time, Google made changes to Linux that the Linux developers didn't want, so Google just maintained its own version of Linux--but this anomalous phase seems to be over.

Figure 1 shows how the VA, OSEHRA, and wider community of corporate and independent developers interact fruitfully to evolve VistA.  We think the VA would find it much easier to commit whole-heartedly to the community version of VistA, with scattered added modules of its own.

Figure 1 Relationship among VistA contributors. Source-Shahid N. Shah, OSEHRA Is a Great Business Opportunity

Despite the continued preservation of the VA's internal code base, the VA is fulsomely praised by both Baker and Mun for its engagement with the community. The VA syncs its version with the open version regularly, and is contributing significant new features such as eHMP, which the community looks forward to enthusiastically.

In 2014 VA was in need of a new immunization forecasting system, which evaluates the immunizations received by each patient and forecasts the immunizations that might be needed currently or in the future, based on CDC clinical recommendations. The system also needed to allow the EHR to communicate with the doctors (either through doctor queries or through alerts) to indicate which vaccinations the patient should receive at the current visit. Instead of creating a system from scratch, the VA looked at existing products and found an open source version created by HLN Consulting.

Noam Arzt

According to the president and founder of HLN, Noam Arzt, the VA seemed impressed both by the comprehensiveness of HLN's immunization rules and the open source license. The VA organized an OSEHRA workgroup to integrate HLN's open source software, known as the Immunization Calculation Engine (ICE), into its VistA testing environment. The workgroup consisted of a number of community members representing the VA, Indian Health Service, VistA Expertise Network, WorldVistA, MedSphere, HLN, and other volunteers who collaborated for this project. The project thus exemplifies the win-win aspect of open source software collaboration.

According to Arzt, ICE has a robust services-oriented architecture that makes integration easy. The VA had to write service calls to ICE, which are now in testing, and is still working on integrating the display of ICE results into VistA's user interface.

Mun admits that some people are nervous hearing that the VA is again reviewing its commitment to VistA, but he rolled out many other signs of their continued commitment to both VistA and the open source community around it. The VA participates in OSEHRA's summits. VA CIO LaVerne H. Council addressed the OSEHRA annual summit this past June where she extolled open source.

More importantly, the White House recently released M-16-21, which provides specific guidance around Federal Source Code Policy for Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software. Notably, there are specific instructions for Heads of Departments and Agencies from Tony Scott (US Chief Information Officer, CIO) and Anne E. Run (US Chief Acquisition Officer, CAO) that require portions of future custom developed software to be released as open source. One of M-16-21’s key directives is:

This policy also establishes a pilot program that requires agencies, when commissioning new custom software, to release at least 20 percent of new custom-developed code as Open Source Software (OSS) for three years, and collect additional data concerning new custom software to inform metrics to gauge the performance of this pilot.

VA CIO Laverne Council addressing the 2016 OSEHRA conference. Credit - Roger A. Maduro

As far as we know, this is the first time that a CIO has tied open source to the federal acquisition process, which means an integrated open source software strategy might be required for contractors to be paid for their work in the future.

This is not the only major news related to open source federal policies. As detailed in this article, CMS To Invest $5+ Billion a Year in Open Source and Cloud-based IT Infrastructure for Medicaid, the Centers for Medicare & Medicaid Services (CMS) has embraced a new modular, open and agile approach to Medicaid health information technology for the federal government and states.

Since the VA has been practicing “open sourcing” for a while through OSEHRA, there’s a lot that the industry can learn from their approach and success. A good example, according to Mun, is the VA's collaboration around Fileman, a critical part of Vista that handles its database. Fileman has been in the community since it was released 10 years ago, and various community members contributed more than 1,000 bug fixes and many improvements. In 2015, this code was certified by OSEHRA as MS Fileman 22.2 and released under the Apache 2.0 license, upon which the VA accepted it for their own use. DSS Inc. has also adopted the code for their customers, and Mun expects others will follow. Just over the past few weeks, the VA has released its own new version, Fileman 22.2, back to the community, thus completing a full cycle of code. This version will soon be upgraded to Fileman 23.

These are a couple examples out of many, Mun says, where the VA was able to save millions of dollars and more than a year of time by taking advantage of the open source community fostered by VA through OSEHRA. This year, OSEHRA is identifying other existing open source products to integrate into their version of VistA. With introduction of eHMP and quicker adoption of open source products, the community will be able to accelerate the rate of VistA innovation for the VA as well as other users in the states and around the globe.

Significant progress has also been made on open source licensing. Mun found that the VA and community members had inconsistent understandings of the complex relationship involving public domain code, copyright, and open source licensing. But recently the VHA and OSEHRA community clarified the situation. Code developed by government employees enter the public domain. But contractors can apply the Apache 2 open source license to code developed under government contract for VistA, while maintaining ownership of the copyright. This means the contractor can continue to develop the code for other users under a proprietary license, while the community gets the benefit of the open source version, a common practice on many open source projects. Mun thinks this change in VistA will help bring in more developers.

Certainly, Baker says, much more development would have been unleashed had the DoD adopted VistA in some format. The potential market is much smaller without DoD involvement, so it's harder for companies to justify investments in VistA development. But there is substantial progress in both the VHA and the developer community in coding, testing, and collaboration.

It is clear that we are going hear more of VistA.