VistA Evolution: What's Wrong With this Picture?

Rob Tweed The VA has begun awarding a number of very high-value contracts under the umbrella of the VistA Evolution initiative (eg to ASM Research/Accenture), but in my opinion, there are problems looming on the horizon.  From what I understand about the direction that these projects are taking (with encouragement, it seems, from within the VA), there’s a real risk that we’ll see a repeat of previous attempts to modernize VistA, the result of which was very expensive failure with essentially nothing to show for it.  The losers, if this happens, are not only US taxpayers: it’s the Veterans whose future welfare depends on VistA4 being a success.

The US Department of Veteran’s Affairs (VA) has embarked on an initiative known as VistA Evolution, the goal of which is to create a new generation of their Electronic Healthcare Record (EHR) that will be known as VistA4. 

The aims of this initiative are entirely laudable: to modernize the aging, but much loved and very successful, VistA EHR, improve its functionality in areas such as scheduling and make it more accessible for others to build upon using modern tools.  The VA also recognizes the importance of making VistA, or rather key parts of its functionality, accessible from the growing range of mobile devices.

The VA has begun awarding a number of very high-value contracts under the umbrella of the VistA Evolution initiative (eg to ASM Research/Accenture), but in my opinion there are problems looming on the horizon.  From what I understand about the direction that these projects are taking (with encouragement, it seems, from within the VA), there’s a real risk that we’ll see a repeat of previous attempts to modernize VistA, the result of which was very expensive failure with essentially nothing to show for it.  The losers, if this happens, are not only US tax-payers: it’s the Veterans whose future welfare depends on VistA4 being a success.

So, what’s wrong with the VistA Evolution picture that is beginning to emerge, and what do I suggest as an alternative approach that will learn from and avoid past mistakes? In order to answer these question, we first need a bit more background.

VistA, the VA’s EHR

At the heart of the issue is VistA.  VistA is the VA’s EHR that supports and manages every aspect of the healthcare provided to US Veterans. The history of its development is steeped in controversy because VistA arose through the largely clandestine activities of a group of VA programmers and clinicians who worked together as what became known as the Underground Railroad. The VA’s strategic approach at the time was to create a centrally-managed, mainframe-based system. However, whilst the official VA efforts cost a great deal and achieved little, the unofficial efforts of the Underground Railroad, based on a decentralized network of mini-computers, proved highly successful, despite efforts by the VA’s IT management to shut down their work. Eventually, in 1982, the VA capitulated and legitimised their work, naming the product the Decentralized Hospital Computer Program (DHCP).  In 1996 the name was changed to Veterans information system technology Architecture, abbreviated to VistA.

Mumps, the Technology on which VistA is Based

A critically important factor is the technology on which VistA is based. The Massachusetts General Hospital Utility Multi-Programming System, otherwise known as Mumps or often abbreviated to M, was first developed around 1966-7 and was specifically designed with the requirements of healthcare in mind. It is testimony to the design and thinking behind the Mumps technology that even to this day it dominates the healthcare sector, worldwide. Almost all of the most successful EHRs are based on Mumps or a modern derivative. The problem with the Mumps technology is that is almost unknown in the wider mainstream of the IT industry, and the few that have actually heard of it generally have a very poor understanding of it. This is due to a number of reasons:

• It is both a database and a language, the language being integrated and designed for both the definition of business logic and also optimized for access to and management of the database. By comparison, in the mainstream of IT, computer languages and databases are normally expected to be separate entities, so Mumps is wrongly regarded as something of a weird anachronism.
• The Mumps language is considered to be old-fashioned to the extent of being obsolete by the IT mainstream. This is hardly surprising, given that it was designed back in 1966-7. Computer language design has evolved considerably since that time, and the current generation of programmers are not used to what they regard as inherent limitations and shortcomings within the Mumps language.
• The hierarchical Mumps database was also regarded as something of an anachronism during the period between the 1970s and 2009 when relational databases (eg Oracle, SQL Server, MySQL etc) dominated the IT industry.  The emergence of the NoSQL movement in 2009 actually brought the unique design and capabilities of the Mumps database back into favor: seen in this context, it turned out to be an extremely powerful, adaptable, high-performance NoSQL database that predated the modern movement by some three decades.

In my opinion, the success of the Mumps technology within the healthcare sector is entirely due to these unique NoSQL characteristics of the Mumps database. Unfortunately, within the IT industry (and, sadly, within the Mumps community itself), Mumps is considered to be first and foremost a language rather than a database. As a result of the old-fashioned nature of and perceived deficiencies in its language, it gets dismissed as an obsolete technology. This is unfortunate, because if it was viewed as first and foremost a database, a very different conclusion would be reached. The IT mainstream therefore throw a uniquely-capable database baby out with the Mumps language bathwater.

If they viewed Mumps as a database, they would quickly realize that the Mumps language is not mandatory and that it can be replaced with something modern and mainstream: something I’ve been focused on for some time.  For a number of reasons identified in my blog postings, it turns out that JavaScript is the ideal language with which to replace the Mumps language. Furthermore, by abstracting the Mumps database as a persistent store of JavaScript Objects, Mumps becomes a very potent, modern technology. These are the underlying strategic principles behind my EWD.js framework, an Open Source project that is available for use by anyone interested in modernizing Mumps applications such as VistA.

EWD.js in a nutshell

EWD.js is an Apache 2-licensed, Open Source framework for building modern browser-based applications. It is primarily designed to integrate with Mumps databases, and is ideal for modernizing legacy Mumps applications. Key factors are:

• The design and function of EWD.js has been based my own extensive experience gained since the mid-1990s in web technologies and their integration with Cache and Mumps since the mid-90s. In addition to my extensive knowledge of the Mumps technology, I’ve immersed myself into the web browser and JavaScript world, understanding the  mindset and thinking of that community. I’ve also made it my business to watch and understand the significance of the many rapid developments in the browser and JavaScript marketplaces.
• EWD.js is designed around what is known as a stateless architecture. A stateless architecture is essential for providing the kind of high-end scalability and performance that is required to support browser-based, mobile and web service-based applications. It is based on the recognition that, as far as a back-end server is concerned, for most of the time, users are not actually doing anything that requires its attention – they are looking at their screen, reading what’s on it and deciding what to do next. A stateless architecture uses a small pool of processes that service the actual activity of those users as and when they need something to be done on the server. It’s a highly efficient use of hardware resources, and makes it possible to support potentially huge numbers of users.
• EWD.js makes use of the official Node.js interface that has been developed by InterSystems and included with their Cache database. Furthermore, EWD.js includes an abstraction of the basic APIs provided by this Node.js interface’s to expose a Mumps database as if it was a set of persistent JavaScript objects. EWD.js therefore makes a Mumps system meaningful to a JavaScript developer and avoids any need to understand the Mumps language or the need to think of Mumps as an old-fashioned hierarchical database.
• EWD.js applications are different from “traditional” web applications that use a technology known as Ajax. Instead, EWD.js makes use of a newer technology known as WebSockets and allows the creation of what, in effect, are client/server applications that run in a web browser. For those familiar with CPRS, the current Graphical User Interface (GUI) for VistA, EWD.js allows the creation of application that have all the capabilities and power of the CPRS “thick client”, but with all the benefits of using a web browser instead of a thick client. In particular that means no more issues related to installation and maintenance of those thick clients on each user’s computer, and the potential ability to provide secure, fast access over the internet instead being limited to access within the enterprise network.

The diagram below summarizes how EWD.js integrates with VistA. The three boxes colored red indicate the areas where developers are required. Everything else either just works or exists already:

The Need for VistA Modernization

VistA is highly regarded by its users – its functionality, in terms of both breadth and depth, equals and in many areas exceeds that of even the most successful and well-known commercial EHRs. However, that VistA needs to be modernized is beyond question, and in two key ways:

• Its user interface (UI) dates from the late 1970s and is designed around the technology of that time known variously as dumb terminals, green screen devices or roll and scroll. A very successful and well-regarded graphical user interface (GUI) known as the Computerized Patient Record System (CPRS) was released in 1997. However, CPRS uses a technology known as Delphi which is now effectively obsolete. Additionally the style of technical architecture used by CPRS, known as client/server or thick client, has now been largely replaced within the IT industry by web-browser-based thin client approaches.
• its highly-regarded core functionality needs to made more accessible in industry-standard ways, in particular via web services and as what are known as REST-ful services. Doing so would open up VistA, allowing a new generation of developers to build new and modern interfaces and extend its functionality still further, in particular allowing VistA to exploit the exciting new opportunities offered by today’s growing range of mobile devices.

These two deficiencies are, in a nutshell, what VistA Evolution aims to address.

A Step Back into Recent History: The HealtheVet Initiative

VistA Evolution isn’t the first time at attempt has been made to modernize VistA.

In 2001, the VA began an initiative known as HealtheVet, the aim of which was broadly similar to today’s VistA Evolution initiative. Technically, the intention of HealtheVet was to replace the Mumps database of VistA with the Oracle relational database, and to implement an overarching technical architecture that was based around the Java programming language and a set of Java-based services that is known collectively as Java 2 Platform, Enterprise Edition (J2EE), now known as Java EE.

In 2008, the HealtheVet initiative was abandoned after the expenditure of some $600 million. At that time it was estimated that the project would take until 2018 to complete, at a cost of $11 billion. One of the outcomes of this hugely expensive and highly politically embarrassing failure was the decision by Roger Baker, VA CIO (2009-13) to make a significant strategic change in direction and to move the VistA EHR into Open Source.

His rationale, ratified by Congress, was the recognition that VistA was functionally adequate and it made no sense to replace it outright, and furthermore to recognize that there was a growing community of developers outside the VA who had detailed technical knowledge of VistA and were already embarking on modernization exercises, from whom the VA potentially stood to benefit. As part of this new approach, Baker established Open Source Electronic Health Record Alliance (OSEHRA), the custodial agent responsible for managing and coordinating the open source version of VistA.

Why Did HealtheVet Fail?

The official reasons given by the US Government Accounting Office (GAO) for the failure of the HealtheVet initiative focused on management deficiencies, particularly insufficient project management expertise and adequate milestones. What tends to be overlooked is the technical side of the project, in particular the architectural complexity of the Java EE platform on which it was based. This can be clearly seen in one of the official presentation slides that summarizes the planned HealtheVet architecture. This slide was created by one of the project’s technical team at the time:

One interesting thing is that the complexity that is plain to see in this diagram is not actually unusual in a Java EE architecture. The other interesting thing is that the Java EE platform has been very successfully implemented elsewhere and underpins many huge enterprise-scale IT operations around the world. Perhaps the conclusions were right: with adequate management and that eye-watering sum of $11 billion, it could have been successful.

However, as clearly illustrated in the diagram above, with so many moving parts, the HealtheVet project was highly ambitious. Trying to replace the Mumps-based VistA application with a Java/Oracle alternative in one fell swoop made it even more ambitious and risky.

Complexity is Problematic

Even if it had succeeded, there’s another consequence of all those moving parts and complexity: maintenance and ongoing development of the HealtheVet platform would have been extremely difficult, time-consuming and costly. Like any IT platform, it would be fine whilst things worked successfully, but one thing you can guarantee about IT systems is that at some point, a problem will occur. The more moving parts you have in the technical architecture, the more places there are for problems to occur and therefore the higher the probability that problems will occur.

Furthermore, it would have been unlikely that any one person would have been able to, hand on heart, say that they fully understood the workings of HealtheVet from end to end. Much more likely, it would have required a number of specialist teams, each of which would have known about their specific portion of HealtheVet and how it interfaced with its immediately adjacent moving parts. Whilst most of those teams would have been Java specialists, on the user-facing side of HealtheVet (the team who had designed and maintained the User Interface (UI)), the team(s) would have been specialists in HTML, CSS and JavaScript, and unlikely to be capable of easily conversing with the Java teams at a technical level.

So imagine a critical problem had arisen in some part of the HealtheVet architecture.  How would it have been tracked down and who would have been capable of such troubleshooting? One look at that diagram above, and mere common sense should suggest that this would have been time-consuming and problematic at the very least.

Another thing you can guarantee about an IT application is that it is never cast in stone. At some point it will need to be amended and/or extended. Once again, the HealtheVet complexity would have made any modifications very difficult, time-consuming and expensive. Multiple teams would have probably have been involved, and the consequences of even small changes would have potentially rippled down through those many Java EE layers.

Java EE: The Benchmark Architecture

All this having been said, there’s nothing intrinsically wrong with the Java EE Platform, and as I’ve indicated earlier, there are many large organizations around the world that run successfully and depend on the Java EE Platform. In the IT industry it has been regarded as something of a benchmark architecture that offers demonstrable and proven scalability and reliability. Performance, whilst perhaps not exceptional as a result of all those intermediate layers, is usually more than adequate, particularly when the price/performance of hardware keeps increasing in line with Moore’s Law. As a result, it’s hardly surprising that Java EE is the IT architecture that tends to be recommended by the large management consultancies, and is the IT architecture for which most of the big contractors have lots of expertise.

In a nutshell, many, if not most, enterprises have come to accept the downsides of Java EE’s complexity against the benefits of its perceived scalability, reliability and ubiquity. A case of “nobody ever got fired for recommending Java EE”.

The Times They Are A’Changing: Enter Node.js

Over the last three or four years, an interesting change in thinking has begun to take place amongst some of the largest users of the Java EE Platform. Companies such as EBay, PayPal and Walmart, all of whom have relied on massive Java EE Platforms, have begun to replace them with a relatively new alternative technology called Node.js. This is all summed up nicely in this Infographic.

Node.js is a server-side implementation of JavaScript, the language that grew up on and was previously was only found in web browsers. Node.js takes the JavaScript language into the enterprise. It also provides a significantly simpler alternative to the Java EE architecture, and yet it offers all the capabilities of a Java EE Platform.

The benefits that these large organizations are realizing by moving to Node.js are nicely epitomized by PayPal’s experience.  For example:

“Node.js helps us .. by enabling both the browser and server applications to be written in JavaScript. It unifies our engineering specialties into one team which allows us to understand and react to our users’ needs at any level in the technology stack”

“The Node.js app was:

• Built almost twice as fast with fewer people
• Written in 33% fewer lines of code
• Constructed with 40% fewer files”

“.. the Node.js application had:

• Double the requests per second vs. the Java application. This is even more interesting because our initial performance results were using a single core for the node.js application compared to five cores in Java. We expect to increase this divide further.
• 35% decrease in the average response time for the same page. This resulted in the pages being served 200msfaster— something users will definitely notice.”

Similar findings are documented by the other large enterprises that have begun evaluating Node.js and have found that it has revealed shortcomings in their established Java EE architectures that are now beginning to make them question its position as the reference standard enterprise IT architecture.

If you want the difference in complexity between a Java EE architecture and a Node.js-based architecture to be thrown into stark relief, you need look no further than the two diagrams above: compare the EWD.js / Node.js / VistA architecture diagram with the HealtheVet diagram.

So What’s Wrong with the VistA Evolution Picture?

Well, let’s first sum up these various threads I’ve outlined above:

• The VA has already tried and spectacularly failed to implement a Java EE based architecture to underpin VistA modernization.
• Even if the VA had been successful, it would have left a legacy of expensive, complex, risky and time-consuming maintenance and onward development.
• There’s a growing groundswell of opinion amongst some of the biggest and previously most conservative (in terms of IT) companies that Node.js is proving to be a better alternative to the Java EE Platform.
• It turns out that JavaScript is a perfect modern alternative language to replace the aging Mumps language, and the exceptionally capable, powerful, performant and scalable Mumps database can be transformed into a Document Database, storing JavaScript Objects. EWD.js is an Open Source framework, built on top of Node.js, that makes this all possible
• EWD.js allows a legacy application such a VistA to be handed over to the huge and growing new generation of developers who understand JavaScript, and allows them, over time and in a low-risk, measured way, to replace the logic that is currently written in the Mumps language with equivalent logic written in JavaScript.  This contrasts with the high-risk alternative of a whole-scale replacement of VistA or parts of it with some alternative solution that was attempted in the failed HealtheVet strategy.
• EWD.js has emerged from the Open Source community that has been building around VistA – the very community from which Roger Baker wanted the VA to benefit.

What’s wrong with the VistA Evolution picture is that it looks, to me, very much like a repeat of HealtheVet:

• A focus around Java and the Java EE Platform.
• A desire to rip and replace entire parts of VistA with Commercial Off the Shelf (COTS) products instead of building on and enhancing VistA’s own built-in capabilities.

Why should such an approach to VistA Evolution succeed this time around when HealtheVet failed, no matter how much money and expertise was pumped into it? Surely that lesson should have been learnt, or is it the case that there has been a whole-scale replacement of senior management within the VA since HealtheVet was canned, and so there’s nobody left who even remembers HealtheVet?

What’s wrong with the VistA Evolution picture is that the knowledge and expertise of the Open Source VistA Community is apparently being ignored, despite the VA’s supposed commitment to the Open Source-based strategic direction that was instituted by Roger Baker as a direct result of the HealtheVet failure, and despite having set up OSEHRA whose very remit includes the creation of a two-way flow of ideas, solutions and best practices between the VA and the Open Source Community. Instead, it appears that solutions are being prescribed and contracts awarded to companies whose standard solution will be inevitably based around Java and the Java EE Platform, because that’s the skills they have available, not because it’s the most appropriate technology for the VA.

What’s wrong with the VistA Evolution picture is that those large contractors who are being awarded huge and expensive contracts are not being mandated to consult with the Open Source Community, and made to ensure that they don’t drive the VA into a repeat of the HealtheVet attempt to modernize VistA.

Industry leaders are moving away from Java and adopting Node.js.

EWD.js allows the VA to follow suit and modernize VistA with fewer moving parts and much less complexity, and therefore in a much lower-risk way.  EWD.js will allow to VA to realize the same benefits that those industry leaders are reporting.

EWD.js allows the thousands of man-years of effort and skill that has gone into creating VistA to be handed on to the next generation of developers. With EWD.js, VistA, the EHR that consistently comes out top in surveys of user satisfaction, and that is universally loved by its users at the sharp-end within the VA, doesn’t need to be confined to the scrap heap. In the light of the evidence I’ve presented here, surely that would be a serious crime and an act of immense stupidity.

It’s time to fix the VistA Evolution picture. It deserves to be a masterpiece that can stand the test of time and ever-changing fashion.

Isn’t that what Veterans deserve?