Why use EWD.js with VistA rather than a standard Node.js framework?

I’ve seen this question posed about EWD.js, something along the lines of:

Why should the VA and others put their faith in and backing behind EWD.js, a framework developed and supported by a small UK company, when there are plenty of well-known and community-supported Node.js frameworks for web application development?

I’d like to try to answer that question fully in this article.

It is quite true that the Node.js community has created a large number of frameworks for Web Application development.  Natural selection has tended to whittle them down, and the most popular and well-known are Express and Meteor.js.  Here’s a more complete list.

InterSystems has included a native Node.js interface for Caché, but you can, if you wish, use a TCP-based connection: the Node.js “Net” module makes it very straightforward to make such a connection.

So why indeed should the VistA development community use EWD.js instead of one of the more well-known frameworks, and why use the InterSystems interface instead of a simple TCP-based connection and a Mumps-based socket-server?

The simple answer is that there’s nothing to stop anyone having a go and “rolling their own”.  But before you do, I’d encourage you to stop and think.  What you get from using EWD.js is:

• A framework that has been built on my 30-plus years of experience of Mumps and Caché, and my 20 years of experience in integrating them with web technologies, working with some of the very largest developers of Caché-based web applications to create internet-facing applications that are fully secure, highly scalable, highly performant and easily maintained (not, as it happens, a simple combination of requirements!).
• My extensive experience of using Node.js in conjunction with Caché and GT.M since early 2011 when Node.js was just over a year old.  I was the first person in the Caché and Mumps community to work with Node.js, and have used it to create a number of non-trivial applications, learning what works well and what doesn’t work well when integrating with Caché and Mumps and what techniques are required for rapid application development.
• A standardized way of working and developing browser-based Caché and Mumps applications, based on tried and tested techniques that allow team collaboration, rapid development and easy maintenance.  Rather than promoting a free-for-all of different techniques and approaches to the use of Node.js, Caché and Mumps, wouldn’t it be better if everyone conformed to a single approach?
• A framework whose underlying aim is to allow the Mumps database and the major legacy applications that are based on Mumps (such as VistA) to be supported, onwardly developed and modernized by the new generation of JavaScript developers, and yet, at the same time, to allow Mumps developers to continue to provide support for the core functionality of those legacy systems for as long as there are sufficient Mumps language skills available.  EWD.js allows this best of both worlds: migration over time to a position where the Mumps language is replaced with the JavaScript language, but equally, EWD.js allows legacy Mumps functionality to be wrapped (by Mumps developers, using the Mumps language, at whatever level of granularity they feel important) in such a way that the functionality is exposed as a JSON-interfaced JavaScript function as far as the Node.js/JavaScript developer is concerned.  With EWD.js there is no need to replace such existing Mumps code, except if long-term support, maintenance and onward development of core functions is seen as an issue in the face of dwindling technical Mumps expertise.

Let me drill down into these issues a little more.

The Benefit of Experience

You can read more about my experience in the About section of my blog site, the EWD Files.

Securing the Future for Mumps and VistA

I’ve been concerned for quite some time about the dwindling pool of developers with Mumps expertise and the disdain with which the wider mainstream of IT views Mumps as a technology.  The situation is unsustainable and approaching a critical state where there will, quite simply, be insufficient people with the knowledge and expertise to understand VistA’s Mumps internals, let alone continue to support and develop it.  The stark and unpalatable truth is that without a viable strategic solution, VistA’s days are numbered, no matter how good an EHR it is.  One of the core design goals of EWD.js is to provide a strategic solution to this critical issue. 

I’ve not seen or read about any other alternative that is workable in the longer term or has any chance of success.

Combining Mumps Knowledge with a JavaScript Mindset

For a solution to successfully allow the new generation of JavaScript developers to carry the baton of VistA development into the future, it needs to do a number of things:

• Work properly and efficiently with the Mumps technology stack in a way that makes the best of the features it provides.  Only someone with broad and deep understanding and many years of experience of the Mumps technology can do that.  The Node.js community does not have the necessary understanding of Mumps and their frameworks were not designed with Mumps in mind.  Indeed, from my experience, there’s barely anyone in the Node.js community who has even heard of Mumps or Caché.
• Avoid the need for immediate whole-scale replacement or re-engineering of logic and code before anything works, and yet allow for longer-term migration to a future where the code can be re-written and expressed in JavaScript instead of the Mumps language.  That’s been a key design goal of EWD.js.
• Aroject the Mumps database and existing core Mumps logic in terms understandable by a JavaScript developer, without the immediate need for JavaScript developers to understand anything about Mumps (that’s not to say that a JavaScript developer shouldn’t learn about Mumps over time).  In order to make this possible, I’ve immersed myself for the last 3 years into the JavaScript and Node.js communities, learning and understanding their mindset and perspective. 

Asynchronous Logic: Callback Hell!

One of the features of Node.js is the use of asynchronous logic whenever any resource is accessed that is not already in memory and that will take time to fetch, for example, files, external web services or data held in a database.  The truth is that asynchronous logic is hard to deal with.  I bear the scars!  It’s difficult and non-intuitive to write, particularly for developers who are used to the more usual synchronous style of coding seen in most other programming languages.  Even more importantly, asynchronous logic is also extremely difficult to maintain.

I have a great deal of practical experience of this: I re-wrote using Node.js an emulation of Amazon’s SimpleDB database that I had originally written in the Mumps language.  Whilst the Mumps version took less than a week to write, the Node.js database took a lot longer and was actually quite an unpleasant task in the end: indeed I had to give up even attempting to re-write the SimpleDB SQL-like query interface logic that had been easy to express as synchronous logic.

Even though I wrote it, I can barely understand how or why my node-mdb project now works!

This experience made me realist just how difficult it would be for a non-trivial Mumps application such as VistA to be integrated with Node.js and what an unmaintainable result would likely ensue even if we could find developers to do it.  What node-mdb proved to me was that whilst simple file access and web service access is pretty simple to express using asynchronous logic, handling the complex business logic that goes hand-in-hand with database access (involving multiple combinations of conditions, loops and recursion) becomes a completely unworkable nightmare very quickly.

The standard Node.js solution to this is to use syntactic tricks that allow asynchronous logic to be expressed in terms that have a synchronous appearance, eg Promises, Streamline etc.  My experience of these techniques is that whilst potentially helpful, they have kinks and aren’t entirely intuitive, and are still a far cry from standard straightforward synchronous logic that the average application developer understands.

Exploiting CPU Cores in Node.js

Node.js applications run in a single process as a single thread of execution.  This means that a Node.js application cannot make use of multiple CPU cores.  Additionally, CPU-intensive logic can kill the performance of a Node.js system.  The standard Node.js solution is a built-in module called Cluster which hives off incoming requests to child processes that share the load.  This would work fine against Mumps systems provided you used a thread-safe, asynchronous interface.

Before the native InterSystems Node.js interface to Caché became available, I experimented with asynchronous TCP-based interfaces (eg node-mwire).  Whilst these worked, it turned out that such interfaces were far too slow for large-scale production systems, mainly because they relied on indirection within the underlying Mumps server code.  Of course, they forced me to write all my database-handling logic in asynchronous code.

The native Intersystems Node.js interface uses Caché’s very low-level call-in interface and is therefore significantly faster than a TCP-based interface backed by Mumps indirection.  It also provides a synchronous version of its APIs.  Unfortunately, Caché is not thread-safe, so simultaneous access from multiple requests from within the standard Node.js Cluster module cause pretty spectacular crashes!  Getting the benefit of the InterSystems Node.js interface plus full utilization of your server’s CPU cores isn’t so simple!

WebLink’s Architecture Revisited

A key part of the design of EWD.js was to build on the tried and tested nature of the WebLink architecture and re-implement it using JavaScript and Node.js. In this architecture, incoming requests are placed in a queue and dispatched to an available pre-forked child process.  Unlike the Node.js Cluster module, but just like WebLink, EWD.js prevents a child process from handling more than one request at a time.  This is crucial and distinguishes EWD.js from the other available Node.js solutions.  

Not only does EWD.js therefore allow full use of the available CPU cores by the child processes, it also allows use of the synchronous, high-performance APIs provided by the InterSystems Node.js interface.  Because each child process connects to Caché via its own instance of the InterSystems Node.js interface, and because only one user’s requests is serviced at a time by each child process, thread safety is not a problem.  Furthermore, because each child process is servicing just a single user’s request, any minuscule delays waiting for a synchronous API to return data from the Mumps or Caché database affect only the user in question.   Meanwhile the core EWD.js process is fully asynchronous, so continues to handle the receipt and queuing of other incoming requests and returning completed responses from other child processes.

EWD.js: The Best of All Worlds

And so, if viewed as a whole. what EWD.js provides is the best of all worlds:

• You don’t need to use pseudo-synchronous syntactical tricks.  All your database-related business logic can be expressed in true, standard synchronous terms, making it fast to write, simple to understand and quick and easy to maintain.  .  You can nevertheless use any standard asynchronous Node.js module for file handling, external web site and web service access etc.
• You can use all your server’s CPUs to the maximum
• The front-end of EWD.js that handles incoming requests and returns outgoing responses is, however, fully asynchronous and runs in a single process: it’s standard Node.js stuff that gives you all the performance expected of Node.js.

EWD.js: Goodbye Ajax, Hello Client/Server

EWD.js has one further trick up its sleeve.  Rather than being a simple HTTP(S) / Ajax-based framework, EWD.js is based around WebSockets and allows you to create what are, in effect, event-driven client server applications that run in the browser.  They are akin to CPRS, except that instead of a thick client, the UI is provided by a standard browser.  Unlike an HTTP/Ajax application, it’s completely event driven, meaning no more inefficient or ineffective polling by the browser when it needs to react to changes on the server or database: the back-end can send a message to the browser at any time of its choosing.  This changes everything as far as the design of browser-based applications is concerned.

Oh and by the way, just to in case anyone has concerns about the high-end scalability of WebSockets, EWD.js provides an option to allow messages sent from the browser to server to be handled using Ajax.  Of course, in doing so you lose ability to autonomously send messages from the server: you’re back to Ajax polling.

Still Interested in Express?

So before trying out something like Express, remember the following:

• You’ll be limited to HTTP / Ajax.
• All your back-end database-handling logic will have to be asynchronous: good luck with that!
• You won’t be able to take advantage of the Cluster module if you want access to the high-performance Intersystems Node.js interface to Cache because it isn’t thread-safe, so you’ll have to revert to a much slower TCP-based interface.
• JavaScript developers will have to understand the Mumps database in Mumps terms, not their own.

EWD.js is Available Today, and Works

Meanwhile EWD.js is available today, it’s working in production Mumps environments such as at Oroville Hospital.  It’s tried, tested, fast, scalable and even includes a secured web service interface and accompanying REST server module.  It allows you to implement and support as many simultaneous applications as you wish.  It allows you to control the run-time environment from a browser (via an EWD.js application named ewdMonitor).  It allows you to debug using the standard Node-inspector toolset.

Critically, EWD.js looks after all the underlying infrastructure of browser applications, leaving you to focus on your application’s logic: the UI logic and back-end database logic.  That’s all you need to write: leave EWD.js to do the rest.

It’s super-fast to develop in, and the code you write is easy to understand and maintain.

Putting All your Eggs in the M/Gateway Development’s Basket?

What about the fact that EWD.js has been written by a small UK company and you’d be putting all your eggs in their basket?   What if Rob Tweed get’s run over by a bus whilst he’s out on his favorite road bike?

Well that’s the very reason it’s been made available as an Open Source project.
It’s actually a very small application in terms of JavaScript code.  it consists of two JavaScript files:

• The core process is just 1650 or so lines of code
• The child process logic is just 880 lines of code

EWD.js is just JavaScript, and not much of it!

Avoid Fragmentation and Let’s not Reinvent Wheels

The more people who use EWD.js, the less reliant you become on support from M/Gateway. I’ve already set up the community site for EWD.js support.

And if everyone uses EWD.js, then everyone is building on top of VistA in exactly the same way, and the VA will be able to share applications and VistA extensions with the Open Source community and vice versa (EWD.js applications run identically, without any change, on GT.M).

I’m passionate about ensuring that the Mumps database that I’ve had such fun working with throughout my career continues in the hands of the next generation.  I’m also passionate that we don’t lose hugely important Mumps applications such as VistA. 

That’s why I’ve written EWD.js and made it available as an Open Source project.  In my opinion, it’s the best hope for the long-term future of VistA and Mumps.  I want people to benefit from my experience in and knowledge of both Mumps and JavaScript, and avoid them wasting their valuable time exploring blind alleys, repeating the mistakes I’ve made over the years with Node.js and reinventing the wheels I’ve created.

Now, let’s start modernizing VistA!