Twitter turns to 'open source' to prevent service disruptions

Twitter started out in 2006 using a monolithic Ruby on Rails application rather than a distributed platform. That worked out fine back then because the service wasn't that busy, but the setup led to growing pains in 2008 when a lot fail whales -- the term Twitter uses to describe service disruptions -- started happening.

...Rather than improving the system and rolling out new features, Twitter's engineers went on "whale hunting expeditions" to solve specific failures, which wasn't really what the company needed to do.

Twitter ultimately decided that it was time to invest in new infrastructure and eventually doubled down on JVM (Java Virtual Machine). This allowed them to break the monolithic, single application into different services such as a service that specifically handles messages, Aniszczyk said.

...To cut costs and reduce the number of machines it uses, Twitter also turned to Apache Mesos, which originally started as a research effort at the University of California at Berkeley. Mesos is a cluster manager that allows users to run multiple processes on the same machine so hardware can be used more efficiently in order to save money, said Aniszczyk.

...One of the lessons Twitter learned is that basing its infrastructure on open source is a good idea. "That is where you find the best software these days," said Aniszczyk...