After years of handwringing over what cloud computing means and how it should function, a startup called Mesosphere might have the answer. Its vision is that applications and services should be easy to launch, easy to scale and always have the resources they need. It’s the kind of automation you’ll find managing servers inside Google or Facebook data centers or, more accurately, at Twitter.
Mesosphere is in the early stages of commercializing Mesos, an open-source resource-management system developed first at the University of California, Berkeley’s AMPLab and now an Apache Software Foundation project. Among its biggest users are webscale companies such as Twitter and Airbnb, which use it to achieve the same type of data center automation that Google has built for itself via its vaunted Borg system. Companies like these are able to launch new applications in a hurry and ignore dead servers for weeks at a time, if they choose.
This is because in Mesos, resources are all part of a big shared pool, and the system is designed ensure services get what they need. Systems administrators don’t have to wake up in the middle of the night if a server dies, and developers don’t have to concern themselves with the intricacies of building highly available applications. If one server fails, its workloads move elsewhere — automatically.
Enter Mesosphere
Now, Mesosphere is trying to bring Mesos into the mainstream. It has raised seed funding from venture capital firms Andreessen Horowitz, Kleiner Perkins, Foundation Capital, Data Collective and Fuel Capital to help make it happen. The company’s founder and CEO, former Twitter and Airbnb engineer Florian Leibert, describes Mesosphere’s technology as “the way to write distributed [applications] in the future.”
Aside from Mesos, the two other key components of the Mesosphere stack are Chronos (which Leibert helped build at Airbnb) and Marathon, which Mesosphere built itself. We’ve covered them before (when Mesosphere open sourced Marathon in September 2013), but the simple explanation is that Chronos is a framework that runs on top of Mesos and manages scheduled tasks such as Hadoop jobs. Marathon is a framework for launching long-running applications and services (including Chronos or oven more Mesos instances) by, essentially, telling it how many nodes it needs and how much capacity.
“Marathon is like the PaaS … on top of Mesos,” Leibert explained during a recent interview at the company’s San Francisco headquarters. Taken as a whole — the Mesos APIs, Chronos and Marathon — Mesosphere will allow companies to “build your own Heroku, basically, within your data center,” he said.
Leibert hopes more companies and their developers will take note of the platform as a service, or PaaS, analogy because he thinks developer velocity is as big a deal as availability. He has had to build similar systems everywhere he has worked but, he asked rhetorically, “If I could just get that from the get-go .. how much quicker can I move?”
Pretty quick, it turns out. Marketing software startup HubSpot began using Mesos over the summer and is already running about 200 different services on top of it, Leibert said. Many Mesos users are using it to run popular big data frameworks and services such as Hadoop, ElasticSearch, Spark, Storm and Kafka. Airbnb runs the Facebook-created SQL-on-Hadoop query engine Presto on top of Mesos. In mid-February, Mesosphere published a tutorial for launching Cassandra database clusters on Mesos.
A whole new cloud?
But until the company releases the commercial version of its software late this year or early next year, Leibert said Mesosphere’s goal is just to grow the Mesos community (and provide the occasional commercial support for big users). If enough companies begin using Mesos to manage their server pools or clusters (it already has a solid group of public users), it could have a significant effect on the IT business — including by helping companies rethink how cloud computing is done.
As we’ll discuss in great detail at our Structure conference in June, traditional application architectures are really beginning to show their age in an era where everything from services to clients are distributed. For example, server virtualization as delivered by companies like VMware has proven a remarkably successful technology, but not exactly a transformative one. You can cram more applications into a single server, but tools for managing a virtualized environment like a Google data center are not always easy, usually not cheap and certainly not designed with next-generation software in mind.
In the public cloud, users and analysts have been clamoring for improved portability for the better part of a decade without a whole lot of success. Because Mesos users write against the Mesos API, they can run it across any pool of Linux nodes — physical servers, virtual machines or cloud instances — without really worrying about the lower-level difference between them. Airbnb, for example, runs its business on Amazon Web Services, but it could switch cloud providers or even move operations in-house and still get the same experience because Mesos and Marathon handle launching services and managing where they run.
Mesosphere has already created a free (save for AWS charges) tool called Elastic Mesos for people who want to get started using it on Amazon’s cloud. Leibert said HubSpot has cut up to 50 percent off its monthly AWS bill since switching to Mesos and improving its resource utilization in the process.
When people talk about running like Google or Facebook do, they’re largely talking about that sort of automation and efficiency. Mesos looks even better if it helps companies take it even a step further by easing concerns over adopting the kinds of open source hardware designs that Google, Facebook and other large web properties prefer.
The challenge: getting ‘enterprises’ to bite
Mesos has the kind of momentum and user community that probably means it’s here to stay among savvy web companies, but Mesopshere has a much more difficult task. It probably has to convince some of those companies to pay for its commercial software and support eventually, and even more daunting is convincing mainstream companies that they really do need to run as efficiently as Google and that they shouldn’t look to VMware, Microsoft and Red Hat to help them do it.
Other companies have tried similar things the past decade (erstwhile vendors like Cassatt and DataSynapse, whose products are toiling away somewhere inside Tibco right now), but Mesosphere does have open source on its side. And after the success of NoSQL, Hadoop, Linux, and Xen and KVM, among other projects, open source software is riding high right now. Maybe that’s enough to make the dream of automation happen this time around.
Update: This article was updated to correct the names of Mesosphere’s investors to Foundation Capital from Founders Fund, and to Fuel Capital from Brad Silverberg. Also, a clarification: HubSpot saved up to 50 percent off its AWS bill, not more than 50 percent.
