The management and orchestration of a Google-sized data center may seem like an impossible task for most IT operations. You may have heard of Google's semi-secret data center management software system, Borg. The system treats an entire data center as a single machine; as you can imagine, this would be almost required to manage such a massive infrastructure. See: A Guide to Open Source Cloud Computing Tools
Twitter and Airbnb, among others, have been using a similar system, called Apache Mesos, for some time now. Mesosphere, the company behind Mesos, has just closed Series A funding at $10.5 million following seed funding from Andreessen Horowitz, Kleiner Perkins, Foundation Capital and SV Angel to the tune of $2.25 million.
"Managing your datacenter as if it's a single computer is the holy grail of cloud computing, and Mesosphere actually delivers on that compelling vision," says Brad Silverberg, a Mesosphere investor and former top Microsoft executive, adding that it's "a concept that's been heavily exploited by marketers over the last 20 years, but Mesos is the first technology that actually executes the vision."
Mesos currently runs on Linux infrastructures. The system creates new abstraction layers on top of a distributed file system; it uses the Hadoop Distributed File System (HDFS) as the default. APIs and command lines are then used to control the distributed abstraction layers; supported languages include C++, Python, JVM and Go. The abstractions interact with frameworks such as Apache Spark, Storm and Jenkins, which, in turn, interact with the desired apps.
Mesos can abstract and manage most aspects related to service deployment, VMs and data center scaling. When all is said and done, Mesosphere claims its data center software system can improve resource utilization by up to three times.
The open source system has already proven its viability as a management option for data centers through eBay, Netflix, PayPal and Vimeo among other companies. Now Mesosphere is making the push to expand the system for commercial use.
"Companies that set out to handle scale as efficiently as Google typically stall out because they don't have enough engineering muscle or tools to orchestrate compute resources at this level of sophistication and automation," says Florian Leibert, co-founder and CEO at Mesosphere. "The most efficient path to cloud computing is to run on top of Mesos, and Mesosphere is unlocking that opportunity and making it more applicable for mainstream enterprise developer and ops teams."
Mesos has already added support for Marathon for orchestration and Deimos for Docker.
For more information on Mesos head to Mesosphere's website at mesos.apache.org.