Product and service reviews are conducted independently by our editorial team, but we sometimes make money when you click on links. Learn more.

Understanding Big Data Scalability

Understanding Big Data Scalability

The following is an excerpt from Understanding Big Data Scalability written by Cory Isaacson. See more book excerpts and op-ed articles on Tom's Expert Voices section.Understanding Big Data Scalability is the first installment of a four-part series. To order the full version, go to

The dream of an elastic, on-demand application platform has been long-sought, and with today’s Cloud infrastructures the goal is more realistic than ever. High-intensity applications in a variety of areas (as discussed in the section on Today’s Big Data Explosion) require the ability to handle the load from a growing set of users or input volume.

This chapter covers these important questions:

  • What are the common application architectures?
  • What are the important tiers of the application infrastructure?
  • What is required to scale each tier?
  • How can you keep your application available 24X7 with an elastic platform?
  • How is an Elastic Platform supported by a scalable Big Data infrastructure?

The Goals of a Scalable Application Platform

Every new venture is hoping for the next “hot thing” – a huge blockbuster success. Similarly, larger enterprises wish to grow new product introductions, or expand into a major online or mobile presence in conjunction with their traditional offerings. When managing a successful expanding application, the ability to scale becomes a critical need. Whether you are introducing the latest new game, a highly popular mobile application, or an online analytics service, it is important to be able to accommodate rapid growth in traffic and data volume to keep your users happy.

Therefore, the first and most obvious goal of an elastic application is to be able to automatically scale up to meet growing demand.

However, with a truly elastic platform, the objective includes not only scaling up to handle increasing loads, but also scaling down during lower volume periods as shown in Figure 5.1.

Figure 1: Elastic scaling

For example, when operating an application in a public Cloud, managing cost is essential. The same is true in your own data center. Therefore, when resources are not required, it is vital to reduce usage, otherwise significant waste is incurred – waste that threatens the economic viability and profitability of the application provider.

Similarly, many modern applications are based in trends or popularity, and as popularity wanes, it is important to reduce infrastructure to extend the profitable life of the offering.

Lastly, an elastic platform must be reliable.

Application outages are deadly, particularly when they occur in a new venture or product offering. An outage not only causes lost revenues, other ramifications include lost customers or users who do not wish to return and a tarnished reputation.

The Excitement of a High-Growth Success

In my experience with our scalable database technology, we have had the pleasure and opportunity to work with several customers that became virtual overnight successes in their field. A memorable example was a Social Gaming application, one that took off like a shot rocket from its very inception.

The game ramped from 0 to over 5 Million Daily Active Users in just a few short months. I can tell you that there is nothing like the thrill of working on a project like this – it is invigorating, and challenges every aspect of your technical and database knowledge.

This application (and others like it) have a way of exposing every weakness in an application infrastructure in a big way. Not only was scaling the database tier an absolute requirement, keeping it operational for non-stop 24X7 operations was an incredible feat. We saw every failure scenario imaginable due to the high volume, from the underlying public Cloud infrastructure, to the caching layer, to the database tier. The database had to scale so fast we literally expanded the cluster by over 8 times in that first few month period.

I don’t mind admitting that while we believe our technology is solid and very sound (every software CEO will tell you the same), early on there were definitely issues that surfaced which caused some quick rework, patches and demanding support response. We made it successfully through the “bumps”, and helped our customer realize phenomenal success – but it wasn’t without some truly hair-raising experiences and some late nights.

Given the popularity of Big Data, I hope many more top-notch database and application architects see this type of success first-hand, and are able to overcome any obstacles blocking total success. It is the most rewarding thing I know of in this industry.

Cory Isaacson is CEO/CTO of CodeFutures Corporation, maker of AgilData, a Big Data platform designed to remove the complexity and limitations of current data management approaches. Cory has authored numerous articles in a variety of publications including, Database Trends and Applications, and recently authored the book Understanding Big Data Scalability, as well as Software Pipelines and SOA (Addison Wesley). Cory has more than twenty years experience with advanced software architectures, and has worked with many of the world’s brightest innovators in the field of high-performance computing.
Cory can be reached at: