Message Queues in the Cloud

Distributed applications often need to exchange data among processes. Coordinating the execution of a workflow and exchanging results of a computation are just two examples.  Message queues are often used for this kind of data exchange.

 You might think exchanging simple messages is simple enough that you do not need middleware for it - don’t kid yourself.

Like so many operations that sound simple at first, message passing can quickly get complicated. This article outlines tips on when to use message queues and the pros and cons of two popular options.

In its most basic form, a message queue is a service that receives data from a sending process and delivers it to a receiving process. This allows us to design system components which can operate fairly independently of each other. This is a critical property of systems that are designed to scale.

For example, you might have one application that takes requests from a Web site to perform some calculation. That application calls a second application which does the actual computation while the first application waits for a response. When the computation is done, the answer is sent to the Web site for the user that initiated the process.

Since the first process waits for the computation to complete before working on the next request, the whole system is limited by the computation.

One way to improve throughput is to increase the number of servers running the computation. If we distribute the requests among multiple computation servers, but still wait for each to finish before sending the next request to a computation server, then we still waste time waiting for each computation to finish.

A better approach is to send requests to computation servers, but not wait for a response. This is where message queues come in. The first process can send requests to the message queue which maintains the list of requests. Computation servers remove messages from the queue, calculate the result, and send the results back to the calling program, possibly using another queue for results.  As the load on the system increases, more computation servers can be added. The extra servers read from the same queue, increasing the rate at which requests are processed.

No changes are required in the application generating the requests.

Messaging services generally implement advanced features, such as transactions, delivery acknowledgement, security controls and other functions required for production quality message delivery.

Dan Sullivan is an author, systems architect, and consultant with over 20 years of IT experience with engagements in systems architecture, enterprise security, advanced analytics and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail, gas and oil production, power generation, life sciences, and education.  Dan has written 16 books and numerous articles and white papers about topics ranging from data warehousing, Cloud Computing and advanced analytics to security management, collaboration, and text mining.

See here for all of Dan's Tom's IT Pro articles.

(Shutterstock image credit: Chalk Drawing)