The Advantages of Message Queues for Cloud Design

Message queues can significantly simplify coding of decoupled applications, and - in the process - improve performance, scalability and reliability. 

One way to take advantage of multi-core processors and clusters of servers available in the cloudis to decouple components. Rather than build a monolithic application that performs multiple functions within a single executable you can build multiple programs that exchange information by sending messages between processes.  This approach avoids some of the common challenges you face with working with multiple processing threads.

For example, you might have a Web service that accepts requests to perform long running calculations. You could design a linear processing model in which you take a request, perform the calculation, return the result and then move on to the next request. The obvious drawback of this approach is that you can quickly accumulate a backlog of requests waiting for processing.   If you are running your application this way in the cloud then you are not taking advantage of a key cloud feature: scalability.

Another way to process a request is to have a process that accepts the requests and another process that performs the calculation.  You can run multiple versions of the calculation process in parallel which returns results to the calling process when the calculation is complete. In this case you will need to manage the back and forth communication between the request accepting process and the calculation process.

Message queues, like RabbitMQ, can significantly simplify coding of decoupled applications, and - in the process - improve performance, scalability and reliability.

Let’s take a look at the main components of RabbitMQ to see how much can be done with a small number of calls to RabbitMQ services. It’s worth noting that RabbitMQ implements the Advanced Messaging Queuing Protocol (AMQP); other messaging services, such as the Amazon Simple Queue Service (SQS), have basic features but do not have all the capabilities described here.

Messaging is an exchange of data between producer and consumer processes. In the simplest case, the producer could send a message directly to a consumer. This sounds simple but in the long run can lead to more complex code. In the case described above, you might run a single producer process while executing ten consumer processes. In the tightly coupled, direct communication model, the producer would need code to distribute messages across all the consumers. Alternatively, you could implement a load balancer in front of the consumers and let the load balancer distribute the jobs.

If the producer has to send messages to multiple backend processes, for example, a calculation service, a logging service, and a reporting service, then three messages would be required. (A simple load balancer would not work here because we are dealing with different services.)

RabbitMQ offers a better approach to hard coding information exchange patterns. When you use the messaging service, you create a producer that writes a message to a broker, which is responsible for accepting messages and routing them to consumer applications.  RabbitMQ brokers consist of three components: exchanges, bindings and queues.  Exchanges receive messages and distribute them to queues where consumer applications can read them.  An exchange may seem like an unnecessary level of indirection but it’s not because of the third component of the RabbitMQ messaging system: bindings.

Sponsored: Learn more about the Cloud and HP Converged Infrastructure.

Dan Sullivan is an author, systems architect, and consultant with over 20 years of IT experience with engagements in systems architecture, enterprise security, advanced analytics and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail, gas and oil production, power generation, life sciences, and education.  Dan has written 16 books and numerous articles and white papers about topics ranging from data warehousing, Cloud Computing and advanced analytics to security management, collaboration, and text mining.

See here for all of Dan's Tom's IT Pro articles.

(Shutterstock image credit: Cloud Data Folder)