Streamlining Workflow Integration in the Cloud

Amazon’s Simple Workflow Service, a SaaS service, streamlines the incorporation of workflow with distributed cloud applications.

Few business applications function independently of other programs. Often business processes are implemented as a series of tasks that are executed by multiple, distributed programs.

For example, a marketing analyst might collect data from an internal sales database and an inventory system as well as a third-party demographics data source.  Once all the data is downloaded, it has to be filtered and integrated according to some business rules established by the analyst.

Next, the consolidated data is analyzed with a statistical analysis program and reports are generated for executives and the results saved for interactive analysis by the marketing team.  This is the kind of process you might want to run on a fairly regular basis except for the fact that it is a tedious process to make sure all the steps run correctly and that none of the steps start before its predecessor is finished.

One option is to write a custom script that calls all the right programs, checks for typical errors, and coordinates the flow of execution. This is a tried and true method that is time consuming, error prone and difficult to scale to a large number of business processes.

A better option is to use a workflow system that abstracts the basic operations in a business process: running tasks and executing business logic.

You could deploy a workflow engine yourself, such as Apahce ODE (Orchestration Director Engine), or you could use Amazon’s Simple Workflow Service (SWS).  As with other software as a services (SaaS), you do not have to deal with deployment and maintenance issues with SWS.  You still have to write code or choose programs to execute specific steps within the workflow, but SWS provides the workflow framework to coordinate the execution of those applications.

SWS allows you to execute distributed processes using an abstraction of a workflow, which is a combination of coordination logic and activities. You register these coordination and activity programs with SWS through a management console. After that, you can execute the workflow as a single logical unit of work. In the case of the data analysis example above, the entire process of collecting, integrating and analyzing data can be started and managed as a single operation.

You will have to develop your applications with SWS model in mind. This means creating three types of actors: workflow starters, deciders and activity workers.

Workflow starters are applications that make calls to SWS API functions to begin the execution of a workflow. This could be an existing application, such as a Web application that responds to user request, or a script that is run by a cron job on a regular schedule.

Deciders are programs that implement business logic. These actors are particularly tied to the SWS model; SWS works with the concept of an event, such as completing a processing step. When an event occurs, SWS starts a task to execute the coordination program and passes that program a history of the workflow up to that point. Histories are data structures that contain information about activities and include attributes such as activity IDs, activity types, input, task lists and other data a decider can use to determine the next step.  The decider actor creates a decision data structure that is sent to SWS. The data structure includes information needed by SWS to determine what process to execute next and what data to pass it as input along with other details.

Activity workers preform operations and communicate the results of their activities back to SWS.  You decide what operations you perform in an activity worker but the program needs to communicate results back to SWS in a standard data structure.

There are additional features of SWS workflows that can be used to implement complex processing requirements, including tasks lists, polling for tasks, service limits and signaling.

Dan Sullivan is an author, systems architect, and consultant with over 20 years of IT experience with engagements in systems architecture, enterprise security, advanced analytics and business intelligence. He has worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail, gas and oil production, power generation, life sciences, and education.  Dan has written 16 books and numerous articles and white papers about topics ranging from data warehousing, Cloud Computing and advanced analytics to security management, collaboration, and text mining.

See here for all of Dan's Tom's IT Pro articles.