Product and service reviews are conducted independently by our editorial team, but we sometimes make money when you click on links. Learn more.

From Decision Support To Big Data: Quantifying Business Operations

From Decision Support To Big Data: Quantifying Business Operations

Techniques for analyzing business data have existed for decades but the advent of formalized practices, big data, and dedicated business intelligence (BI) tools have changed the scope of business topics we can analyze.

If you are just getting started in business intelligence and analytics, here are some tips to help you start a BI practice in your organization. 

The first rule of business intelligence is: start with a business problem. There are so many tools, methodologies, and hardware to work with in BI that it is easy to get caught up in the technology at the expense of the business.  You know you have fallen for the siren call of BI technology when you take the Field of Dreams approach, also known as the "build it and they will come" strategy.  These BI projects are driven more by what data is available and what the BI designers and architects think the eventual users want than by a pressing business need.   

There is a chance that the systems built with this approach will be useful, but there is a high risk of failure.  With a good technical team you have a chance of building a reasonably sound BI platform, but it may not answer the right business questions.  When starting in BI, start with a business problem. Set out to help managers who feel they do not have enough information to understand which products are performing well and which are not, or delivering more informative and timely management reports on product sales.  Give the marketing team the combination of sales and demographic data they need to understand variations in markets and their relationship to sales.

Identify Data Sources, Create A BI Database

Once you have your driving business problem, identify your data sources. This will almost certainly include transaction processing systems in your organization.  You might be tempted to build analysis tools using the transaction processing systems, but you should not attempt it.  

Transaction processing systems are designed for a mix of read and write operations, typically with small amounts of data and short response times to end user operations.  Business intelligence systems generally support many read operations that require large volumes of data. Databases are tuned differently for different tasks.  For example, the transaction processing system might minimize the number of indexes on a database table to improve write performance while a BI database may have many more indexes to improve query response time. 

Perhaps more importantly, BI applications can take advantage of data from multiple data sources.  You may want to analyze data from multiple transaction processing systems as well as data provided by third parties.  A database designed for BI-type analysis will more readily accommodate additional data sources than an operational system tuned for a particular set of processes. 

Formalize Data Collection Procedures

BI and analytics are motivated by the promise of insight into your business, customers, or market.  To get to that "promised land" you must first pass through the rough terrain of the data collection process. In BI parlance, this is known as the extraction, transformation and load part of BI workflows.   Extraction is the process of copying data out of a source system.  This sounds trivial and it often is, but there are exceptions.  Extraction often depends on the types of tools available on the source system.  

If you are extracting sales transactions from a point of sales system, you will probably want to make an initial extraction to collect historical transactions and then perform daily extractions to get each new day's transactions.  This is simple, as long as there is an easy way to identify transactions that have not been extracted already.  Sometimes a simple technique like checking a record created timestamp is available, but in other cases more elaborate methods are required. Change data capture tools specialize in identifying new or altered data and can be useful for efficiently extracting incremental data when simpler methods are insufficient or unavailable. 

Before data is loaded into the business intelligence database, it typically needs to be reformatted and aggregated to fit with the BI data model.  This is the purpose of the transformation stage.  Many tools are available for streamlining the ETL process and they often include support for a wide range of transformations, from simple string manipulation to data quality assessments. 

Choose Appropriate BI Data Model(s)

How you store your data for analysis will largely depend on its volume and the types of analysis you plan to perform. Relational data models work well when you need to preserve detailed information about transactions, but will not require too many expensive join operations.   

Multi-dimensional or cube data models are good choices when you are more concerned with analyzing aggregate data, such as the number of units sold of a particular product in a sales region across a period of the last three months.   

When you have to collect and analyze large volumes of data, e.g. terabytes, you may find big data platforms, such as Hadoop and query tools like Hive, that allow you to extract the information you need without requiring expensive, high-end database servers.  (Hadoop scales horizontally using increasing numbers of commodity servers; relational database servers often scale vertically by adding more cores and memory). 

Invest In Query, Analysis And Visualization Tools

BI designers and developers spend a great deal of time and effort getting data into a data warehouse, but end users need to get it out.  Some users will know enough SQL to query a database but many will benefit from tools that hide the implementation details. Look for query tools that allow users to retrieve target sets of data in an ad hoc manner.  There is no point trying to write queries for them since their needs change, often in response to learning something about their data from a previous query. 

Invest in analysis tools that allow users to better understand the data they retrieve.  This can be as simple as providing sums by categories or exporting data to Excel. In other cases, end users may need more advanced data mining and statistical analysis tools to create models that shed light on aspects of your business.  Visualization tools available today can help render complex interaction and relations in visually accessible ways and complement other query and analysis tools. 

Big Data, Big Hardware, Big Software

As data grows so must the hardware, software and access tools to give it meaning.

VIDEO: Big Data, Big Hardware, Big Software