Colonists had settled in North America for nearly 200 years before they pushed beyond the Appalachian Mountains in the great westward migration that led to the settlement of the rest of the vast continent. In the same way, organizations that today use business intelligence (BI) primarily to analyze historical data to ascertain trends and assist with strategic planning decisions are now pioneering the use of BI to manage operational processes.

By so doing, they are opening an enormous new territory for the application of BI that promises to empower workers, both inside and outside the organization, with information and insights to work more proactively and productively.

Operational BI represents a turning point. Traditionally, BI has been the province of business analysts who spend many hours with sophisticated tools analyzing trends and patterns in large volumes of historical data to improve the effectiveness of strategic and tactical decisions. But operational BI changes this equation. It intertwines BI with operational processes and applications that drive thousands of daily decisions—in essence, merging analytical and operational processes into a unified whole.

enlarge Operational business intelligence (BI) includes a variety of approaches and architectures. Most organizations implement different levels of operational BI over time.

In context
Most BI solutions provide an aggregate or snapshot view of business performance by department or product. However, operational BI can track the activity of each step in a multi-step business process that may cut across departmental boundaries. It can also anticipate activity as well as facilitate responses or actions in well-known business processes.

Operational BI delivers information and insights on demand to all workers—from the shipping clerk to the CEO—so they can work smarter and faster to achieve critical business objectives.

In its sophisticated form, operational BI encapsulates business insights into rules and models that organizations can use to automate decisions and responses, eliminating the need for human intervention. Automating decisions not only streamlines processes and reduces costs but also improves service and gives organizations a competitive advantage in the marketplace.

The framework
Operational BI encompasses many different approaches, architectures and technologies, and it can be used to support a variety of decisions and accomplish multiple tasks. While most organizations already support some form of operational reporting, many have yet to embrace more complex types of operational BI that generate greater business value. Thus operational BI gives organizations a chance to reap greater dividends from their BI investments.

Figure 1 represents TDWI's framework for understanding the levels that organizations can ascend as they gain experience and sophistication in the use of BI techniques and technologies. As organizations reach new levels of operational BI, the data latency—the time between an event and when data about that event is presented to a user or application—shortens, sometimes to zero in event-driven systems that automate decisions and actions.

Increasing sophistication
Operational reports allow users to analyze operational processes using traditional reports. They can be implemented in many ways, but two major approaches to operational BI stand out. Companies can build operational reports by querying transaction systems directly, or they can offload the transaction data into an operational data store or real-time data warehouse and query the data there.

enlarge An event-driven analytic engine captures and correlates a continuous stream of business events to provide real-time insights into operational processes and automate responses through alerts and business workflows.

Operational dashboards involve monitoring business processes using performance dashboards. An operational dashboard is, in effect, a graphical report tailored to each user that focuses on a few metrics that represent the performance of key operational processes populated with near-real time data. Also, dashboards can proactively alert users to exception conditions through e-mail or wireless devices.

Composite applications facilitate processes. Companies can reach this level by embedding metrics or reports within operational applications or portals. In other words, instead of requiring users to use two different systems—one to run the process and another to analyze it—organizations can embed the analysis directly into the process and the application that drives it. The many approaches to this include embedding source code or remote function calls within applications and using a service-oriented architecture to stitch together application components across different platforms and systems.

Event-driven analytic platforms are perhaps the most exciting stage of operational BI when an organization uses analytics as an engine to execute processes and workflows. Here, BI does not just provide insights into the process; it is the process. These event-driven analytic platforms (also known by the terms business activity monitoring [BAM], complex event processing, and business process management [BPM]) enable companies to capture business events and apply rules to assist or automate the execution of business processes. (See figure 2.)

These platforms are like intelligent sensors that organizations can attach to their transaction streams. The sensors take a continuous reading of the health of various business processes and trigger actions based on prior experience. In other words, these engines can execute processes and automate decisions. More sophisticated engines use analytical models to improve the sophistication of their rules.

Technical challenges
Despite its promise, operational BI introduces several challenges. It stretches the architectural boundaries of current solutions, forcing BI professionals to rethink the way they design and build systems. Queries must return in seconds rather than minutes or hours, and reports must update dynamically. Operational BI systems must capture large volumes of data in near-real time without degrading the performance of existing processes and jobs on source or target systems. The systems also have less time to recover from a server outage, making it imperative for BI professionals to build resilient, highly available systems with sufficient backup and recovery.

Perhaps the biggest challenge is simply deciding whether to adapt an existing data warehousing architecture to deliver just-in-time data to support operational BI. Many architects believe it is critical to do so. "The big showstopper is whether you are going to apply the same business rules to integrate, cleanse and validate operational data streams as the rest of the data in your data warehouse," says John O'Brien, a BI consultant and former data warehousing architect at several telecommunications companies. Pulling operational streams out of the data warehousing process undermines data quality and creates divergent data sets that may not reconcile, he claims.

But some disagree, believing that a data warehouse becomes a bottleneck if you try to load all data into it that users may possibly want to query. BI vendors that support federated query believe their tools provide an easy, low-cost way to capture real-time data and deliver it. Likewise, vendors of embedded BI, event-driven analytic platforms, composite applications and in-memory analytics believe their offerings provide the most suitable way to meet high-end operational BI requirements. These vendors say a data warehouse is critical for applying historical context to real-time data but not necessary for managing the real-time data itself.

Nonetheless, data warehousing is a well-established IT practice in corporate environments, and many organizations seek to protect their data warehouse investments by adapting the architecture to support just-in-time data and operational processes.

To create a real-time data warehouse, you need a database platform that provides robust support for mixed workload processing. A relational database management system (RDBMS) that supports mixed workload processing lets organizations load current and historical data in the same database and optimize performance across all types of queries. In this way, an RDBMS that supports mixed workload enables organizations to use a data warehouse for operational BI. Organizations should be sure the mixed workload capabilities will meet their needs because the capabilities can vary depending on the vendor.

New territory
Operational BI opens a vast new territory that will make BI a more mission-critical resource for driving operations. Despite the promise, organizations have many technical and business challenges to hurdle. Retooling a data warehousing architecture for just-in-time data delivery is a significant undertaking, while educating users about the need and nature of operational data is another. BI teams that surmount these challenges are already finding that the payoffs are truly worth the effort. T

Survey says Currently, only a small percentage of organizations have implemented a mature operational BI environment, according to a recent report by TDWI Research titled "Best Practices in Operational Business Intelligence: Converging Analytical and Operational Processes." Yet this activity will greatly increase, according to the response from 423 respondents to the question "What's the status of your operational BI environment?" (See figure 3, above.) The TDWI report also shows that many organizations are adapting their data warehouse architecture to support operational BI. About half of organizations (51%) run both operational and analytical reporting from the same environment, according to 225 respondents to the question "Is your operational BI environment the same as your regular BI environment?" (See figure 4, above.) Note: Because of rounding, percentages do not add up to 100. For more information, download the 35-page TDWI report free from http://www.teradata.com/t/page/171360/index.html

Wayne Eckerson is the Director of TDWI Research, an author and a noted speaker and consultant.

Teradata Magazine-December 2007