Teradata's Real-Time Enterprise Reference Architecture:

A blueprint for the future of IT.

by Todd Walter

A flight from San Diego to Chicago encounters a thunderstorm and is re-routed to avoid the inclement weather. A monitoring system receives the new flight plan and transmits that information to the airline's active data warehouse (ADW). Within the active data warehouse, event detection services determine that the flight will arrive 45 minutes late. A message is sent to a set of tactical business services, which perform an analysis of the flight manifest and determine that five passengers will miss their connecting flights to New York and three passengers will miss their connecting flights to Pittsburgh.

The ADW identifies two alternate flights from Chicago to New York and one alternate flight from Chicago to Pittsburgh. After checking the status and seat availability on these flights, it prioritizes the eight customers relative to their profitability, current ticket class, mileage status, preferences and other variables.

The ADW then sends a set of messages to the reservation system to book seats on the flights for the eight passengers affected by the delay; to re-route the baggage accordingly; and finally, an alert is sent to each customer who has provided a messaging method.

During the analysis and processing of the situation, it is determined one of the affected customers recently has experienced multiple delays. An event notification is forwarded to the appropriate application where an apology letter and an upgrade coupon for a future flight are generated and subsequently mailed to the specific customer.

The airline has just transformed a negative situation into a positive demonstration of its commitment to offering superior transportation services by resolving the issue in real-time. A real-time enterprise IT infrastructure allows the airline to take action when it is directly relevant to the customer.

A real-time enterprise (RTE) is an organization that leverages technology to reduce time delays between the occurrence of a significant business event and the intelligently derived set of responses to that event by the organization. Users in a real-time enterprise want access to all system services in a seamless fashion using enterprise portals that simplify interaction between the user and the enterprise. Teradata's active data warehouse provides a robust yet flexible platform for delivering decision-making services to the real-time enterprise.

So how do all these services interconnect?

Teradata's Real-Time Enterprise Reference Architecture serves as a blueprint that illustrates techniques for integrating an active data warehouse into the real-time enterprise. Teradata's RTE Reference Architecture describes an assortment of contemporary technologies that can be used for the integration of data, applications and users with business services throughout the enterprise.

Transactional Repositories

Transactional repositories are a set of specialized databases that provide the "system of record" for all transactions and manage the company's current state. These databases typically hold the minimum amount of historical data required to serve the transaction processing requirements.

For example, when a banking customer withdraws funds from an ATM, various business transactions, such as debit/credit, take place. ATM business services modify one or more transactional repositories, based on the ATM transaction executed, so that the company's bookkeeping entries—debits, credits, balances, etc.—are recorded in a consistent fashion. Each transactional system, e.g. checking, credit card, loans, etc., is finely tuned to minimize response time and maximize throughput for transactional business services.

Analytic and Decision Making Repositories

Analytic and decision making repositories store the "historical record" of the enterprise, detailing every business area and the reference data that describes the business. This "record" can be used for analysis and retrieval to support the enterprise's analytic and decision-making processes.

A wide variety of architectures and components going by many different names (data warehouse, data mart, ODS, hub and spoke, staging area) have been utilized to implement this repository. Teradata strongly advocates a single integrated enterprise repository—an enterprise data warehouse (EDW)—because of its inherent flexibility to ask wide-ranging and changing business questions. In Teradata's RTE Reference Architecture, the analytic and decision making repository is represented by an EDW but is intended to be inclusive of all choices made to house the enterprise's historical record.

Conventional decision making repositories represent historical views that are days, weeks or even a month behind the transactional repositories' current state. In the RTE, data freshness is based on service-level needs dictated by business process. The historical view must be much more current to support decision-making on the frontlines of the enterprise (customer service, factory floor or at the airport gates).

All data freshness levels, from one month to one second, are represented by the analytic and decision making repositories in Teradata's RTE Reference Architecture.

Data Acquisition and Integration

Much of the company's detailed data is born in its transactional repositories. The data acquisition and integration element collects data from various sources inside and even outside the enterprise, integrates across various business areas, then stores the result into the decision making repository.

Traditionally, this data movement was performed as a batch process on a scheduled basis. Extract, transform and load (ETL) tools provide facilities to collect data from a variety of sources; perform the data transformations, cleansing and integration; and move the data to a decision making repository, where it interfaces with database-specific mechanisms to provide high-performance load into the target repository.

As data timeliness requirements for decision-making increase, several toolsets are evolving to provide the streaming, more immediate form of data acquisition. ETL tools have added streaming capabilities to capture data, apply the integration transforms and continuously load the data into the repository utilizing a continuous load facility such as Teradata's TPump. Enterprise application integration (EAI) and message bus tools play a role in streaming data from continuous sources to the decision making repository. An adapter on the bus might apply the data directly, or it may be used in concert with an ETL tool to perform the transformations and apply the data. Heterogeneous replication tools can gather changed data directly from transactional repositories and pass it to an ETL tool or apply it directly to a decision making repository.

(For more about data timeliness, see "The ins and outs of the Teradata active warehouse" by Todd Walter. For more details about Teradata's Data Acquisition and Integration tools and strategies, please read "Choosing wisely.")

Transactional Services

Transactional business services are a set of application components, each focusing on a narrow set of business processes. These services usually interact directly with the people on the frontlines who perform transactions with all areas of the enterprise.

The services interact with the transactional repositories using standard database interfaces. Today, they are constructed using service-oriented architecture (SOA) techniques, but legacy client-server applications will continue to exist for the foreseeable future. Transactional services will be a combination of custom and purchased applications that access the current state of the enterprise and record bookkeeping changes to the transactional repositories.

Analytic and Decision Making Services

Analytic and decision making services perform analysis on the integrated "historical record," providing broad-based intelligence that drives the business processes of the enterprise. These services enable strategic as well as tactical decisions. Strategic decision making services provide information necessary to make long-term decisions in the business. Market segmentation, product (category) management strategies, profitability analysis and forecasting are a small sample of the business functions supported. Tactical decision making services provide assistance to people on the frontlines such as customer service representatives, factory floor operators, gate agents, etc., whose actions reflect an implementation of the enterprise's strategy.

(For more information on tactical decision-making, see "Strategically tactical" by Todd Walter.)

Analytic and decision making services interact with the analytic and decision making repositories via standard database access interfaces. Strategic decision-making access typically is provided by client-server applications such as BI, reporting, OLAP, query and data mining tools or analytic applications focused on specific business processes. Frontline applications must deliver to a large, varied and distributed user base. An SOA approach mirroring the transactional services is today's best practice for supporting the tactical decision making services. The tactical business processes on the transactional side are moving more and more to an SOA approach for flexibility. The SOA allows for easier integration of business intelligence right into the business processes.

Enterprise Message Bus and Service Brokers

The Enterprise Message Bus represents a collection of middleware products that enable a diverse range of computing services to interact with each other. Integration can occur at the data level, message level, application level or business process level and can include business automation. XML plays a key role in the integration of applications by providing an application-neutral format for data exchange. EAI components are found in application services, application brokerage services, data acquisition services and user interface environments.

Service brokers are specialized software components that provide dispatched service requests from clients to application service providers. Web Services, .NET and CORBA are three examples of service brokers.

Together, the Enterprise Message Bus and service broker elements provide the infrastructure of an SOA. The SOA model, combined with a portal, allows both transactional and analytic and decision making services to be deployed to the frontline using a common interface and infrastructure. Best practice today is a single common infrastructure and interface to deploy all services to the frontline.

Enterprise Users—Browsers and/or Portal

An enterprise user is any person or application component accessing the enterprise IT infrastructure. Users can access the enterprise in a number of ways. For example, consumers might interface with the enterprise via POS terminals, self-serve kiosks, Web browsing, customer service representatives or call center personnel. Internal users and suppliers depend on client-server tools and Web-based tools or paging devices. Trading partners typically use specialized application models based on EDI, but they are moving slowly toward Web-based models such as ebXML for B2B interaction. Today, frontline users are being equipped with portals providing enterprise services at their fingertips.

Legacy Environment

Most organizations will keep legacy applications and usage models, which can easily and freely co-exist with the SOA delivery of new applications. On the transactional side, this represents everything from "green screen" to fat client applications. On the decision-making side, it represents the traditional tool-oriented access for executing queries against a data warehouse, as well as client-server analytic applications.

Business Process Automation

The business process automation element represents the next frontier: business process automation and business driven by exception. This element represents those services that initiate, detect, filter and propagate exceptions and events.

Some "event detection" processes are time-based and initiated on a schedule, such as an alert on the end-of-day balance of accounts, while other event-detection processes are driven by newly supplied transactional information, such as a product being sold. Still other event-detection processes result from cascaded events—for example, a late plane arrival initiates events on passengers, baggage, reservations, agents and others.

Raw events need to be filtered by "business rules" to identify those that require action. For example, people only want an alert on their end-of-day balance if it exceeds a change threshold. Or a volume of product sold only requires action if it is above or below the forecast for the week, day or hour. The filtering is done using specified or calculated criteria, including forecasts, plans and sophisticated behavior models developed from the detailed analysis of the historical record of the business.

Once events are detected and filtered, they must be propagated via "event notification" services. The message bus and service broker infrastructure provide the means to deliver the events anywhere they're needed. Humans want them delivered via e-mail, pager, voice message or the latest electronic leash gadget. Some will be routed back to the analytic and decision making repository, either to store for future retrieval or to begin a chain of events.

For example, a late plane "event" launches analysis of the passengers onboard to determine missed connections, which in turn launches analysis of other flights for seat availability, etc. When an event requires automated action, it is propagated back to transactional systems—which, in our example, would result in a new reservation on the appropriate flights for those passengers with missed connections.

(For more details about exception-driven business, see "Taking exception to reports" by Todd Walter.)

EDW-A and EDW-B (Dual Active)

Analytic and decision making repositories traditionally have not been business-critical components of the IT infrastructure. As they are integrated into the real-time enterprise, these repositories support business-critical operational processes and must deliver 24/7 service levels under any conditions, including disasters. Strategies similar to those used for transactional repositories are now appropriate to apply to the decision making repositories. Two (or more) systems, geographically separated, provide protection from disaster. When an EDW approach is used, the applications have a wide range of business criticality. This allows the second system to be a subset of the first, containing only that data which is truly business-critical. Once this determination is made, data synchronization methods can be determined. The data acquisition and integration elements can be extended to "dual load" the high-volume incoming data streams to both repositories. Direct and moderate volume changes are synchronized between the systems via replication services (RS). Finally, the decision making services must be routed appropriately to a repository that has the required data. The Query Director (QD) component provides that service.

(For more details about decision making repositories becoming business-critical, see "Out of hiding: Active data warehousing comes front and center" by Todd Walter.)

The big picture
A real-time enterprise is a requirement for business survival and is the future of IT for a modern organization. Decision making repositories must move from traditional data warehousing to new levels of freshness and accessibility for both strategic and tactical decisions. All parts of the enterprise IT infrastructure must be closely integrated to provide timely, accurate information to everyone who takes action on behalf of the enterprise. SOA concepts and infrastructure provide the glue to combine transactional services with analytic and decision making services to enable a real-time enterprise. Teradata's Real-Time Enterprise Reference Architecture is a blueprint that illustrates how all of the parts can work together to take a business to the next level of responsiveness, accuracy and efficiency. T

Todd Walter, CTO, Teradata Development Division, oversees R&D efforts for Teradata database software and systems. He also is responsible for the future vision and development of the active data warehouse. You can send your data warehousing questions to the expert at todd.walter@teradata-ncr.com.