Getting to service-oriented architecture

A closer look at your path to service-based data warehousing.

by Daniel Linstedt

Service-oriented architecture (SOA) is the phrase du jour. Just about every data warehousing article, white paper and research report uses the term at least once, but few do a very good job of defining it. And unless you have studied the topic, you might not have a real understanding of what SOA means.

SOA introduces to the data warehouse both business and technical challenges that require the implementation of risk mitigation strategies for security and compliance initiatives. It also requires a strategy for migrating to an SOA environment.

Because of these critical issues, we all must have a clear understanding of SOA. This article is a chance for us to get back to the basics. What is an SOA? Why is it important? And where do we go from here?

What is SOA?
An SOA is a conceptual architecture used to design applications and business processes at a component level, meaning every computing unit of work is defined as a service. For example, segmenting customers, ordering products or determining inventory levels are components of an overall business process and can be considered services

At the highest level, SOAs have three roles (see figure 1). Let's say those roles are played by people. The person who wants a service is called a service requester. The person who provides a service is a service provider. The person who connects the service requestor with the service provider is a service broker.

Those three players have three operations to perform. When a service requester wants a service, he must first find someone who can provide that service. This is accomplished by asking the service broker. When the service broker finds a service provider, the service requester can then interact directly with the service provider.

By the same token, a service provider must publish the existence of himself and the service to the service broker so that she can pass that information on to the service requester. Find, interact and publish are the three key operations of the three key roles.

SOA includes executable components-such as Web services-that can be invoked by other programs that act as clients or consumers of those services. A developer does not need to know how the programs work; he only needs to know what input that they require, what output they provide and how to invoke them for execution. Each of the components lends value to the overall SOA and, if architected into the enterprise solution properly, can provide tremendous corporate value. Lest we forget, each component has its own intrinsic value to the corporation as well. There are basically four types of components comprising an SOA: tools, applications, business components and data components. Each of those components will consist of elements that are unique to the infrastructure of the organization.

Figure 2 is a representative list of each of the four components.

Services can be invoked based on decisions made by business rules. Developers can swap out one service and replace it with another service that is designed to achieve the same or similar result, just like the standard parts in a car can be interchanged without having to strip down the whole car and rebuild it.

SOA has the opportunity to deliver business value on many fronts. By "wrapping" existing applications as services, their functionality can be reused, extending application lifetime and reducing the cost of developing new functionality. By leveraging existing services, new applications can be deployed-and bring value-much more quickly.

The business of data warehousing has been going on for years. By now, nearly every competitive business has some form of decision support repository. Those organizations that have grown their data warehouses into active data warehouses containing historical, current and, in some cases, future views of their enterprise information are beginning to share the value of business intelligence (BI) with the organization.

As Lou Agosta, leading industry analyst at Forrester Research, Inc. noted in his November 2004 DM Review article entitled "Data Warehousing Lessons Learned: A Time of Growth for Data Warehousing": "(In 2005), firms ... will invest in third-generation, active data warehousing to attain integrated BI. ... Rolling data warehousing into your overall SOA is critical for enabling integrated BI. Data warehousing ... (requires) service interfaces to expose data access, data transformation, aggregation, reporting and a host of related decision support services."

"Web services are unequivocally gaining traction within businesses," reports Whit Andrews, an analyst with market research firm Gartner. He was quoted in the April 2005 issue of MIT's Technology Review as saying, "Analysts estimate that from 50% to 85% of large companies now use Web services. According to Gartner, 63% of enterprises deploying Web services are using them for internal integration projects, while the rest are connecting their internal systems to those of partners and customers."

The packaging of "reusable" components that use technologies for the delivery and exchange of information provides a mechanism to expose the analytic value of information stored in the data warehouse. Data-oriented services encapsulate both the data and the technologies that move data in and out of the systems and the applications that access and interact with the data warehouse based on rules and events. This helps orchestrate the business requirements of the enterprise.

SOA drives the need to change the way IT builds and architects data warehouses. SOA and data warehousing provide strategic and highly integrated data at the right time to support services. Data warehouse developers must better understand business processes (used by SOA), and data warehousing integrators must work together to answer the real business questions. In fact, putting a service-oriented architecture on top of an enterprise data warehouse increases the value of the integrated data.

An SOA isn't simply Web services layered on top of an existing hodge-podge of databases and data integration tool sets. Nor is value added to the organization if SOA is simply "slapped on top" of a data warehouse and called "good." As Michael Curry noted in an August 2004 blog entry on ascential.com, "You can easily over-engineer services into full-blown applications, or just call any old application a 'service' by slapping a Web services API onto it."

What does SOA do to data warehouse architecture?
SOA brings a wide range of new data warehousing standards and requirements to the table, such as:

• Can your organization agree on the services that you need-for customers, for partners, for your internal organizations?
• Can you agree on what parts of those services require access to data that is or should be stored in your enterprise data warehouse?
• As for active data warehouses, can you agree on what key business events should cause the enterprise data warehouse to kick off an automated business workflow?

SOA and the resulting business changes will force IT to think about a new concept: truly dynamic data warehousing that allows the data warehouse to evolve to match the services needs according to the business process changes occurring within the organization.

Of course, human involvement is still required. For instance, the organization will need to figure out the business processes, write them down and get agreement from all affected departments. Then it must begin the process of automating. Are there some business processes that users could access over the Web? Are there others that can be completely automated for normal cases yet still alert someone when exceptions occur?

For example, a CRM campaign manager can use the data warehouse for segmentation and preliminary customer contact information. Then analysts can turn to an external Web service to double-check that the customer has not been added to the Do Not Call list. After updating the data warehouse, the data can be handed off to a call center.

SOA broadens the range of impact that the data warehouse has on the enterprise. What will happen is that many more applications will be written because they are easier to write. More manual operations will turn into workflows, and those workflows will drive applications and components that touch the data warehouse.

The business should decide what the value of the SOA is and whether it will be funding such an effort. This is not to say that the SOA effort should be funded in a big-bang mode. The healthy business that chooses to support SOA on top of a data warehouse will identify specific pain points and develop specific questions that can only be answered through a service model.

Building a single centralized enterprise data warehouse with highly integrated answer sets allows all the data integration routines that already exist to be utilized as possible services. The services only have to retrieve the data sets from the enterprise data warehouse, instead of forcing re-writes or re-invention of the conversion and integration routines during operation.

What are some SOA pros and cons?
It's clear that SOA offers many benefits. It:

• Speeds application development and reduces integration costs by using standards-based tools
• Derives more value from the data warehouse by making it easier and faster to write decision support applications
• Broadens decision making throughout the value chain (B2C and B2B directions) with automated workflows
• Raises the semantics of the data warehouse by making reusable analytical services more widely available
• Uses more of the information within the enterprise
• Offers the ability to ask new questions
• Allows for hiding or seamlessly integrating backend information stores

As one of the leading vendors in the active data warehousing world, Teradata brings additional benefits to the table through enablement tools such as the data warehouse, active feed mechanisms, in-database data mining algorithms and Teradata Application Platform (TAP).

SOA is implemented incrementally, starting with broad applications like dashboards or specific priorities like price optimization and building a proof of concept. As with all IT implementations, issues will arise. Remember that:

• Web services technology is young and needs to mature. There is currently a lack of standardization across corporations that inhibits the heavy adoption of Web services. The Web service protocol stack is a gradually developing collection of protocols used to define, locate, implement and make Web services interact with each other. These protocols include a wide range of recently established protocols that are working toward standardization.
• SOA is highly metadata dependent. We are not talking about just the technical metadata; we are talking about the business metadata that defines the meaning of the information itself and, now, the meaning of the service. That's why universal description, discovery and integration (UDDI) is important and why application metadata needs to be given equal importance to "data metadata." (UDDI is a platform-independent, XML-based registry that businesses worldwide can use to list themselves on the Internet.)
• Active data warehousing must be seen as operationally relevant and be utilized as a tactical information store. Teradata fills this need by providing the platform and processing capabilities to handle the new requirements. This doesn't mean pushing all data from OLTP and data marts into an active data warehouse; it merely means focusing on the right data at the right time for your specific business purposes.
• Data warehouse architects will need to make decisions about data currency. Such decisions could impact the currency of Web service results, so there may need to be a way to gauge the accuracy of results or to force the Web service to use only the "latest" data.

Conclusion
SOA is here, now. The time has come to implement it within every enterprise. SOA can be used for competitive advantage and enterprise value by integrating systems and technologies seamlessly behind the scenes.

Building your initial SOA implementations can be a struggle; integrating roles, defining responsibilities and formulating standards are not easy tasks. However, building Web services components that use technologies for the delivery and exchange of information provides a mechanism to expose the true business value of the information stored in the data warehouse. T

Making the move to service-based data warehousing

THERE ARE SPECIFIC SKILL SETS needed to migrate from a traditional data warehouse to a service-based data warehouse. In fact, the move requires that the data warehouse team take on a few more roles, which might include the following: Metadata steward-This role defines how application and component metadata will be collected, managed and kept up to date. It also requires interfacing with business users in order to integrate business metadata (UDDI, XML) that provides value as a service. Web services implementation expert-This role uses the specific Web services technology that has been chosen by the enterprise. Implementing Web services comes with a different set of requirements than a standard data warehouse. It is important to include this role on the project team so that the application writers are not dual-tasked. This role interfaces with the other three roles described here. Compliance expert-This role covers business compliance initiatives at a technology level. It is responsible for defining what data must be kept and tracked, and how that information meets the compliance initiatives of the corporation. This role is also responsible for interfacing with the business users in order to define compliance requirements. It is not this person's role to dictate how the compliance initiatives will be implemented. However, it is critical to ensure the proper information is tracked at the appropriate locations throughout the SOA. Security expert-This role overlaps the metadata steward to ensure that the secured data has the appropriate access constraints according to user login. This expert also defines who can access what types (classes) of information, identifies what sources should be available in which classifications of service and defines classifications for the services made available by the Web services implementation expert.

How do I begin?

DON'T FALL INTO THE TRAP of thinking that SOA projects are one size fits all. Instead, re-scope the project and implement a component of the service with a small release of data. Monitor the progress and implement the proper compliance solutions before scaling the system. The following are some high-level steps to follow: Establish a small scope of services that will answer two or three key business performance indicators. Start with internal-only applications or a handful of customer-facing or partner-facing applications and build from there. Begin thinking about the business rules (or business rules engine) that may be involved. Design the internal- and external-facing services and name them. Also design workflows, applications and reusable components. Identify and design data-oriented services that encapsulate both the data and the technologies that move data access to support the EII, EAI and ETL technology that will be utilized. Build the metadata layer and the cross-sourcing architecture for where the data will be pulled from or written back to. Identify master systems for data sets that will feed the Web service and examine their backup, restore and system of record (SOR) capabilities. Approximate the frequency of access, which will help define the frequency of updates. Begin integrating the data sets for testing purposes; overload the Web service to see where the scalability of the current solution stops. Check the data set to see if it appears seamlessly integrated from cross-system queries. Ensure that all access is tracked. Develop a maintenance plan for keeping the metadata and business rules up to date. Demonstrate the ROI case to the stakeholders; ensure that the KPIs your team set out to answer are in fact being answered.