Lose fat, build muscle

Investing wisely in your decision support environment.

by Dan Higgins

Successful companies know that managing cost is as important to long-term corporate health as increasing revenues. In "Best Practices in IT Portfolio Management," a paper published in the spring 2004 issue of MIT Sloan Management Review, Mark Jeffrey and Ingmar Leliveld state the following:

"An estimated 68% of corporate IT projects are neither on time nor on budget, and they don't deliver the originally stated business goals. Some even claim that during the last two years, $100 billion to $500 billion of U.S. IT projects have failed altogether."

On the other hand, some would argue that investments in information technology over the past four decades have had a huge impact on productivity and have radically changed the way corporations operate for the better. So the challenge is not to curtail investment but to make wise investments, which can unlock considerable business potential.

According to Kevin Strange, vice president and analyst at Gartner, "The cost management of a data warehouse at the enterprise level is more important today than ever before, and that it goes beyond the common TCO exercise of making IT infrastructures more efficient." He adds, "Enterprises execute a data mart consolidation strategy, which can reduce costs by 50% while increasing the value proposition of BI applications by 500% as a result of the increased flexibility, durability and business value of a DW infrastructure."

Investing wisely in a decision support environment means increasing those investments that lead directly to the greatest business value in both the short and long term, while minimizing those costs that inhibit business value over the long term. It is somewhat like caring for our own health. In order to be lean, strong and flexible it is not enough to simply reduce calories. Aggressive reduction in calories through fad diets or other temporary approaches may lead to an unhealthy body. The body needs the right kinds of calories and nutrients in the right amounts and at the right time. And along with the right calories we must also build strength and resiliency through proper and disciplined exercise. Let's look at ways we can lose fat and increase muscle in a decision support environment.

Losing data warehouse fat
Data warehouse "fat" is anything that consumes system and staff resources while providing little or diminishing value. It can also be anything that may provide value in a manner that increases costs at a steeper rate than it increases benefits. Let's look at a few examples and how to address them.

• Reduce technology and architecture complexity. A common cause of "fat" in the data warehouse is overly complex technology or an overly complex technical architecture. A symptom of this condition is too many valuable IT resources expended on solving or working around technology complexities or issues rather than providing business solutions. Of course, at the lowest level all technology can be complex. But technology platforms that eliminate or hide complexity through self-management and virtualization free IT resources to focus on delivering business solutions (queries, reports and applications.) (See "The value hierarchy for data warehouse support activities" below.) A specific example would be when IT must spend considerable time monitoring and managing low-level physical resources such as disks, files, tablespaces, bufferpools, memory, server nodes, etc.
  
  An example of architectural complexity might be a "hub and spoke" architecture, with many data marts requiring additional extract, transform and load (ETL) processes. In this case the "fat" is a consequence of data redundancy and the staffing required to wrestle with data synchronization, ETL processes and the additional server platforms. This is not to say that complexity is not necessary in data warehousing. The needs of business users are increasingly complex and constantly changing. As the data warehouse evolves it is usually the case that the data model and the queries will become more complex. But in this case the complexity is directly associated with delivering greater business value. It becomes an enabler whereas too much complexity in the underlying technology infrastructure can become an inhibitor.
• Consolidate or eliminate data marts. Physical data marts significantly increase system costs (hardware, often redundant data, network bandwidth, software licenses, etc.) and staff costs (more complex extract, transform and load processes; more system and database administration.) Because data marts tend to serve the needs of a specific application or business function, the benefits are not leveraged across the enterprise as would be the case if the investment were made in an integrated, cross-functional data warehouse. Furthermore, there is a "half-life" for data marts. Although they may provide value when first implemented, once the predefined reports and queries have been in use for a period of time the narrow scope of the environment—their limited context—prevents data marts from providing new business insight. Yet they continue to consume system and staff resources.
• Consider multi-temperature data management. Even with the dramatic drops in cost per gigabyte over the past decade, data warehouse volumes have been increasing at an even greater rate. Some would suggest removing less valuable data from the data warehouse. Yet removing the data from the system can prevent a timely answer to an occasional but very important business question. One technique for managing this is to use higher-density disk coupled with database Views and partitioning techniques, in order to segregate lower value data until it is needed.
• Use appropriate query tuning. It is critical that a data warehouse provide query freedom and support for a steady stream of new, previously undefined queries. It is equally important that frequent, known queries be carefully reviewed to ensure they execute efficiently. Be sure they are exploiting any functions and features which further improve query execution, reducing their use of system resources.
• Manage system resources. Not all workloads are equal in terms of their benefit to the enterprise. But this should not be an all-or-nothing decision. Through effective use of workload and resource management tools, higher-value and urgent workloads can be assured the needed resources at the time they are required. And other workloads which are important but less critical can share the same data resources.

Building data warehouse muscle
Data warehouse muscle is anything that delivers business intelligence that is broader, deeper, timelier, more accurate, more insightful and more relevant. Here are some of the steps we can take to build data warehouse muscle.

• Build on a normalized data model foundation. Companies that exploit an application-neutral normalized data model for the foundation of their data warehouses can realize tremendous value as the data warehouse evolves. They are able to accommodate the widest possible range of business questions. And they realize considerable time and cost savings in the development and delivery of new applications because the data sourcing and preparation process has already been completed to support earlier application development.
• Add subject areas. Often the greatest insight from a decision support system is found in the relationships between subject areas. Adding subject areas into a single integrated model substantially increases the range and value of business questions that can be addressed.
• Retain detailed, historical data. Availability of detailed data is critical to understanding complex relationships, to drill down to root causes, to perform predictive analysis and to support active or tactical decision support. Limiting the data warehouse to summary data severely cripples its analytical potential.
• Accommodate query freedom. Much of the power of a decision support environment is the ability to address an increasingly wide range of business questions in a timely manner. Technology or processes that inhibit query freedom limit the value of the data warehouse.

Periodic checkups
As with any ongoing program, it is important to have periodic assessments. Have needs changed? Are reports or data marts (virtual or physical) still delivering new value? Are there frequent, known queries or workloads that can be tuned to relieve system capacity? It is important to maintain consistent metrics of staff and system utilization. Schedule regular checkups to audit the data warehouse staff activities, processes, the system resource utilization and the various workloads and applications.

Accountability
We all know that to succeed in building a lean, strong, flexible body, it ultimately comes down to us. We may read a book on dieting or engage the services of a personal trainer. But we are the ones who must make it happen. This is also true in organizations that desire an effective, healthy, successful data warehouse. But who in the organization is accountable for making the proper investment decisions? Should it be the procurement team, IT or business executives? Each of these has an important perspective but may not appreciate the full implications of a decision.

Ultimately, teamwork will yield a better, more holistic investment decision for the business. The best approach is to establish an executive steering committee for the data warehouse consisting of leadership from the business units, IT and possibly procurement or the CFO's office. This group should be accountable to the CEO.

All major decisions should be brought before this group including business requirements, plans for project phases, resource requirements and decisions regarding architecture, technology and vendors.

Off to a running start
And finally, it does not happen overnight. Success requires discipline, effort and persistence. No excuses. The same is true with a healthy, high-value data warehouse. So let's get started! T

The value hierarchy for data warehouse support activities

STAFF COSTS (system administration, database administration, ETL work, application development, end users, etc.) can easily surpass technology costs, especially over multiple years. It is critical that staff resources be invested in activities that most directly deliver value to the business. Consider figure 1. It represents a hierarchy of activities or effort associated with a data warehouse. Business applications. At this level we are combining various queries with other programming logic to deliver ongoing business solutions. Queries. These are the activities associated with getting answers to business questions. Data. This includes the data modeling/design effort, the sourcing and loading of data and the ongoing maintenance of data. System/platform. This includes all of the activities associated with the setup and care and feeding of the hardware, OS, file system, DBMS and other platform components. Consider this. When system/platform- and data-related activities are "complete," how much value has been delivered to the enterprise? Zero. None. The platform and data layers are necessary infrastructure requirements but not sufficient to bring business value in and of themselves. Business value is the result of answering business questions. The real value of the data warehouse is directly related to its timeliness and efficiency in answering a large volume of varied business questions. We must choose technologies and architectures which free our staff resources to focus on the higher level activities which directly impact business value. Figure 1 (click to enlarge) When counting staff costs, consider the following: Average fully burdened, annual cost for a single full-time employee $150,000-$200,000 For a data warehousing team of 10 $1.5M-$2M For a team of 10 over a five-year period $7.5M-$10M For a six-month project slip with a team of 10 $750,000-$1M There are cases where the data warehousing and business intelligence team has exceeded 100 members. All the more reason to ensure they are spending their time on high-value activities and are not encumbered by inadequate or overly complex technologies or architectures.

Counting calories

A SIGN OUTSIDE ALBERT EINSTEIN'S OFFICE at Princeton said, "Not everything that counts can be counted, and not everything that can be counted counts." Clearly CIOs are under tremendous pressure to reduce or contain costs while delivering more. This pressure often leads to a focus on total cost of ownership (TCO). TCO, by definition, should consider the costs associated with successful exploitation of the technology, where all of the benefits come from using it rather than owning it. However, a broader consideration of costs includes the impact on staffing costs, project timelines and opportunity costs—downstream costs that are not always accounted for in TCO efforts. Call these costs the total cost of success (TCS). And when initial attempts at data warehousing fail due to technology or architecture decisions, call it the total cost of failure (TCF). It may seem harsh, but there are many examples of companies that spent millions of dollars on efforts to make an inappropriate technology work, only to fail to meet the goal. Amazingly, it is never the technology that was at fault; instead the processes and the goals themselves are at fault.

Dan Higgins is the director of Teradata Warehouse Sales Support and the coauthor of Evolving Through Action. He leads a team of subject-matter experts who assist Teradata's worldwide sales force.