Teradata

With overwhelming amounts of data pouring into existing disparate systems, the change from symmetric to asymmetric warfare and the need to support real-time actionable intelligence in theater, the Intelligence Community (IC) has a need for new technologies that can enable solutions to these challenges like never before. The mandate to the Office for the Director of National Intelligence (ODNI) to transform analysis from production of product into a service of actionable intelligence, as well as to implement information sharing to provide more complete and rapid response to the warfighters, can be enabled through successful commercial technologies.

This white paper identifies the intelligence community's overarching information management challenges and describes representative solutions achieved by commercial powerhouses. These solutions are achieved using an active data warehouse from Teradata that allows disparate data to be consolidated.

In the past, limitations of collection and focus of targets enabled stove piped environments to be implemented for each new area of interest; there was no need to seek complex relationships in disparate data because these targets were relatively static and known. Finding hidden social networks that are geographically dispersed and dynamic in nature requires that information be shared and leveraged across multi-INTS, as well as fused to form intelligence that is actionable.

The current environment has data embedded in legacy applications in disparate systems with no easy way to access, share, or understand what has been collected. This was recognized in the 9/11 Commission Report, as well as the WMD Commission and has been mandated to change through Presidential Executive Orders, the National Intelligence Strategy, and Congressional Directives focused on shifting this current manner of business in the IC. Cultural, policy, and legal issues all conspire to create inertia that must be overcome to face the current threat of terrorism.

During the past decade, with shrinking R&D budgets in the IC, commercial markets have led the way in the integration and utilization of new technologies to improve their competitive posture. Security, continuity of operations, and an overabundance of detailed data have created environments similar to the IC, which allow success in these markets to be leveraged. Technologies enabling information sharing and knowledge management are crucial to proactively addressing today's terrorist threat.

Fortune 500 companies such as a major, U.S.-based telecommunications provider and a major, U.S.-based retailer, have remade their businesses through usage of technologies that enable companies, even in commodity markets, not only to recognize historical trends and understand their current business posture, but also to actively predict their customers' behavior. This major retailer's initial data warehouse started in a single business area that grew into a corporate-wide asset of more than one petabyte of data in a single instance of a Teradata® system. This enables insights that were previously unattainable.

Customer relationship management allows successful commercial companies to focus their business resources on their best customers. With limited resources in the IC, it's critical that terrorists of the highest interest be quickly identified and understood in order to circumvent or eliminate their behavior. Warfighters can also be viewed as customers because their diverse and real-time interests are satisfied through the service of providing intelligence that is relevant, available, and real time.

Current data in the IC are replete with inconsistencies, redundancies, and tied to legacy environments. There is not a single, authoritative source of all of the information available. Attempts to provide federated searches across the enterprise have failed largely due to the current method of organizing, storing, accessing, and processing data. The tremendous interest across the IC in enterprise and service oriented architectures validates the need to shift into the DNI CIO's motto of "think big, start small, and scale fast".

Hundreds and hundreds of databases exist within each agency of the IC, all on classified networks. Each application has its own database, and access to specific intelligence by analysts may go away with the change of a position where, in their new environment, they no longer have access to the previous data. Data are stored depending on the application format for the database supported. It is impossible to discover what information is even available.

Multiple competing and conflicting sources add to the confusion of the lack of an authoritative source. Master data can be found in multiple formats and repositories causing invalid or incomplete conclusions to be made. Efforts to search data found across the enterprise are constricted by the dependence of data programmatically linked in applications. Data models and formats are different, therefore, data integration cannot be achieved. Stove pipes of data developed over decades were built of tightly-coupled data structures that often contained the same information. To add to the complexity, objects of varying file types are stored within the enterprise, again without formats of consistent identifiers.

Duplication of data through copying into one's source compounds the issues when modifications to the original data are made and not controlled through dissemination. Even if data are copied from one database to another, the format of the new database may not even accommodate the old data structure. Lack of consistency creates an inability to validate redundancy of data of varying certainty. Complex queries that would ask higher value questions cannot be made because the data are spread across the enterprise in many different formats.

Analysts spend significant amounts of time seeking, as opposed to analyzing, data. Data found through search are not consolidated in ways that facilitate these analyses. This creates an incoherent set of data, full of inconsistencies and redundancies.

Some successful commercial companies have progressed to enable integrated data warehouses that provide data discovery, sharing, and access to enable immediate business results that can grow without reengineering as additional data and dynamic analytic requirements are necessitated. This promotes information sharing, which deepens the value of the data because they are useful, timely, and accessible.

Businesses now face global competition, rapid deployment of new products, new and complex financial reporting, and new, low cost competitors. Constant change is the norm. Huge volumes of internal and external data are collected and must be assimilated and turned into information, intelligence, and then, knowledge. To address these new challenges in the business environment, data warehouses have been adapted to solve these issues. At a major U.S.-based retailer, the usage of their one petabyte of data has transformed a commodity market challenge into a strategic competitive asset.

Their active data warehouse contains current detailed transaction data, as well as many years of historical data. This provides a source of data aggregated and organized for immediate analysis within a single environment. Discovery of new information is enabled as queries against consistent and reliable data are made. Historical queries are relevant and become part of the ability to identify trends across the entire corpus of information. New data sources can be easily incorporated into the existing data model and queried in a cohesive manner.

An active data warehouse understands the business requirements, and it provides an environment to enable questions to be answered by discovering the location of the data and transforms the data for use in this environment. The Teradata methodology incorporates a tightly integrated hardware platform with a database that is rapidly scalable, highly available and enables ad hoc, complex queries to be answered across huge volumes of data.

A Teradata solution provides a process that can start with just a few data sources where the aggregation provides the ability to answer questions that were unanswerable and scale to include many data sources and petabytes of data. The Teradata solution can support the smallest to the largest active data warehouse.

Many companies started with a vision of enterprise corporate data that could be shared and enabled through an active data warehouse. They began the process in a small, focused effort to prove value and then continued to scale to the size required by their growing businesses.

In this section, we will highlight commercial business implementations of active data warehouses from Teradata. These businesses have faced many of the same challenges as the intelligence community.

Using an active data warehouse from Teradata, the intelligence community can achieve the same success as these companies who are known for their information and decision management expertise.

The common thread in these companies is the selection of an active data warehouse from Teradata. Each company described is in a different market, has a different type of internal organization, and is a sophisticated leader in information management technology. Their shared information management concept is that when the data, both current and historical, are organized in an active data warehouse, they provide the most effective way of discovering and exploiting the relevant relationships to achieve timely decisions.

First, a major U.S.-based Telco and our major U.S.-based retailer illustrate very large and continuously growing implementations. Second, the success at a major, U.S.-based aerospace company is enabled by the aggregation of many legacy systems and the shared access over their intranet and the Internet. Third, a major, U.S.- based manufacturer with many products, separate business units, and customers, used Teradata as the enabling technology to grow and manage their business.

The telco and the retailer examples illustrate the evolution of capabilities and accomplishments in an enterprise. Today, each company operates a nearly petabytesized (1,024 terabytes) active data warehouse based on Teradata systems.

A critically important concept illustrated by the telco and the retailer is that active data warehousing with Teradata supports unbounded data growth. Information requirements are ever changing and often unknown and unpredictable. An active data warehouse supports constantly evolving business environments by providing a platform that scales with the data growth, enables a consistent data model for complex analysis, and ensures reliability of answers to questions that constantly change.

The aerospace company planned to implement a single Teradata Database to collect all the in-service airplane fleet data. There were more than 30 disparate databases that were not communicating with each other. Information sharing and collaboration by multiple communities of interest were facilitated by an active data warehouse from Teradata. The company, airline customers, and suppliers contribute data to the single active data warehouse with each receiving direct analytic benefits to support their own decision making.

Another major U.S.-based manufacturer chose to implement Teradata because queries took days, and answers were not always accurate. Now, with integrated data stored in a single location and a single view of their data, they have achieved rapid response times and accurate data for analyses and decision making.

One big issue at this manufacturer was business unit cooperation. For years, the company had operated as 40 separate companies all under one roof, all with unique business practices and policies. When the CEO sought information from within, it would take weeks, maybe months, to collect the data. To address these challenges, the company implemented a Teradata system as their Global Enterprise Data Warehouse containing facts, products, customers, geography, and time data. The goal was profitable growth, enabled by timely decisions based on results. The data are now harnessed to enable timely decisions to address events that are often unknown and unanticipated.

Successful businesses recognize that strategic advantage is gained with the effective integration of information technology. There is a need for consistent, complete, and trusted information sharing and collaboration. Many approaches have been tried including federated databases, multiple data marts, and specialized statistical data files. These approaches have data replication, data inconsistency, and timing issues, as well as information and physical architectural integration complexities.

An active data warehouse from Teradata eliminates these issues and complexities by providing a single architecture solution that has been designed, developed, and improved over the past twenty-seven plus years. And it's a solution that has been designed from inception to scale in a nearly linear fashion. This solution consists of a tested and proven data warehouse methodology and process, Teradata platform, Teradata Database software, and analytic applications and tools. A key differentiator of the Teradata solution is the parallel processing architecture that provides easy scalability as data and analytics are added. The easy scalability allows for data and application growth over many years and platform advancement. There is no need to redesign the analytic applications or re-architect the database as a certain data size or processing complexity is reached.

Teradata has led the data warehousing market for more than 27 years, currently serving more than 900 customers and more than 2,000 implementations. Sixty percent of the most admired companies use Teradata including:

• 90% of the top 10 global telecommunications companies
• 50% of the top 10 global retailers
• 70% of the top 10 global airlines
• 60% of the top 10 global transportation logistics companies
• 50% of the top 10 global commercial and savings banks

There is no other provider of technology of this caliber who can provide commercial references of this variety and size.

The companies mentioned in this paper began by identifying business requirements that would significantly improve their competitive position or customer relationships. Teradata, as the primary provider of mission critical platforms of this size, became involved to enable best commercial practices and provide expertise on the implementation of active data warehouses. Teradata and the commercial entity next identified specific data sources that would enable answering questions that could not previously be answered to provide focus and a rapid time to solution. In rapid increments, the strategic goal came closer by adding data sources and additional decision functionality in a continuous process. Each additional data and analytic increment enhanced decision-making capabilities with insights that were never before possible.

The extensible Teradata architecture was architected to directly enable the "think big, start small, scale fast" enterprise vision. It supports the elimination of unnecessary databases, as well as decreases the need for replicating data across an enterprise. Legacy applications can ingest data into a format that allows analysts to query across data sets previously unattainable. Data and applications are decoupled as data are populated into the active data warehouse. The mission requirement for information sharing is satisfied by a single query to search all relevant integrated content of interest.

Standing queries can be generated to automate processes for analysts to save time and focus on more challenging areas of interest with alerting functionality providing the real-time prompts. Quick responses to complex queries facilitate the rapid continuing focus to drive to conclusions. Data quality problems are eliminated as tools to extract, transform, and load data provide consistency and reliability.

Risk is mitigated from the growth standpoint because Teradata provides an integrated solution that cannot be matched. For mission critical problems, knowing that one call will result in a solution resolves the anguish of having multiple vendors involved.

No longer is word of mouth required to find the existence of data needed to understand and solve an analytic problem. A solution from Teradata eliminates the time critical challenge of gaining access to data once they are found by providing them in a cohesive, available, scalable way. More reliable and complete intelligence can be produced quicker and easier with greater sharing of information within the enterprise and across agencies. Information technology bottlenecks no longer need to throttle the evolution of ideas to answers.

As the Founder and CEO of Mosaic, Inc., Ms. Whittaker has developed a consulting and solution implementation company that can identify and implement competitive solutions required by the government, as well as commercial customers.

Ms. Whittaker has been a strategic consultant to some of the most prominent organizations in the war on terror. With a focus on the U.S. Government during the past 25 years, particularly on intelligence community enterprise requirements, she has consulted with executives to bring new technologies into the mainstream of the intelligence community.

For more information about how to implement a Teradata solution in the intelligence community, contact Robert Withers at (703) 443-6903, or http://www.teradata.com/governmentindustry-solutions.