Teradata

Picture these scenarios:

• Shipments are returned because an organization has shipped the wrong product as the result of discrepancies in UPC/product information.
• Customers send orders with incorrect information on product specification, resulting in delayed processing and shipment.
• Different divisions purchase the same product from different vendors at different prices – and neither division is aware that this is occurring.
• It takes too long to get a consistent definition of the customer, the locations that are serviced, and the products that are shipped to him.
• It takes significant energy to get a report across to distributors about a single customer because different organizations (both internal and external) use different terminology.
• There are multiple customer databases – one per division or one per functional area (customer, service, marketing) – and, as a result, there is no consistent view of the customer across the organization.
• A customer's address is incorrect, resulting in invoices that are returned without payment.
• Marketing sends mailers on product offerings to existing customers because there is no integration across customer support and marketing.
• Bill-of-material information is not consistent across engineering, manufacturing, marketing, and procurement. As a result, the wrong parts are put on the assemblies or the parts are not procured properly.
• Recent acquisitions result in a very inconsistent view of company assets.
• Multiple addresses for different customers exist, and no one knows which one is right.
• A view on total purchases made by a customer cannot be retrieved because one customer may be designated several ways based on which division is selling to him.

Although these seem to be unrelated problems on the part of an organization or its supply chain, they actually point to the same underlying issue – the lack of clean and controlled master data. Master data in most corporations is inconsistent, incomplete, and uncontrolled, leading to large losses in business efficiency.

This document discusses the role of master data, the challenges facing corporations, and the business impact of these issues. It also provides a historical perspective on how enterprise architectures have gotten to this stage of development, describes prior solution approaches, and introduces an innovative approach to solving the problem of inconsistent and uncontrolled master data.

Master data refers to the data assets within a corporation – information on customers, products, items, vendors, locations, routes, and bills-of-material. These are the components that define the corporation – who it is; where it is located; who it does business with; and what it sells, buys, and makes. It is important to note that all other types of data within the corporation (such as transactional, analytical, and operational data) are derived from or based on this master data. For example, if Customer A buys 400 units of product X on Jan 20, 2006, that transaction is based on the master data of customer and product. Master data is also referred to as foundational data – the data on which all other data is based. It is not difficult to understand the importance of having accurate master data. 

Master data management refers to the systems, technologies, and methods by which master data is managed within an enterprise. A comprehensive approach to master data management must include the following five critical areas.

This refers to the business processes and the associated workflows around how master data is managed. For example, the process of managing customer data might involve a customer service representative, an individual from finance who handles credit checks, and a salesperson. The business process, the user interface actions, and the necessary steps required to maintain that entity are referred to as the data management business process. Each user has a specific view of this data that helps him or her complete tasks. Enterprises must clearly define these processes across different entities (customer, item, vendor) and type (direct customer, indirect customer, web customer) to ensure the standardization of these processes across divisions and geographies.

Data quality is integral to master data. Information propagated to all users and systems within and outside the enterprise must be consistent and accurate. Well defined processes need to exist, along with the associated systems and tools for data validation, cleanup, and maintenance of master data.

Meta data is defined as "data about data"; or, the definition of master data and associated systems. Because master data is distributed across enterprise systems, it is important to understand from a system perspective where each element of master data resides, where it is authored, where it is modified, and where it is used. In a data-intensive environment, one simple change in a data definition can adversely affect multiple systems across the enterprise – instantly invalidating data models, breaking reports, and creating global inconsistency.

As master data is distributed across multiple systems, it is necessary to define methods by which this data is replicated and synchronized. Some systems need real-time updates, while others may require only batch refreshes.

Master data must be governed using standard methods and processes. Governance encompasses the people, roles, and organizations that manage data as well as supporting groups.

In the days of the mainframe, all data was integrated on a central computer. There was one set of files, known as master files, that contained information about master data. All programs and applications read from this common set of files. Because the master files were centralized, great control could be exercised over their management. As a result of having a single copy, all applications were consistent.

Over time, corporations migrated to purchasing best-of-breed applications from independent software vendors. In addition, different departments within corporations made autonomous choices on their system strategy. As a result, departments made purchasing and deployment decisions that met their departmental requirements, but not necessarily the goals of the entire organization. Each application required that master data had to be created or loaded so that business transactions pertaining to that department could be conducted.

The advent of marketing systems meant that information on customers and products were defined for that specific use, while research and development systems (such as product life-cycle management systems) also had established definitions of products and bills of materials. The following scenarios further exacerbated these challenges.

1. Multiple divisions and business units having similar systems. This situation could be the result of mergers and acquisitions.
2. Multiple instances of ERP applications existing due to company size.

These problems can be partially overcome by using an integration architecture that employs either flat file transfers or realtime transfers between systems. However, this is only a partial solution because the complexity of these interfaces and the point-to-point connections create a major expense in the IT budget. Also, in this scenario, it is very difficult for employees to get a single view of an entity. For example, to get a complete customer definition with address, locations, payment methods, contacts, and shipping carriers, employees would need to access multiple systems, including ERP, distribution, shipping, and CRM systems.

In fact, departments that need enterprise-wide visibility, such as enterprise reporting/ business intelligence and supply chain departments, often complain about bad data – the most symptomatic issue of broken master data processes. These groups require a full view of data across departments and divisions. The presence of duplicate and incorrect data can skew such analyses and lead to an enterprise making inappropriate decisions. This can lead to a loss of confidence in the underlying systems – thus perpetuating the "spreadsheet behavior" of users trying to create the complete picture outside of corporate systems.

The solution to this problem is not in ripping out the databases from under the applications and making them read from a central source/database. The applications are outside the control of the enterprise – often developed by third-party software providers that need to cater to a large set of customers and often have proprietary rights over their database. The central data model that a particular enterprise uses is unique to that company. It is a representation of the business. 

The principle that appears to have gained traction among widely regarded experts is one of separating the "business process" from the "data." Applying such a concept, the resulting enterprise architecture would appear as follows, with a clean separation of data out of the enterprise applications. 

The data management and reporting layers are horizontal applications that are responsible for feeding data as inputs into the business applications and then collecting the output from them to provide cross-enterprise visibility.

Master data management is a layer that provides this function. It is a software platform whereby corporations can manage a single version of master data and then feed it into business applications that can, in turn, further leverage master data. An approach such as this is clean and consistent. It is a solution for the master data challenges that are faced by large corporations. All applications can connect into this data hub rather than to each other. This simplifies integration, as the MDM layer operates as a single canonical data model across the enterprise. Users enter this data layer to add, modify, and delete information on all master data entities – including customer and vendor.

The life cycle of each of the entities (such as item, vendor, and customer) can be managed through a series of workflows that ensure that the right data is entered by the right people in the right sequence.

Setting aside the elegance and intuitiveness of this approach, two questions must be asked. Why it is important for enterprises to embark on such an initiative? What are the business drivers for and value in adopting this type of approach?

Better management of item, product, and customer data can result in the following benefits to the various company functions.

An approach to reference and master data management should incorporate the ability to manage reference data, which is typically available across multiple sources within an enterprise. Having consistency across the enterprise reduces significant complexity and rework around the use of reference data in applications. Reference data, after creation, is used across disparate business processes throughout the enterprise. Each user has a specific view of this data that helps him or her complete tasks. Because data inconsistencies and errors can be extremely costly to the enterprise, the creation, maintenance, and editing of reference data must be managed carefully.

Although the value of reference data is acknowledged widely, this area receives less attention than it should. The growth of a business, changes in the business, and acquisitions are a few reasons that result in coexistence of a wide variety of end-to-end reference data management processes and systems within an enterprise. It is important to develop a strategy to minimize data errors, duplication, and inconsistencies and make the overall process simpler to deploy. In addition, the strategy should aim to reduce duplication in the process wherever possible.

As a result, companies are not able to control their IT infrastructure to meet the needs of the corporation. Changes to the company (as defined by master data) take time to percolate through the various systems.

Now let us dive a little deeper on the requirements for this horizontal data layer.

1. Need for an enterprise data model – representation of the business. Configurable data model to support changing business needs and different industries.
2. Process model for managing different master data elements. Should be able to configure business process and change on demand. Configurable user interface and experience to suit the needs of the business.
3. Need to manage data quality, data validation, and cleansing – Master Data Management allows an organization to define attribute-, record-, and table-level business rules to ensure data conforms to requirements and meets data integrity rules. Validation rules can be defined in a 4GL environment using very simple rule based syntax.
4. Need to manage systems and data assets, mappings and translations among them. Meta data management – that can be distributed as well as centralized.
5. Best practice templates for domain areas.
6. Role-based data management – MDM allows a user to define views and business rules for managing data based on predefined roles. This allows different roles in the organization to access different views of the product data. The display names of the attributes can also be changed.
7. Ability to author and maintain the following objects:
• Item
• BOM
• Routings
• Resources
• Distribution lanes
• Customers, channels
• Vendors
• Organization, users, and roles
8. Parameter management on any data object.
9. Multi-language support.
10. Ability to develop business rules to automatically complete or trigger alternative processes.
11. Integration compatibility – MDM should be compatible with multiple integration frameworks –Web services, EAI, ETL, etc. This provides extract, transform, and load functions.
12. Ability to detect changes and audit data – conform to Sarbanes-Oxley requirements.
13. Data export/import capabilities for ad hoc requirements – Excel, CSV files.
14. End-user reporting capabilities – need to provide a seamless interface for business intelligence application.
15. Need vertical and horizontal scalability.
16. Data Synchronization – MDM keeps track of which upstream systems are the sources for the attribute data and is able to recognize and communicate discrepancies among various upstream and downstream systems for the same type of data (i.e., product data out of sync between systems…)
17. External connectivity – let suppliers, customers, and partners interface with the application via system to system or HTML model to be able to deliver and receive information.
18. Hierarchical data management. This is a key concept that needs to be explained.
19. Permissibility – data-level permissibility.