News Stay informed about the latest enterprise technology news and product updates.

Building better business intelligence systems

Shaku Atre analyzes the non-technical infrastructure required for operational business intelligence applications and explains why creating this infrastructure requires cross-organizational activities.

This article originally appeared on the BeyeNETWORK.

Part 1 of this series, discussed the technical infrastructure issues for operational business intelligence applications. This article will concentrate on the non-technical infrastructure.

It is expected that one knows what the new or the revised operational business intelligencesystem is supposed to accomplish. In order to have a better understanding, it is necessary to evaluate what is in existence. This requires a review of the existing physical model (how everything that supplies current business intelligence capabilities is implemented physically) and what was this existing physical model is supposed to accomplish (which is the existing logical model). This exercise will helps to understand what was tried in the past, what has worked and what has not worked.

Need for Cross-Organizational Non-Technical Infrastructure
In most of today’s environments it is very difficult – if not impossible at times – to answer cross-organizational questions because of the fragmented operational systems. Creating an operational business intelligence infrastructure involves cross-organizational activities, such as:

  • Extensive business analysis involving business representatives from many lines of business. During this activity, the lost complex interrelationships among business functions and business data are defined or redefined. Agreement on the business rules and business policies is obtained. (This is one of the most challenging activities.)

  • Resolving age-old disputes about data definitions and domains (valid data contents) by supporting cross-organizational attendance and evaluation of business analysis activities.

  • Standardizing data names and data values to reflect true business rules and business policies.

  • Creating a regular forum for business representatives to maintain and review the standards, business rules and business policies on an ongoing basis.

  • Creating (over time) one consolidated, non-redundant data architecture for the entire enterprise to reflect the complex reality of the business; that is, creating an enterprise logical data model. This model documents the data inventory of an organization. It is also the primary vehicle for mapping the inventory of operational data to the inventory of business intelligence data.

  • Creating an inventory of source data and mapping it to the applicable BI target databases, as well as creating an inventory of other system components, such as programs, reports, screens, etc., thereby identifying reusability of data and process components.

  • Creating and managing one expanding central staging area (per load periodicity) for the extract/transform/load (ETL) processes, rather than allowing independentETL processes for each data mart solution.

In order to support these cross-organizational activities, a logical data model, standards, guidelines and procedures have to be in place and key questions must be answered. These include:

Logical Data Model   

  • Do we already have logical data models for the source systems? If not, who is responsible for creating a logical data model for this BI project?
  • Who are the data owners and business representatives that have to participate in the validation of the logical data model and the business metadata? 
  • How many trained data administrators do we have? Will we have to hire more?
  • Who will integrate our logical data model into the enterprise logical data model?
  • Who will validate the expanded enterprise logical data model?

Standards, Guidelines, and Procedures

  • Are our current standards too lax or too stringent?
  • Where are the standards documented? Are they being followed?
  • How effective are our data quality guidelines for measuring and triaging dirty data?
  • Are our change control procedures easy to use? Do we have a template?
  • Do we have an issue log template?
  • What are our testing standards?
  • Are we habitually testing too much or too little? Are we testing the right things?
  • How are we currently resolving technical and business disputes?
  • Do we need to create or change our dispute resolution procedure?
  • What are the current roles and responsibilities of the core team?
  • Is our current team structure effective?

Enterprise-Wide Non-Technical Infrastructure
The non-technical infrastructure should not be constrained only to a departmental environment; it should address the issues of the organization at large. Without a cross-organizational infrastructure, BI applications would only contribute to the existing chaos of stovepipe applications and databases.

Stovepipe Development
From the early ages of IT, the mental model for providing an automated solution to a business problem has been to “divide and conquer.”

  1. Divide a large problem into smaller “digestible” pieces; that is, prioritize and separate the deliverables.
  2. Conquer the problem by working on each piece individually until solved; that is, build each deliverable separately.

This approach works very well for reducing risk by breaking a complex problem into small manageable chunks. However, this approach also has a severe drawback when applied without a non-technical infrastructure. Namely, it produces stovepipe systems (automation silos). The effects of stovepipe systems are lost business knowledge and lost cross-organizational business view, severely impacting business analytics and data mining activities.

Lost Business Knowledge
Most businesses are very complex; and as organizations mature, their business complexity increases. As business complexity is broken into smaller and less complex components, the interrelationships among those individual components are lost. Much of the business intelligence is contained in these lost interrelationships, and that is a problem for BI applications. Most BI applications, especially data mining applications, expect to find “golden nuggets” of business wisdom embedded in these complex interrelationships.

Lost Cross-Organizational Business View
Although business managers can answer most questions about their own business functions, when asked a question spanning two or three lines of business (where complex interrelationships have been lost), they have to scramble for weeks to piece together the answer. Fundamental business questions, such as the ones illustrated in Figure 1, are presenting multimillion dollar problems to large organizations.

Figure 1: Fundamental Business Questions

Enterprise infrastructure activities, technical as well as non-technical, are strategic cross-organizational activities, which must be managed and coordinated by a central enterprise architecture group, as illustrated in Figure 2.

Figure 2: Management of Enterprise Infrastructure Activities

Enterprise Architecture
An enterprise architecture (EA) is comprised of a set of pictorial representations (models) of the organization in terms of business functions, business processes and business data. Each EA model is supplemented with supporting metadata, such as standard definitions, business rules and policies. The purpose of EA models is to document the set of business actions performed on any real-world object in the course of conducting business. In other words, EA models describe the actual business in which the organization is engaged.

A fully documented enterprise architecture includes at least five architectural components, as shown in Figure 3.

Figure 3: Architectural Components of an Enterprise Architecture

  1. Business Function Model
    This model depicts the hierarchical decomposition of an organization's nature of business; it shows what the organization does. This model is instrumental for organizing or reorganizing a company into its lines of business. It is common that one vertical line of business supports a major business function. Two examples are the loan origination division and the loan servicing division of a mortgage lending institution.

  2. Business Process Model
    This model depicts the processes implemented for the business functions; it showshow the organization performs its business functions. This model is essential for business process reengineering as well as for business process improvement initiatives, which often result from BI projects. An example is loan payment processing, a process performed by a loan servicing division of a mortgage lending institution.
  3. Business Data Model
    This model, which is commonly called the enterprise logical data model or enterprise information architecture, shows what data is part of the business activities of the organization. This model depicts the:
    • Data objects participating in a business activity
    • Relationships among these objects as they exist in the actual business activities
    • Data elements stored about these objects
    • Business rules governing these objects

    Since data objects and data elements are all unique, they appear in the real world only once. Therefore, they are documented in the business data model only once, even though they are redundantly stored in many hundreds of physical files and databases. There is only one business data model for an organization. This model and the metadata repository are the two most important non-technical infrastructure components for an evolving BI decision support environment.

  4. Application Inventory
    An application inventory is an accounting of the physical implementation components of business functions, business processes and business data (objects as well as data elements). It shows where the architectural pieces reside in the technical architecture (e.g., programs, databases, tables, columns). 

    Organizations should always identify, catalog and document their applications as well as the business rules about their business data as part of the development work on every project – but they seldom do. Such inventories are paramount for performing impact analysis. Case in point: the colossal effort of Y2K impact analysis without such inventory!

  5. Metadata Repository
    Although “a picture is worth a thousand words,” business models without words are not worth much. The descriptive details about the models are called metadata. Business metadata is collected during business analysis, and technical metadata is collected during design and construction. The two types of metadata are linked to each other and are made available to the business community of the BI decision support environment. Metadata is an essential navigation tool.

Enterprise Standards
Organizations must establish architectural standards for their BI decision support environment in the same way they establish standards for their Web site. It would never occur to anyone to build a Web site where each Web page had a different look and feel. In the same vein, it should never occur to anyone to build a BI decision support environment where each BI application has a different look and feel. Therefore, all BI applications must adhere to the same enterprise standards, which are partially listed in Figure 4.

Figure 4: Enterprise Standards

Development Approach
The Business Intelligence Roadmap includes a set of all major activities and tasks that are appropriate for BI projects. Not every BI project will have to perform every single activity in every step. Some BI projects may justifiably skip activities within a step, combine activities from different steps into one or skip entire steps. However, no BI project should be developed ad hoc. Organizations should have some guidelines for their project teams listing the minimum number of activities required (work breakdown structure), the mandatory deliverables, signoff requirements and workflow dependencies in order to control the risk of the projects. (To receive a complimentary copy of the Business Intelligence Navigator, designed to help chart the business intelligence journey, please visit

Data Naming and Abbreviations
Similar to creating a Web site, some standards should be applied when creating BI applications (e.g., naming and abbreviation standards).

Abbreviations are part of naming standards, but they only apply to physical names (e.g., column names, table names, program names), not to business names. A standard enterprise-wide abbreviation list should be published, and it should include industry-specific and company-specific acronyms. Every BI project team should be required to use these abbreviations and acronyms.

Metadata Capture
Metadata is a world unto itself. Large amounts of descriptive information can be collected about business functions, business processes, business data objects, business data elements, business rules, data quality and other architectural components. There should be standards or guidelines for who captures which metadata components, how, when and where. The metadata repository will then have to be set up in such a way that it will support the standards for meta data capture and usage.

Logical Data Modeling
Logical data modeling is a business analysis technique (not to be confused with logical database design). Every business activity or business function uses or manipulates business data in some fashion. A logical data model documents those logical data relationships, irrespective of how the functions or the data are implemented in the physical databases and applications.

Project-specific logical data models should be merged into one cohesive integrated enterprise logical data model. This activity usually is – and should be – included in the job description for data administration, which is part of the enterprise architecture group. The enterprise logical data model is the baseline business information architecture into which physical systems (operational or decision support, including BI applications) are mapped.

Data Quality
Information can only be as good as the raw data on which it is based. Most organizations have a lot of dirty data – too much to cleanse it all. Guidelines must be established to triage (categorize and prioritize) dirty data for cleansing. In addition, standards must be created to define acceptable quality thresholds that specify how to measure data quality during database loads. Instructions for error handling and suspending dirty data records should also be part of the standards.

Testing standards should specify what types of tests should be performed and who should participate in the various types of testing. Guidelines should be provided that describe the types of test cases that are required at a minimum, how much regression testing to perform and under what circumstances. A brief description of a test plan, maybe even a template, as well as instructions for how to organize and manage the various testing activities should be included.

The BI decision support environment will have multiple target databases and multiple BI applications. Since BI applications are not standalone systems, their development must be coordinated and reconciled so that consistency across the BI decision support environment can be guaranteed.

BI data is derived from operational data. Therefore, security guidelines applicable to the operational data are also applicable to the BI data. If data is summarized and drilling down to the detail is not enabled, some of the security features can be relaxed. But rather than allowing each project team to make up the rules as they go along, the owners of the data should establish security standards. These security standards should be used to guide the project teams on what type of security measures are mandatory for what type of data exposure. These standards should include guidelines for categorizing security risks. Security risks should be considered for data sensitivity, application security, network security, security against intrusions, hacks, viruses and other nuisances on the Web.

Service Level Agreements 
Organizations function according to explicit or implicit business principles. Business principles are explicit if stated in mission or vision statements, implicit if they are just “understood” by the staff.

Policies and Procedures
Standards and guidelines should also cover the policies and procedures of an organization, such as operating procedures, project change control procedures, issues management procedures and dispute resolution procedures. Additional topics such as the communication processes, estimating guidelines, roles and responsibilities and standard format of documents should also be part of the policies and procedures. These  policies, procedures, standards and guidelines must be value-adds for the organization as a whole  or they should not exist.

Non-Technical Infrastructure Activities
The non-technical infrastructure activities need to be performed linearly, as indicated in Figure 5.

Figure 5: Order for Non-Technical Infrastructure Activities

  1. Assess effectiveness of non-technical infrastructure components.
    The policies, procedures, guidelines and standards, which are all part of the non-technical infrastructure, exist to assist in the coordination and management of the BI decision support environment. They should not hinder the project teams or slow them down unnecessarily. Therefore, the appropriateness and effectiveness of all non-technical infrastructure components must be reviewed with each BI project. If some components are inadequate, they must be expanded, reduced or revised as necessary.
  2. Write a non-technical infrastructure assessment report.
    Once all the components of the existing non-technical infrastructure have been reviewed and assessed, a report should be prepared, outlining the findings and giving recommendations for improvement. If there are missing non-technical infrastructure components, it will be necessary to prioritize which ones to include in the next BI project and which ones to defer.
  3. Improve non-technical infrastructure.
    Time estimates for modifying or improving non-technical infrastructure components as well as for establishing new components must be reflected in the project plan.

Deliverable Resulting from These Activities

Non-Technical Infrastructure Assessment Report
This report should document the deficiencies of the existing non-technical infrastructure and should cover:

  • Standards
  • Use of a development methodology
  • Estimating guidelines
  • Scope management procedure
  • Issues management procedure
  • Roles and responsibilities
  • Security process
  • Metadata capture and delivery
  • Process for merging logical data models into the enterprise logical data model
  • Data quality measures and triage process
  • Testing process
  • Service level agreements
  • Support function
  • Dispute resolution procedure
  • Communication process

Include a section for proposed improvements for those selected non-technical infrastructure components, which will be included in the BI project.

What to Do

  • Do identify the missing or ineffective non-technical infrastructure components because they provide the glue for cross-organizational integration.
  • Do pay attention to metadata. Metadata is much more than documentation. It facilitates navigation through the BI decision support environment and is an integral part of every BI project.
  • Do provide standards for your formal project documents. For example: each document must have a title, description, purpose, author, owner, creation date, latest update date, latest version number, revision history, page numbers and signoff space.

What Not to Do

  • Don’t attempt to build all components of the non-technical infrastructure at once. Start with standards, then model the common business data architecture, and go from there. This is an iterative refinement process.
  • Don’t skip data standardization because you think it is too difficult or takes too much time. One of the main reasons for a BI application is to address data quality and data standardization.

Tips for Success

  • Use a meta data repository if you have one. If you do not have one, but you have a fairly sophisticated CASE tool, use that CASE tool in the interim until a more permanent repository solution is established. At a minimum, document the source to target data mapping and the transformation rules in some format that can be accessed by the businesspeople and that can later be used to populate a metadata repository.
  • Determine what types of standards, guidelines and procedures already exist in the organization. Identify what is useful, what is not and what is missing. The fastest way to assess this is to talk to the staff of the following groups: data administration, strategic architecture, quality assurance, security, audit, standards, training and even operations.
  • Include at least one non-technical infrastructure requirement with every BI project. Implement at least 80% of its functionality, and plan to revise it in future BI projects.
  • Reuse as many as possible of the standards, guidelines, procedures, etc., that already exist at the organization. Try not to reinvent the wheel. Get input from as many people in IT and on the business side as you can to develop a practical and useful non-technical infrastructure at your organization.

With both the technical and non-technical infrastructure in place, the chance of a successful business intelligence system is high.

(Source: Moss, Larissa and Atre, Shaku. Business Intelligence Roadmap – The Complete Project Lifecycle for Decision-Support Applications, Addison Wesley Professional, 2003.)


Copyright Shaku Atre 2006. All rights reserved.

Dig Deeper on Operational business intelligence

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.