News Stay informed about the latest enterprise technology news and product updates.

Competition fuels business intelligence in government

The competition in government for the capture of the public's approval and attention is fueled by data warehousing and business intelligence.

This article originally appeared on the BeyeNETWORK.

It was always fascinating to me during the early days of The Data Warehousing Institute (TDWI) to observe how specific industries led the charge in adopting data warehousing and business intelligence principles while others lagged behind. In looking at some of those early leaders – retail, banking and telecom – it was interesting to explore what market forces were driving these sectors that might have been absent, or lagging, in the other industries that seemed to be in no hurry to implement what was obviously an important new tool.   

It didn’t take very long to discover one key driver: competition. Let’s take a look first at retail, where Wal-Mart pioneered data warehousing and, in turn, used their data warehouse to catapult itself to the top of the industry. It was Sam Walton’s mantra that the only way it could compete with the other retail giants was if Wal-Mart could track every single sale of every single product in every single store every single day. Though the technology did not exist at the time to meaningfully handle the huge amounts of data that Walton envisioned, his push gave us one of the earliest successes in data warehousing through the emergence, even if after Sam Walton’s death, of the Teradata technology. Wal-Mart soared above its competition and the retail world has never been the same.

This intense competition was also present in telecom and banking. In terms of telecom, the era of deregulation and the breakup of AT&T created a free-for-all in competition. This was happening for every subscriber’s long distance business. Companies like MCI, Sprint, GTE and AT&T, as well as smaller organizations, needed to understand much more about their own users and those of their competitors. These companies did this to offer prospective customers deals they could not refuse and lure them away from their current providers. We all remember those days when everyone was competing to get our long distance business and offering us discounts, frequent flier miles and rebates to win us over as new customers, or keep us on their books as satisfied ones. The churn rates were so high that it forced the major players into action.

The banking and credit card industry was also extremely competitive. It seemed as if they were pre-selecting everyone with “exciting” new offers. Such credit card deals, which frequently gave away “great prizes,” were difficult to turn down. This undoubtedly meant the companies knew a lot about their customers. If you liken the situation to a war, competition was the driver and business intelligence was the ammunition.    

So was there a lack of competition in the industries not using data warehousing? Did this indicate a lack of business intelligence as well? More importantly, was there competition in government? And what was the role of its presence or absence in the adoption of systems to produce business intelligence?

As I have oftentimes mentioned in my articles, business intelligence in government is an interesting phenomenon. However, the U.S. government may be the world’s greatest collector and keeper of data. This is especially evident in collecting the decennial Census; the annual submissions to the IRS; the myriad of mandatory surveys orchestrated by the Departments of Labor, Commerce and Agriculture; the research data on education and health care; the massive repositories of information from every application for a passport, a visa, a pilot’s license or housing assistance; the records from our law enforcement and court systems; etc.

The raw materials for business intelligence are available, and the government effectively uses them to generate statistics on countless topics. This is obvious by reading the titles published by the Census Bureau, the Bureau of Labor Statistics, the Agricultural Census or the Bureau of Transportation Statistics. Despite this, there are precious few dynamic, real-time, state-of-the-art data warehouses that can be queried on an ad-hoc basis to serve as analytical platforms for the production of business intelligence—hence, for problem solving. Even in terms of homeland security, where the push for real-time information is an on-going priority, much work must still be done.

What’s the snag? Where’s the problem? Is there no competition in the government?

Before going further, we must remember what competition really is. Webster’s Dictionary defines it as “a contest between rivals.” Essentially, two or more parties are trying to obtain something that only one can possess. If you frequently lose to your market rivals in the private sector, you will eventually go out of business. Clearly, you must find smarter ways of doing business. This is why business intelligence is so important.

In the public sector, certain agencies often hold monopolies over a line of business. While this is usually due to statutes, monopolies are occasionally jurisdictional. This is the case when it is based on some geographic or organizational disposition. However, you do not have to worry about another government agency (with superior performance) taking over your organization.

Well there is clearly competition in government, and the result of that competition is often seen in reorganizations, reassignments, consolidations, etc. But ultimately, I feel that the competition in government is of another kind. The competition in government is for the capture, directly or indirectly, of the public’s approval and attention as expressed in the allocation of resources and responsibilities.

Every department needs to capture their share of dollars in the federal budget. This needs to be established—program by program, bureau by bureau, agency by agency—within the Executive Branch in order to conform with the President’s budget submission to Congress. Then it needs to be defended on Capitol Hill to both preserve and incorporate the requirements and aspirations of our elected representatives in their eagerness to outguess and oversee what the Executive Branch has proposed. This effort is on-going, painful and extremely competitive. The political well-being of appointed officials is often measured in how well their agencies did in the budget allocation battles. And this entails intense competition.

But why doesn’t this competition drive the development of more powerful business intelligence systems, environments and solutions? Why aren’t more federal data warehouses being utilized for the creation of knowledge to help solve societal problems that affect us all?

I believe it is because the “marketplace” for this intelligence is flawed. Whereas most markets operate by supply and demand, the market here is different. It simply takes too long for the public to express its pleasure with an agency’s performance. Furthermore, service offerings are often delivered in confusing packages due to practices, policies and even laws. Although the media pressures people to see issues a certain way, the general public must wait for changes to occur. This wait often seems to go at a glacial pace.

What if people were allowed to purchase specific government offerings, much like the traditional marketplace? While this sounds like opening Pandora’s Box, it would definitely be an interesting option to explore.

Dr. Barquin is the President of Barquin International, a consulting firm, since 1994. He specializes in developing information systems strategies, particularly data warehousing, customer relationship management, business intelligence and knowledge management, for public and private sector enterprises. He has consulted for the U.S. Military, many government agencies and international governments and corporations.

Dr. Barquin is a member of the E-Gov (Electronic Government) Advisory Board, and chair of its knowledge management conference series; member of the Digital Government Institute Advisory Board; and has been the Program Chair for E-Government and Knowledge Management programs at the Brookings Institution. He was also the co-founder and first president of The Data Warehousing Institute, and president of the Computer Ethics Institute. His PhD is from MIT. Dr. Barquin can be reached at

Editor's note: More government articles, resources, news and events are available in the BeyeNETWORK's Government Channel. Be sure to visit today!

Dig Deeper on Business intelligence strategy

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.