News Stay informed about the latest enterprise technology news and product updates.

Paths, self-learning, and business intelligence lessons

Are business intelligence, data integration and systems development practitioners reaping the benefits of the "path" phenomenon?

This article originally appeared on the BeyeNETWORK.

One of our recent clients has particular protocols for the locations where contractors and consultants may park. This is essentially irrelevant to the actual engagement, although the fact that the contractor garage was physically separate from the client offices meant that to get from one’s car to the building, one had a five-to-seven minute walk across a largely unkempt, dusty field.

The interesting thing about this walk, though, is that a handful of “natural” paths leading from the garage to each building, about one foot wide, have been etched into the field. Essentially, the necessity of contingent workers has created what are likely to be the most efficient routes between the garage and the buildings. This actually reminded me of a very similar story told about the quadrangle at my university. Before paths between school buildings were paved, the planners let the students walk in the direction of their choice between classes, and the resulting dirt paths emerged as the walkways between the buildings and across the quad.

So, given the fact that I was obliged to take this five-minute stroll, I took the time to do a little thinking about this phenomenon, and how it applies within system development, data integration, and business intelligence (those five-minute chunks add up to a lot of thinking over a period of time).

The first notion is what we could call individual intention. It is not clear when the first contractor decided that it would be more efficient to walk across the field instead of the long way around the streets, whether that contractor  thought about mapping the most efficient path, or even explicitly dragged his/her feet along the growing plants to dig out a dirt path. In fact, one might guess that over time, different folks chose different strolls to get to their desks, but there was no standard or convention established for the way to go, nor did the contractors meet to discuss the layout of the eventual path. However, as a path began to emerge, the natural decision for any specific individual was to follow that path. The intention evolves along with the collective result, and it becomes second nature for the individual to contribute to path creation – this is the individual intention.

The second notion is one of implicit collaboration. No organized meeting was held among the contractors to map out the path, but as we just discovered, the natural inclination is to follow it. As opposed to trying to create a new path, traipsing over foot-high weeds that can scratch or ruin clothing and shoes, it is much easier for a person to walk along where the plants have already been worn away. But doing so actually contributes just a little bit more to wearing away the path; by observing the existing clearing, each person is implicitly (and perhaps inadvertently) collaborating with all the others who have opted for this approach.

The third notion is the self-organized emergence of form where none exists. Paths emerge over time because traffic iteratively tramples and destroys any vegetation. But no single individual takes a rake to the field to clear the plants, nor does any specific trip appear to have a specific impact in creating the path. But the cumulative effect of individuals clears the path, with each trip contributing a small bit to make the next trip easier.

The fourth is explicit compliance. Once the path begins to emerge, it becomes the clear choice. In essence, each time a person chooses the emergent path, it contributes to the “knowledge base” that new contractors will consult when they have to decide how to walk from the garage to the building for the first time.

The fifth (I told you I spent a lot of time thinking about this) is repeatability. If it works in one place, then apply the same process in others. This is clear when you look down on the field and see more than one path between the different buildings and the different entries and exits from the garage. Multiple paths exist. Some cross one another, but there is never more than one path between a garage entryway and the same building.

These same notions exist all over the data integration and business intelligence landscape. For example, identity resolution schemes are enhanced by reviewing the decisions analysts make when faced with a similar process. If two records are presented as possible matches for a candidate entity string and the analyst selects one of the two presented, some aspect of that decision could be integrated into the “knowledge base” to smooth the pathway for making the right decision some time in the future. As a different example, patterns of fraudulent activity that emerge in transaction data contribute to the ability to recognize fraud in real time. A third example is the development of published business intelligence reports driven by reviewing the kinds of ad hoc queries that are performed over a period of time.

These are just a few examples, and I would challenge the readers to e-mail me with their own experiences. If these aspects of “system development” are readily apparent in tangible artifacts such as self-organized pathways, imagine how valuable the apparent lessons are if they can be applied to business intelligence application design!

David is the President of Knowledge Integrity, Inc., a consulting and development company focusing on customized information management solutions including information quality solutions consulting, information quality training and business rules solutions. Loshin is the author of Master Data Management, Enterprise Knowledge ManagementThe Data Quality Approach and Business IntelligenceThe Savvy Manager's Guide and is a frequent speaker on maximizing the value of information. David can be reached at or at (301) 754-6350.

Editor's note: More David Loshin articles, resources, news and events are available in the David Loshin Expert Channel on the BeyeNETWORK. Be sure to visit today!

Dig Deeper on Business intelligence best practices

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.