BOSTON -- The agile development process for business intelligence and data warehousing begins with a flaw. It assumes that internal customers -- often business users -- know precisely what they want when they request something from IT.
But that may be assuming too much, according to Ken Collier, the keynote speaker at The Data Warehousing Institute's agile business intelligence (BI) conference and the agile analytics practice lead for Chicago-based IT consulting firm ThoughtWorks Inc.
"They know that they need a report that does what their current quarterly financial report does or what their weekly KPIs [key performance indicators] give them," he said. "They don't have knowledge about all of the other things that are possible. And, therefore, they tend to narrow their perspective and write [requirements] that are very down the middle."
Yet, Collier was clear there is value in agile BI and data warehousing, which calls for IT to start with a prioritized list of requests, build a working rather than complete product, share the results and adjust as needed. Before embarking on an agile journey, though, IT departments need to figure out what to build. To do that, Collier recommends BI and data warehousing practitioners start thinking like entrepreneurs.
From construction to creation
A common analogy of IT departments is that they are like the carpenters of complex systems: Plans are drawn up by a customer, and the IT department is asked to build and then implement the design, Collier said.
Referred to as the waterfall model, progress tends to flow in one direction and encourages little interaction with those who will be using the final product and provides little room for change midstream. This, Collier said, is historically how IT departments have conducted business, and in some cases, it has helped to create a disconnection between IT and the business.
More on agile BI and data warehousing
How agile development can be used to boost mainframe performance
Learn more from the Agile Manifesto
Agile BI advice welcome, but confusion remains
But the waterfall model may not be appropriate for every situation, Collier said. Some design decisions carry costly penalties and should be nailed down prior to a project's start, but others are malleable and can be changed or added to along the way.
"Getting it right is not the same as getting it all the way finished," said Collier during his talk, entitled Are We Carpenters, Cabinet Makers or Furniture Makers? "We do need to get our decisions right, but we don't need to get them all the way finished because we know that change is inevitable and we need to be able to adapt."
More recently, businesses have begun using another approach more akin to cabinet making, according to Collier. Under the agile model, plans are still drawn up by a customer, but, like building cabinets for a homeowner, input and satisfaction as things change along the way are also considered. To do this, the agile process hinges on short, iterative cycles that produce a working (but not complete) version for customer review. The IT department can then incorporate changes into the next iteration.
"There is the ability to increase the likelihood of a project's success by partnering with our customers, building what they want, and adapting to the change as it comes along rather than trying to lock things down early," he said.
Collier suggested that it's time to take things one step further. Rather than cling to the cabinet makers analogy and design specs for customers who may not even know what they want, he believes IT departments should start to think of themselves as furniture makers.
"With furniture building the tables are turned," he said. "The craftsman, the artisan, is building something and finding out if it's something customers will buy. And they're building because they have expertise in both form and function."
But, he said, that will require a shift in perspective and a change in approach.
Think like a startup
Inspired by the popular book, The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries, Collier began to see parallels between today's startups and the challenges IT departments face when building new systems. One of biggest: not knowing who the customers are or what resources are needed.
"We have a sense for who they are, but how many of you have worked on projects where you've built the thing you think [a] business wants only to find out nobody uses it because they're entrenched in their spreadmarts?" he asked.
The lean startup approach removes the initial flaw in agile development for BI and data warehousing; it doesn't make any assumptions. Instead, builders operate in a "build, measure, learn feedback loop" by formulating a hypothesis, building something small based on that hypothesis, testing it out and then either staying the course or changing directions, Collier said.
It may sound similar to the agile development process, but unlike the agile methodology, the items produced through these microcycles are so small -- concepts, sketches or prototypes -- they could not be considered deployable.
"With the lean startup approach, we're just trying to figure out what the right thing to build is," Collier said.
Without going into too many details, Collier described a team he worked with that added new buttons to users' reports without telling them. If the users clicked through, they were met by a screen explaining a concept the team was considering and asking the user to weigh in.
"It was a simple, rapid way of just gauging the interest of the customer," he said.
But, Collier warns, the lean startup approach shouldn't replace other development methodologies such as agile, it should be used to answer the question of what should be built.
"As soon as we have a sense for what we should be building, then we enter into an agile development cycle," he said. "The lean startup gives us the ability to be innovative, creative artisans."