The term agile business intelligence can have a broad meaning that covers overall efforts to make BI systems more flexible and improve the alignment between BI teams and business units. But a growing number of organizations are taking it more literally, as they move to replace traditional BI project management processes with agile development techniques that can accelerate deployments.
For example, trucking company U.S. Xpress Enterprises Inc. used an agile BI development methodology to build a real-time BI system that provides its fleet managers with up-to-date information about truck locations and idling levels – i.e., the amount of time that trucks are run in idle mode while stopped or in parking lots. Timothy Leonard, the company’s chief technology officer and vice president of information management, said he set up a “tiger team” that adopted an agile development strategy to get the system up and running more quickly than it could have using traditional development methods.
Leonard noted that speed was essential to selling corporate executives and business users on the value of the real-time BI capabilities. “We knew we had to come up with some real fast wins,” he said, adding that the new system, which was deployed last summer, is already saving U.S. Xpress more than $3 million per quarter through reductions in gas consumption made possible by the idling data.
Agile BI development was one of the main topics of discussion at The Data Warehousing Institute’s BI Executive Summit in Las Vegas last month, where Leonard was among the speakers. The agenda included several well-attended sessions on agile BI, and agile development terms such as “sprint” and “scrum” were commonly heard during discussions between attendees.
Jim Gallo, a BI and data warehousing consultant at Information Control Corp. (ICC) who also spoke at the TDWI event, estimated in an interview that between 15% and 20% of his corporate clients have adopted agile BI development processes thus far. Another 40% to 50% are in the exploratory stage on agile BI, he said.
Using an agile BI development methodology: ‘the new reality’?
Columbus, Ohio-based ICC pushes hard for an agile development approach when planning BI projects with customers, according to Gallo. “Our view is that this is the new reality, so adapt or perish,” he said. In the end, “we do whatever the client wants us to do. But if we’re being held accountable or responsible for the [project] delivery, we do everything in our power to bring agile BI processes into it.”
Echoing Leonard’s comments, Gallo and other advocates of agile BI development said that it can enable BI project teams to quickly deliver high-value features to business users, instead of making them wait for months on end while a full BI system is built using a traditional waterfall development approach. When done right, agile BI projects typically provide initial capabilities to users within four to six weeks from the start of the planning process, Gallo said.
Both Gallo and Ken Collier, a BI and agile project management consultant at Cutter Consortium, recommended rollouts of new features every two weeks after the initial iteration. BI business requirements can be prioritized at the start of the process and then reshuffled as needed so that the most important features at any given time are always at the top of the delivery plan, they said.
“The trick is avoiding a mess on the back end from uncoordinated changes,” Collier said. “But the fact is, [requirements] are going to change whether we do agile or not.” He added that in his experience, an agile BI development methodology usually results in a higher-quality system because the BI features are tested out on an incremental basis. Keeping business users involved is a must, though: They need to participate “on a daily or weekly basis so that they’re continuously giving you feedback,” Collier said.
Because of its incremental nature, agile BI development can also reduce the risk of a project that ends in failure without a functional system after all the time, effort and money invested in it by an organization, according to Gallo and Collier.
Seeking a better way via agile BI development
A BI manager at another trucking company said it was exactly that scenario – a failed attempt to build a BI system using traditional methods – that spurred her organization to switch to agile development for a recently-begun second try. “What we’ve realized is that you can’t do it all at once, unless you want to be here in a year with nothing to show for it,” said the BI manager, who asked not to be identified.
The initial BI project, which was also IT-led, resulted in “lots of missing fields and an incomplete system,” the manager said. As part of the new agile BI methodology, the project team is shooting to complete an initial iteration of the BI system within the four- to six-week window recommended by Gallo and then release updates every two to four weeks. In addition, “we’ve really shifted our focus to business value first,” she said. “If we’re not delivering business value, why are we doing it?”
Matt Ammiller, BI program manager at Forest City Enterprises Inc., said the Cleveland-based real estate development and management company used agile development principles last fall in a proof-of-concept project for a data warehouse and BI system designed to support analysis of data from its ERP system. The work began in September, and the project team demonstrated an initial system to corporate executives in December, complete with some analytical cubes and basic BI dashboards.
Ammiller sees agile development as the way of the future for BI, though shifting to it wholesale won’t be easy. “I’m looking forward to it, but I think it’s going to rock our world compared to the way we’re used to doing things,” he said. “It will be a big change for our company.” That includes making end users “an integral part of the project team,” Ammiller added. “We need them to come on board with it.”