SAN DIEGO -- Change is inevitable, so plan for it. That was the prevailing message of the summer TDWI conference...
here, which was focused on agile business intelligence (BI).
While attendees lauded the conference’s content, some remained confused over agile BI, even two days into the event. Some thought it referred to applying agile development principles to their BI environment. A few saw it as yet another industry buzzword. Others thought it meant the ability of BI to help an organization be more adaptable, nimble and quick.
They’re all partially right, explained Wayne Eckerson, research director for The Data Warehousing Institute (TDWI), a Renton, Wash.-based analyst firm.
“When TDWI uses the term [agile BI], it’s in the broadest context possible," Eckerson said. "It refers to any of the organizational processes, technology and infrastructure required to make the business go as fast as possible.”
Practically speaking, he said, that means agile BI spans many areas and is more mentality than methodology. It means initiatives like revamping an organization’s BI team and program structure for better business alignment and improving how information is disseminated and used. It means developing powerful BI and data management infrastructures that are flexible enough to handle change. And it means applying agile development techniques to BI projects, Eckerson explained. But given that many of these goals are not new, why is the agile BI concept gaining traction now?
Eckerson pointed first to the recent uptick in executive interest in using BI and analytics to be more agile and competitive. Technology innovations in data warehousing appliances, analytical databases, cloud computing and open source are “changing the game,” he said, enabling organizations to ask and answer analytical questions that historically weren’t possible as a result of storage, processing and cost constraints. And there’s a genuine move toward using agile development concepts to help overcome the stigma (and, often, the reality) that IT organizations are too slow to respond to business needs.
“There’s a growing momentum toward failing fast and not trying to be perfect. If something breaks and you make a mistake, the cost of fixing the mistake is often outweighed by the advantage of going fast,” Eckerson said. “So fail. Make a mistake. Build your architecture and fix things after the fact. Let your developers go out, build these solutions quickly and reconcile the models afterwards. It’s OK, as long as you’re all developing to certain core principles. Not hard-and-fast rules, but core principles.”
That may be a tough pill for some BI and data management professionals to swallow, he acknowledged. After years of requirements templates, careful modeling and attempts to get the data perfect, the new directives are to develop quickly, iterate often and be OK with delivering 80%. Granted, this approach might not be ideal for an executive’s financial reporting, but the average business user will probably be OK with getting some data sooner, rather than waiting for the perfect solution.
Eckerson offered even more advice in his Monday morning keynote, sharing “Five Secrets to Building an Agile Adaptable BI Environment”:
- Align with business. Evaluate BI program structures, he recommended, and consider agile development tactics, such as “embedding” BI developers in a business department and cross-training staff in all facets of BI. He referenced a Netflix case study that involved disbanding formal requirements and QA teams in favor of a cross-trained BI staff, where a single developer may build out an entire solution.
- Slow down to speed up. Eckerson cautioned that the emphasis on fast delivery should not mean ignoring the business user or getting seduced by attractive technology before knowing what’s really needed.
- Anticipate the business. The more that BI professionals know the business side ”inside and out,” the better position they will be in to develop applications and infrastructures that can handle future business requests. Given that businesses often don’t want to fund architecture – instead, focusing on projects – Eckerson recommended building out BI architectures incrementally.
- Manage expectations. “The harsh reality of BI is that we’ll never, ever move as fast as the business wants – but we can move as fast as they need,” Eckerson said. BI teams with a good business rapport can help requestors step back to consider what will do the job. Other helpful tactics are developing a visual BI portfolio (a concept pioneered by Baseline Consulting) and having a solid governance structure to manage incoming requests.
- Cede control back to the business. BI organizations need to educate business users and develop standards and governance that enable BI without sacrificing the user’s desire for data. It’s akin to raising teenagers, Eckerson joked. When you can no longer control their actions, you hope that standards and values you’ve espoused will help them make good decisions.
These messages seemed to resonate with many of the attendees at the event. Roger Barker, enterprise data warehouse manager with El Segundo, Calif.-based semiconductor maker International Rectifier, came to the event partially to get advice on developing the executive sponsorship required for him to grow his BI organization beyond his current one-man show. His goal is to evolve his organization’s three-year-old data warehouse into a full-fledged BI program. Barker said applying agile development methodologies to BI is an intriguing idea, but “we have other things to do first.”
Another attendee, Russell Brown, a BI analyst with Jacksonville, Fla.-based financial services provider FIS, was at the event to present, take a CBIP exam and bring back new ideas.
“The focus for us in the next couple of years is scalability,” Brown said. “How do we take these complicated statistical analyses and put those into our client’s hands so they make sense of [them] -- without our end users needing to have a Ph.D. in statistics?”