News Stay informed about the latest enterprise technology news and product updates.

Tackling big problems with free business intelligence tools

Data scientist Brand Niemann recreated large government databases in days using free business intelligence tools and advises other users to “do your own IT.”

SaaS business intelligence (BI) vendors have been trying to advance their business intelligence tools to the point where they can be used by the masses.

Vendors are jockeying for control of the BI market, as witnessed by the many product announcements made by MicroStrategy, Birst, Tibco, QlikView and others in the past few weeks, all promising any number of superlatives from ease of use to mobile and collaborative capabilities. Many offer free or trial versions of their product online for users to test before making a purchase.

Brand Niemann, former senior enterprise architect at the Environmental Protection Agency (EPA) offers this advice to users overwhelmed by an increasingly cluttered BI marketplace: “Do your own IT.”

“If you have enough subject matter expertise, and you have access to the data, then all you need to do is get the tool to do it,” Niemann said.

Niemann, now a data scientist, author and meteorology and air pollution science PhD, certainly had the subject matter expertise. But until SaaS BI tools that met his needs were made available, he was stuck using an outdated version of S-Plus and Microsoft Excel to do his work.

Before finding a less expensive SaaS-based solution, Niemann had to use some of the EPA’s budget to hire outside contractors to get higher level work done in the government sector. The contractors, who used more traditional BI tools, came at a high price – around $350 an hour.

Niemann wasn't actively looking for a SaaS-based solution, but then a colleague informed him that Tibco had purchased the rights to two programs he had used in the past -- S-Plus and Spotfire. Tibco packaged the technology in a way that resembled what Niemann had hoped the former S-Plus would grow into.

Tibco went on to release the software as Silver Spotfire 1.0 [the 2.0 version came out last week], which comes with a free trial version that Niemann tested to see if the software was suitable for the EPA.

Free BI tools vs. government spending

Once Niemann accomplished his goals at the EPA using the free version of Spotfire, he tried to tackle some loftier projects with the free BI tool. is a website run by the federal government designed to show where money from the Recovery Act is going.  The website was founded with the goal of becoming a transparent, interactive database that allowed for the reporting of government waste.

Niemann thought the website was flawed, so he decided to test it himself.  He downloaded the raw data (98 columns x 517,589 rows) that went into the database and set out to recreate it in Spotfire. was created over nearly a year at a cost of $9.5 million, but Niemann says he was able to recreate a more thorough version of it in just a few hours, leaving the proof available for download on his website.

After having success with recreating, Niemann set out on another project -- the Federal IT dashboard.

Designed to detail IT spending in a transparent way, was created over six months at a cost of $8 million, about half of which was used to upgrade databases.

“They bragged in a sense that they were able to do that in only six months,” Niemann said. “I recreated it in three days for essentially free, except for the fact that the EPA had to pay my salary.”

Niemann did need to upgrade beyond the free model because the size of the raw data for both projects exceeded the space limits of Spotfire’s free offering.

Advice for users in the self-service BI market

Aside from, “Do your own IT,” Niemann has plenty of advice for users searching for BI tools and a few capabilities he would like to see in the future.

Niemann recommends users look for tools that allow for easy data integration, preserve the data once a project is finished and are sustainable solutions.

“You’ve got to have your data in a tool that preserves your data, preserves the work you do with your data – otherwise these are one-shot things,” Niemann said, adding that some of the cost savings he has seen come from not having to rework old projects.

Integration proved to be crucial in Niemann’s work -- especially when it came to mapping data columns from one data set to another. He thinks all users should understand their data integration needs before making a decision.

Niemann also recommends working with the user community to discover new applications for BI tools.

“I’ve generally found that whatever I want to do someone else has already figured out how to do it,” Niemann said. “I just need to find that person.”


Dig Deeper on Business intelligence development

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.