Reflecting on the B.I industry as it relates to Agile methodologies: BI tools are not keeping up with user demands The BI life-cycle is a bit different from a software SDLC.
Agile development works best when you can apply object oriented designs that allow for the separation of concerns. This gives developers the flexibility to build and design applications without needing to worry about the data layer. BI on the other hand is all about the data layer.
One of the problems is the mix and match of multiple technologies – ETL, Meta Data, Reporting, Dashboards, Adhoc queries, and so on – each is/has its own development environment.
As part of B.I evolution thus far big companies have gobbled up products and now market them as integrated BI suites. Take SAP for example, which started off with ERP software, then bought Xcelsius and Business Objects, who had bought out Crystal Decisions. SAP now bundles all those products into what is marketed as a "single integrated" BI suite, but what you end up with are several programs that are forced to work together when they weren't designed to do so.
It is very difficult to follow Agile methods when you have to manage each component of your BI solution with a different set of tools.
This is why Meta Data is key – Meta Data allows you to become more Agile with BI because it allows for report development against one integrated source. The Meta Data layer, for example, could be made of multiple databases, data warehouses, text files, xml, etc. – but the report developer sees only the integrated Meta Data.
MicroStrategy is one enterprise BI suite that allows you to manage the entire BI lifecycle within one single environment. Report development and adhoc querying is extremely fast. Some of the new open source offerings make the same claim, but are they there yet?
Still, these products only work effectively if you can map to your data sources, which is why a data warehouse will be very important, regardless of what tools are used . As that database matures it becomes your single point of access to trusted data for BI.
The challenge many face going forward is providing "unstructured, free-form exploration and analysis [of data]... [for] real-time business questions." Right now, that means writing a lot of SQL.
Business Objects and MicroStrategy throw out of a lot of freebies and low priced alternatives to meet needs for 'canned' reports – but they also have tools that allow for unstructured data analysis. These tools basically generate SQL on the fly based on Meta Data definitions, but don't come cheap.
It will be interesting to see how things continue to develop in the industry.