Agile BI and the data warehouse

Agile software development based on iterative and incremental development is now widely practiced and has proven to deliver rapid and flexible response to change. Little wonder then that the methodology is increasingly being applied to any situation where complex solutions are required to solve a dynamic problem, Business Intelligence being a perfect example.

Like agile software development, agile business intelligence is all about small, iterative and rapid change to solve complex problems. With that in mind, the data warehouse presents some very real challenges. By its very nature, any change must wait for an hourly, overnight or weekly batch run. As such any data warehouse enforces an environment where the most fundamental requirement for agile development, the ability to make and test rapid changes, is impossible. This is as true for an in memory data warehouse as a disk resident data warehouse. The in memory data warehouse is all about increased performance when reading from the data warehouse, but still requires the same hourly, daily or weekly batch runs to populate.

We did however build data warehouses for good reason. The extract and transformation of huge datasets has always been too costly to execute in real time, and with the advent of big data this problem is not getting any easier to solve. However, is it not time to accept that the data warehouse model is fundamentally flawed, and to look for innovative solutions to move past it? Clearly there will always be some situations where the volume of data or complexity of the transformation dictates that live execution is just not practical, but with the exponential growth in processing power and memory storage keeping pace with the exponential growth in data, surely these situations should be the exception rather than the rule. Why not focus our efforts on engineering solutions capable of extracting data from live operational business applications, and transforming that data into actionable intelligence in real time? Such solutions will always be capable of reading data from a data warehouse in the minority of cases where this is unavoidable, after all a data warehouse is just another database, but will no longer slavishly enforce all data is passed through a single monolithic storage area. Only by changing our perception that business intelligence starts and finishes with the engineering of a data warehouse can we achieve the goal of true agile business intelligence, capable of answering all our questions in an increasingly dynamic business environment.