Agile Business Intelligence and Data Lake Architecture – Part 2
Building a central data warehouse is a long and expensive process. At the end of this process we often find that the initial requirements were either not completely met, incorrectly specified or they had changed dramatically. In fact BI requirements are as fluid and volatile as the business itself. With our foundation in Custom Software development we are always looking for ways to improve the effectiveness and timeliness of BI projects. We have found that data warehouse projects are typically organized in a waterfall project approach that suffers from several issues:
- A very long development cycle for medium-large data warehouses on the order of 2-3 years to release usable data marts and systems.
- Data models are built that do not accommodate business environment and requirements changes. Needs are often identified once data is used leading to expensive changes and rework.
- Highly precise models require expensive data cleansing, exact ETL programs and careful tracking of upstream data sources.
In the Software Development world much progress has been made using Agile processes that allow projects to proceed in short iterations releasing usable systems frequently. Agile processes are more responsible to business changes.
This presentation outlines a new architecture based on the concept of Data Lakes – unstructured data landing areas that allow experimentation, analysis and creating usable data much more quickly than standard data warehouse methods.
Cas Apanowicz, founder and the first CEO of Infobright, is an accomplished IT consultant and entrepreneur. He has more than 18 years of experience in the field of Data Warehousing and Business Intelligence. Prior to Infobright, Mr. Apanowicz founded Cognitron Technology, which specialized in developing data mining tools, many of which were used in the health care field to assist in customer care and treatment.
Previous to Cognitron Technology, Mr. Apanowicz worked in the Research Centre at BC Tel where he developed an algorithm that measured customer satisfaction. At the same time, he was working in the Brain Center at UBC in Vancouver applying ground-breaking algorithms for brain reading interpretation. As well, he offered his expertise to Vancouver General Hospital in applying new technology for recognition of different types of epilepsy.
Throughout his career, Mr. Apanowicz has held consulting roles with Fortune 500 companies across North America, including Royal Bank of Canada, American Express, The Vancouver Stock Exchange, the Federal Government of Canada, Honda, The City of Toronto and many others. He holds a Master’s Degree in mathematics.
- Cas Apanowicz, The Data Lake – A New Solution to Business Intelligence