Why speed counts in data warehouse development

Automation provides the key to data warehousing success

Why speed counts in data warehouse development

Remember the old saying in software development (and probably plenty of other fields besides): ‘fast, cheap or good. Pick any two?’

When it comes to data warehousing, it applies, but only sort of. That’s because in order for a data warehouse to succeed, its development has to be fast for it to be good. And to accelerate development, you need to automate it.

Whether or not it’s cheap is open to interpretation, but if you get the first two right, the results will justify cost in the best possible way. By delivering value faster.

It may seem obvious, but let’s delve into the reasons why speed counts in data warehouse development. Firstly, this is without doubt an enterprise software project. But it isn’t like other enterprise software projects. Where a careful, steady as she goes and considered approach is necessary for, say, an ERP implementation, taking this path is likely to drive up risk when applied to a data warehouse.

That’s because the similarities end at ‘enterprise software project’.

Data warehousing and the analytics which will in due course rest atop of it do not work off a well-defined set of upfront outcomes. ‘You don’t know what you don’t know’ and the point of most analytical efforts is to change this status quo.

A rapid, iterative approach in which small amounts of exploratory work are rapidly developed, tested, released and reviewed, allows for teasing out the value. If something works, great. If it turns out it doesn’t, no problem – adjust your plans and try again. The amount of effort expended is minimal, so in the latter case, nobody gets fired and rolling it back doesn’t present major cost implications.

That’s one aspect. While all this is happening, there’s another. When development happens rapidly on the data warehouse, you’re putting value in front of the business faster. Putting it simply, people can start using the data warehouse sooner.

Take forever getting something useful back in front of the business users, and the opposite happens: the business disengages, loses interest and by the time you deliver something you think they might want, they’ve all long moved on to wanting something else entirely.

This leaves you stuck with a monolith into which a ton of money and effort has already gone. You’re now faced with the ‘Concorde fallacy’ (after the British-French jet airliner…more prosaically known as a ‘sunk cost’). It might now be obvious that the data warehouse isn’t going to work, but you have to keep throwing money at it just because so much has gone in already.

But when you do take on a data warehouse using the power of automation – you get the ability to create working solutions that your users can get their teeth into in days or weeks. And as those users take a few bites, they’re going to do that most rewarding of things, and start asking for additional helpings. As they see what the data warehouse can do, the requests for more will start coming thick and fast.

In an almost organic fashion, the ability to rapidly create, iterate, change and improve, establishes a sort of virtuous cycle. The analytics users will involve themselves in the development process by becoming part of the feedback loop, because they can see exciting things happening from the data warehouse. Exciting things that help them do their jobs better.

To stay useful the data warehouse needs to stay relevant, and that means keeping up with the demands of the business.  These are quick to change and the data project that can’t keep up will struggle to deliver value.  So speed is essential to success.

That’s great, but how do you automate a data warehouse? WhereScape RED, the world leading data warehouse automation software, provides data teams with what they need to keep up with the business.

With WhereScape RED, 95% of data warehouse code is automated

This is not a mis-print. What the clever people at WhereScape realised is that, once a few decisions have been made and preferences set, the code used to build a data warehouse is much the same from one data warehouse to another. They built a best practice framework and an interface to let you select those preferences and make those decisions.  Most of the code is software generated. The net result, speed.

Frameworks promote collaboration

The same frameworks that the automation is built around provide other benefits. They provide a playbook, a playbook that provides a common approach and understanding of how the data warehouse will be built. This means everyone is on the same page and working in the same direction. Without a common framework, data teams can spend a lot of time in meetings figuring all this out for themselves (or worse, fixing issues created through a lack of  communication and coordination). With WhereScape, it’s already done and ready to use.

Automation and frameworks are at the very core of what WhereScape RED is all about. It does the repetitive, ‘by the book’ stuff which is essential for a ‘technically’ good data warehouse and it significantly reduces the time and effort it takes to get data teams working well together. That leaves people (developers and line-of-business folk) free to do the innovative, clever stuff which is essential to a ‘valuable’ data warehouse.

Speed counts, in other words, because it makes the data warehouse work for the business.

Related articles & projects
RED does things that you simply will not get from traditional ETL. It's the only tool you need.
Practice Areas
Data Integration
Whether it is an on premises, cloud or a hybrid data warehouse, WhereScape RED is a complete data warehouse life-cycle management tool.
Practice Areas
Data Integration Cloud Services
Xero used the power of automation to accelerate their AWS Redshift cloud migration project, helping them reduce costs and get more from their data, faster
Practice Areas
Data Integration Cloud Services
Industry
Finance
Dental group moves to Azure in just 30 days using NOW Consulting & WhereScape RED
Practice Areas
Data Integration Analytics and visualisation Cloud Services
Industry
Retail Health