The less things change, the more they stay the same


John Evans

In 2012, we surveyed attendees at TDWI World Conferences regarding topics related to data warehousing: how they handle change, how long it takes to deploy new data, costs associated with supporting the data warehouse and so on. Over this period we received well over 500 responses and have blogged about it and shared information in other places such as infographics on our website.

The 2013 Teradata Partners Conference was billed as the largest data warehousing conference in the world. Since it had been about one year since we collected data I thought it would be worth a look-see to find out if the results were different.

I’m sorry to report that not much has changed.

One of the questions we asked was how long it would take to integrate a new source of data into the warehouse. The responses from the TDWI set were similar to what the Teradata customers said: 26.4% vs. 37.5% in the survey of TDWI attendees said they could do it in 1-4 weeks, which is interesting that the Teradata response is actually better than the TDWI responses. 34.4% at Teradata Partners and 35.5% at TDWI said between 1-3 months. 27.3% of TDWI and 25.0% of Teradata responders said it took over 3 months to integrate a new source. 10.7% of TDWI responses said they could integrate data in less than a week, but only 3.1% of Teradata customers can. The results are generally similar, so I interpret this as not much having changed. There has certainly not been a big shift towards it getting faster to respond.

But the more stunning result was the difference in the area of cost and resources to maintain the data warehouse.

Teradata Partners attendee survey question on cost

Teradata Partners attendee survey question on cost

55.6% of the Teradata respondents said they spent over $1 million annually to maintain their data warehouse. This is more than twice the response rate we saw from our TDWI respondents, only 25.9% of whom spend over $1 million. However, 54.6% of the Teradata respondents said they require more than 20 resources to support it, whereas only 32.7% of TDWI respondents required that many. The number of resources to support the warehouse is a definite driver of cost, but something else is driving it higher with Teradata. It might be that Teradata warehouses tend to be larger than average and therefore cost more, or it might be an indication of relative complexity. Whatever the reason, these results aren’t particularly promising for data warehousing in general. We are still spending too much money and putting too many resources into building and maintaining the data warehouse than we need to.

The emergence of a class of tools – including Kalido – that automate many data warehouse deployment and maintenance processes offer a promise of reducing the time, effort and resources required. As an example, Kalido has a Teradata customer who deployed 4 subject areas in only 9 months – quite impressive when you consider it can take twice as long for a single subject area using a traditional approach.

Perhaps the winds of change are starting to pick up. For their part, Teradata recognize the value of this automation approach, and last year entered into a global reseller agreement with Kalido. The growth of a data warehouse automation tools market would be good for customers and good for the industry.

Tags: , , , , , ,

3 Responses to “The less things change, the more they stay the same”

  1. Steve Sydney January 10, 2014 at 12:45 am #

    Glad to see Kalido is trying to keep up with WhereScape

    • John Evans
      John Evans January 12, 2014 at 7:07 pm #

      Hey Steve,
      Kalido and Wherescape both advocate an automated approach to data warehousing. Our approaches and coverage are somewhat different but many of the concepts are the same. My post is not about competing with Wherescape which I imply from your response, but more about illustrating a way to break the logjam of ETL-tool-driven warehouses. There’s plenty of room in the market for both of us. Thanks for reading!

Trackbacks/Pingbacks

  1. What’s The Real Burden in ETL? | Kalido Conversations - January 16, 2014

    […] real money is spent. Our survey of TDWI Conference attendees further showed that nearly 26% – and sometimes more than that - spend more than one million dollars per year to support their data warehouse. In Step 2 in the […]

Leave a Reply