Is Data Quality Dead?


Winston Chen

At the Gartner MDM Summit this spring, Ted Friedman pronounced that the Data Quality Magic Quadrant would probably disappear soon. (I’m sure an analyst of Ted’s caliber will find a new Magic Quadrant to own.) Recently, I was on the panel of the TDWI MDM/Data Governance/DQ Super Webinar. Another panelist and a respected figure in data management expressed a similar sentiment. Is data quality dead?

Of course not. If anything, data quality is more important than ever. As I wrote recently, data quality is the key to taming and harnessing Big Data for insights. But data quality software as a market segment is changing in a big way, which may result in the Magic Quadrant going away.

Data Quality is a gigantic topic that encompasses a wide array of technologies and business practices. Within that big space, the scope of data quality software has always been a little murky. A data quality software suite is a loose and uneasy bundle of specialist tools that meet a diverse set of needs. It’s hard to see how data profiling relates to name and address standardization. Fuzzy matching and deduping tools have yet another set of use cases. In spite of efforts to generalize, most data quality tools remain highly data domain specific – most of which are for party data. The notion of an enterprise data quality platform has not seen broad adoption.

From the data quality tool bundle, some pieces have already been absorbed by MDM. CDI is basically a fuzzy matching engine on top of a repository with a GUI wrapped around it. But CDI is the sexier outfit and while the match engine is the brain. Now the mega vendors all have MDM and data quality tools in their stacks, they’ll integrate more DQ capabilities like name and address cleansing into MDM, not the other way around. DQ tools’ domain orientation aligns with MDM products’ domain orientation.

Data governance can offer a few other pieces of the DQ bundle new glory. DQ monitoring is an essential component of data governance: you need to measure data against a pre-defined set of rules and policies. Ditto for DQ dashboards.

It’s no surprise that DataFlux is moving aggressively beyond traditional DQ to MDM, data integration, and data governance. It’s no surprise that DQ products all but disappeared in the SAP and IBM stacks in market visibility. But these by no means are indications that DQ technology is not important. Acquisitions of Silver Creek and Netrics this year shows that DQ technology continues to be highly coveted. DQ technology isn’t going away; instead, it’s getting unbundled and then integrated deep into other solutions and the infrastructure for data.

This is a good thing, because quality should be a part of everything we do in data management.

Tags:

2 Responses to “Is Data Quality Dead?”

  1. Matt Reichert October 5, 2010 at 6:02 am #

    I could not agree more. The days of attacking DQ with point solutions is going by the wayside. Going forward companies will need to think about MDM solutions that have three components, core high quality master files with SLA’s to measure quality, technology that can build and maintain custom databases that are predicated on customers unique business rules, and domain data expertise to manage these solutions and maintain the data quality SLA. Further these MDM solutions will need to operate in a transaction based world. The market trends suggest that most companies do not want to take this on as it is difficult and not their core competency. They would rather work with information (result of the MDM process) and have suppliers build/ operate MDM platforms that can aggregate all of the data into a usable format as there are firms that can provide these solutions cheaper and with higher quality. In the future companies will be purchasing a set of DQ SLA’s from a third party MDM provider and consuming the outputs to gleen information for managing their business.

    • Winston Chen
      Winston Chen October 7, 2010 at 6:39 pm #

      Thanks Matt for your comment. Outsourced MDM is definitely worth considering for certain data domains. I’m think about how much money every pharma spends maintaining physician master. Mostly the same physicians!

Leave a Reply