Dirty data is a sign of a broken business process
There it is. The data. It sits there in front of you, with known origin and with the precision of three decimals. Ready to be acted on. It surely must be accurate and trustworthy! Right?
It might come as a surprise, but data does not appear out of thin air. It is created by people and machines. Even in the best of worlds people enter faulty data in their IT-systems. It could be due to mistakes, inaccurate or outdated instructions, trying to outsmart a quirky user-interface or just out of old-habit. And hardware could break or be improperly configured.
In short, if you have dirty data - data that is of bad quality or unusable for its purpose - the only reason is because either individuals made errors or machines produce inaccurate readings.
Dirty data threatens to undermine critical business processes that uses it. For example, it can trigger a breakdown in the supply chain, with inventory data indicating plenty of stock even though the shelves are empty, and it can lead to inferior customer services and customer defection after personal details are incorrectly recorded or duplicated. In the supply chain case, even the most luxurious robotic warehouse solution will grind to an embarrassing halt if the SKU-shelf is empty where the robot expects to pick-up an unit.
What if every invoice you send is actually accepted by the customer instead of being rejected due to wrong project codes?In these cases, the generation, governance and maintenance of data is frankly inadequate. Your business process is broken.
Dirty data creates problems everywhere
Since data is used "everywhere" dirty data is Everyone’s problem. Inadequate business processes usually do not create dirty data problems in just one place; they create issues in multiple places. From the individual efforts’ perspective, the cost may seem minor, but since the same problem has very likely been fixed in several other efforts in several other locations, the cost quickly ramp up. If the root cause is not addressed, the issue will resurface again and again, requiring either recurring quick fixes or implementing a mending process wherever the issue surfaces.
One example would be the BI-process. Do you think having a mending process put in place by skilled BI-consultants that takes care of every oddity that any employee could think of to enter in the IT-systems is the best use of your highly strained BI-team? Think again.
As individual data points are amassed throughout the enterprise, any small data oddities get aggregated away from the operative process and land in an analysis tool aimed at improving decisions. Depending on the nature of these flawed data points, you will have anything from a Small to a Major headache when you make business decisions based on faulty data.
You need to be able to trust your data
At an aggregated level, the key aspect is that the data You see can be trusted. Without trust it cannot be acted upon. If some data cannot be trusted, then the next question is: what other data cannot be trusted? Distrust then spreads. In the end the plague of data distrust cause too few decisions to be made or take too long time, causing crippled management and loss of business to quicker competition.
The fix – there are no shortcuts!
Solving the problem with dirty data must be seen as fixing the problem at the origin - correcting the business process. Establishing a continuous data improvement process is a allowing the process to heal. Opposite, doing one-time efforts is instead just treating the symptoms - handing out a painkiller pill to someone who has their leg in two pieces.
Granted, the continuous improvement process is slower and long-term, but it is in the end much cheaper. It is a waste of company resources to solve the same problem again and again. And again. And again.
Ironically, the quick fix result in the slowest result.
A continuous data improvement will frequently feedback any found issues to the operative systems and -processes; it will allow corrections to be done at the source of the problem resulting in that everyone does minor data corrections throughout the day that in the end benefit both the operative process as well as data consumers down the line. We begin to establish a culture of continuous improvement that move mountains!
How do you know if you have dirty data? Start by asking the right questions, turning one stone at a time. Do you need help establishing a continuous data improvement process? We are here to help!
Magnus Hagdahl works as an information management consultant at Enfo