There’s considerable discrepancy around the volume of data that is unused, unstructured, and hidden (aka “dark data).” Experts such as IDC and Gartner think 50-90% of the data that organizations have is dark, while IT leaders polled late last year by IDG* say only 25% of it is.

Despite their belief that they have relatively little dark data, IT executives have big concerns about it, including: the impact on quality, security, and risk compliance; loss of control; lost business value; and the inhibition of data integration projects.

So, what is dark data? Gartner defines it as such: “The information assets that organizations collect, process, and store during regular business activities but generally fail to use for other purposes.”

Here’s a shortlist of dark data and the common places it lives:

• Documents. Finance personnel often pull together expense or sales information from different sources, for instance, then do calculations on it inside spreadsheet docs. That number-crunching has no value if it’s sitting on someone’s desktop. The same goes for Word documents, forms, PDFs, and presentations. These files are often overlooked because it requires work to uncover what’s inside them, or there are misconceptions as to their value.
  
• Data repositories. Lots of data—structured and unstructured—gets dumped into data warehouses, lakes, and non-relational databases. These repositories often hold old records such as customer, employee, or financial data that must be kept for compliance reasons yet incur considerable storage costs. IT teams may struggle with the complexity involved to connect BI or analytics solutions to these repositories, and sometimes can only run analysis on individual silos—like the sales or customer relationship management (CRM) app. Plus, they typically require vault-like access with few individuals holding the keys.
  
• Applications. Consider how many emails contain attachments and where those files go. Then think about all the sensor data collected by Internet of Things (IoT) applications; audio and image files sitting in social media apps; video surveillance; customer call records; and log files. The Slack app probably contains great employee insights, but is any of that data captured—let alone analyzed—to improve user experiences or tap business opportunities?

Mine dark data for business value

Pep Boys is a good example of what uncovering dark data can yield. The automotive retail and service chain wondered why some of its stores were underperforming. So, it integrated PBX system data into the Domo platform and analyzed how many rings it took before a service rep answered a call. It quickly became apparent that better-performing stores had faster call pickup times—and hence a greater probability of customers booking service appointments.

CIO Jarrod Phipps said his team never would have thought that PBX data had any value until they saw it in action. That one small project built data confidence and curiosity throughout the group.

“Now, we’re unshackled,” he said. “IT is less of a constraint, the business can operate more independently, and my team has probably 100 times the capabilities it had before. Nothing is off the table anymore.”

Next steps

No organization can solve its dark data problem all at once. Start by asking simple questions like: How do we arrive at our sales figures? What are the sources of all that sales data? Then, get going with a small pilot project using a modern BI tool like Domo.

“The first day, we hooked up the connector and we had data up in production,” Phipps said. “The next day, we took two years of historical data and validated it. And within two days, we not only had dashboards, but we actually had data assets that we had never contemplated being in a BI platform, because it’s so hard to manage some of these old platforms.”

To discover how to shine a light on dark data hidden in unexpected places, get Domo’s Cloud Integration for Analytics report. To learn more about Domo’s cloud integration capabilities, click here.

* MarketPulse Research, Data Integration and Exploration, December 2020

NOTE: This post was written by a representative of International Data Group, Inc. (IDG) and originally appeared on CIO.com as part of a Domo-sponsored marketing campaign.