The State of Dark Data

industry Leaders Reveal the Gap Between AI’s Potential and Today’s Data Reality

Data is our greatest asset — yet we’re lucky if we can find half of it.

Data shapes every facet of the organization. It inspires ideas, solves problems and makes money. In 2017, The Economist declared data to be the world’s most valuable resource, “the new oil.” Gartner says to treat data as an asset. Forrester calls it “the new currency of business.” For such a valuable asset, it’s surprising how much of it we lose track of.

Fifty-five percent of an organization’s data is “dark” — unquantified and untapped — according to new global research by TRUE Global Intelligence, sponsored by Splunk. The 1,300 business leaders surveyed recognize that data is key to success, now and into the future, yet very few say their organizations can successfully tap the value of all of their data. Or find it.

The power of “big data” and increasingly sophisticated analytics tools — the revolutionary promises of machine learning and artificial intelligence — rely on our ability to combine disparate data to uncover new insights. New customer segments. New efficiencies. Tougher security. Whole new lines of business.

We live in a data-saturated world. Billions of interconnected devices communicate with countless cloud services. We accumulate data from server log files, GPS networks, security tools, call records, web traffic and more. Every digital transaction, from back-end handoffs to the customer’s fingertips, is catalogued. Everything from the contents of a warehouse’s shelves to the temperature of our server rooms to the time and location of every login to our secure networks is recorded and stored … somewhere. Most of it, today, is unstructured, untagged, untapped. In a word, useless.

Out of the Darkness

Dark data may be the biggest untapped resource in business today. Dark data, which includes the “data exhaust” generated as a byproduct of our online lives, is all the unknown and untapped data across your organization, generated by systems, devices and interactions. Maybe it’s siloed off somewhere; maybe the format or metadata is inconsistent. Maybe no one’s figured out what to do with it. Maybe you literally don’t know it exists.

Most organizations struggle to capitalize on the full potential of their data. A full third of respondents report that more than 75 percent of their organization’s data is dark. Just 11 percent — one in nine — report that less than a quarter of their organization’s data is dark.

Inadequate processes, resources and technology hamper the intelligent use of dark data. Neglected by business and IT managers, dark data is an underused asset that demands a more sophisticated approach to how organizations collect, manage and analyze information. Yet respondents also voiced hesitance about diving in.

It’s a challenge, and an opportunity, of surprising scale. Machine data, a major source of dark data, is growing much faster than traditional organizational data, with an accelerating importance to decision making and organizational success. And because dark data can be a powerful fuel for artificial intelligence, organizations that fail to tap its power will also fail to keep up with, much less surpass, their competitors.

And competition is global. Our survey found considerable differences of opinion about data and artificial intelligence. Some European countries seem to lag behind a global average, while other nations, notably China, voice a much more forward-thinking approach.

Organizations must grasp the opportunities and confront the challenges of dark data — through greater strategic thinking, targeted technology investment, and more energetic and comprehensive skills training — to take full advantage of the next data revolution. Organizations need to think now about how to bring dark data into the light.