Big Data has been all the rage for the last couple of years as companies try to figure out how to mine all of the bits and bytes that are captured and stored from their business processes.

While there are many rewards that can come from this work there are an equal number of challenges in getting an ROI on your Big Data investment.  Organizations need to:

• Ensure they are capturing the “right” details,
• Confirm that their conclusions are accurate, and
• Create the ability to incorporate their findings back into business processes.

To be certain there are incredible insights that can be gleaned from all of this data.  Mining sales information across our retail and online channels for insights was part of my job for a couple of years while working at Best Buy.  During this time I recognized there were other ways to leverage our data that could be more cost effective.  We could glean insights just by paying more attention to our failed projects – we could learn from our Dark Data.

Dark Data Can Lead To Big Insights

I had first heard about the concept of Dark Data from a Wired magazine article in 2007.  The author, Thomas Goetz (@tgoetz), explained how organizations that routinely discarded data from their failed experiments or projects might be able to learn more from these failures if they stored them for future analysis.

Goetz described how a 1981 New England Journal of Medicine study had mistakenly linked drinking coffee with pancreatic cancer.  It ended up taking twenty years for the medical community to refute this claim because scientists needed to completely recreate a study in order to disprove the hypothesis.  The kicker is that there were likely many previous studies that had already proven that there was no link between coffee and pancreatic cancer but because that was not the conclusion they were looking for the results were thrown away.  Most studies at the time were looking for links with smoking and cancer or alcohol and cancer.  In science only positive correlations see the light of day and everything else gets discarded.  Similarly in business, only successful projects get talked about.

Thrown Out With the Trash

A company may have spent thousands or even millions of dollars on their latest innovation initiative.  If that initiative fails few people within the organization will know the details of why.  During my tenure at Best Buy I had launched an initiative to try and address this issue.  We too had launched many innovation projects with only a handful of successes.  To share the lessons learned from our errors I had launch a series called the Failure Forums.  Project leaders would prepare and present three things:

1. What had the group accomplished – a chance to share their wins,
2. What had the group learned – capturing the key takeaways, and
3. What would they have done differently?

Each Failure Forum would end with a Q&A session where others could ask questions that pertained to their projects.  We wanted to shine a light on our failures so that we could learn from them, not just throw them out with the trash.

Start With a Little Support and a Lot of Courage

Tapping into Dark Data doesn’t need to just come from archiving failed scientific experiments.  It can come from capturing failure in many parts of the organization.  While it would be ideal if every CEO embraced the concept of learning from our failures, we know that it is not reality.  Most leaders don’t reach the c-suite based on their long list of failures.  They might get there in spite of their failures but not because of them.  But tools like hosting Failure Forums can be utilized by groups as small as a team and still be effective.  All you need is a supportive leader and the courage to be vulnerable.

Other interesting articles on the value of Dark Data:

• University of Texas – Dark Data Needs an Advocate
• Gartner – Hidden Value in Hidden Data
• Information Age – Analysts urge IT leaders to deal with “dark data”

Food For Thought:

• Are you archiving data from your failed projects?
• Is your organization purposely trying to learn from failed projects?
• What is preventing you from taking the first step in prompting your team to learn from failures?