There’s Gold in Them Hills! Mining Big Data for Value, Literacy and Sense
Think of big data as a rocky mountain, looming over your business, housing tons of valuable ore. You know the ore is there, but not how to find it. Mining big data for valuable and actionable small data poses a challenge for many SMBs and manufacturing companies, and there are many “mining packages” out there promising they are the best at data exploration and extraction. So, when it comes to approaching your mountain, how do you proceed? The best approach is the same approach a mining company utilizes to extract ore from a mountain: in well-planned phases.
Phase 1: Hire the experienced miner to help you establish your goals:
First, if you aren’t an experienced data miner, you need to start by hiring one, meaning, find the right consultant. Before you even begin the mining process, you need to clearly define what it is your company needs to know from big data. A consultant will guide you in asking the right questions, allowing you to better define your true business needs before you waste time exploring the wrong parts of the mountain. That big mountain contains more invaluable than valuable data, and the former can distract you from the true course. The best way to define your data mining goals is by first defining your business objectives. Once you’ve established your goals and objectives, your consultant can help you lay out a detailed mining plan, a blueprint so to speak, centered on finding the data that you truly need. Once you’ve hired your expert, you can proceed to phase two…
Phase 2: Begin your prospecting (or data exploration):
Without clear goals and objectives, it is easy “to embark on big data initiatives without a clear understanding of the business value it will bring”(1). Think of this phase as if you were exploring your mountain, looking for the “veins” that contain the most valuable deposits of data, the data you have targeted to help you fulfill your established needs. Data exploration, or prospecting, begins with “exploring a large set of unstructured data, looking for patterns, characteristics, or points of interest. Summarizing the size, accuracy and initial patterns in the data is key to enabling a deeper analysis” (2). This phase may sound daunting, but your consultant will employ key solutions designed for this very process. Once you’ve located where your “mineral deposits” are, it’s time to build.
Phase 3: Build the mine and extract the ore:
You cannot extract the valuable deposits if you don’t build the mine. There’s no easy way to say this: if you don’t invest in the structure and the software that can drill down and do the work, everything else is for naught. It’s just like mining for minerals. After a mineral deposit has been identified through exploration, one must make a considerable investment in mine development before production begins(3). It’s no different with your mountain of big data. Building a mine is akin to updating or upgrading your legacy system so it can deploy solutions that target and extract valuable data from the mountain. Yes, data extraction can be a costly investment, but luckily, your consultant will ensure your investment yields the best results. Once you’ve extracted your valuable data, you can enter the final phase.
Phase 4: “Smelt” your big data yields into small data:
“Smelting” your big data into a valuable commodity that will do the most for your business is the final step of the mining process. In mining, smelting uses “heat and a chemical reducing agent to decompose the ore, driving off other elements, such as gasses or slag, and leaving [only] the metal behind”(4). What a great word to translate into the technology world for extracting small data from big data. It’s time to make sense of the data you’ve invested in retrieving and “smelt” it into valuable, actionable intelligence. Think of your analytics software tools as the reducing agents that carry out this final process. All the unnecessary slag and gasses not needed from big data is funneled away, and only the valuable remains, your small data. Once smelted, you can package it into accessible, understandable, manageable, and actionable reports, metrics, and alerts for your company’s key decision makers and employees. But one thing to remember: one department’s “slag” is another’s treasure, so be careful what you discard. Analytics tools, such as Qlik, can assist in locating data that other query-based solutions often overlook, data known as grey data, and make sense of it to companies. But we’ll delve into grey data in a future discussion.
Overall, it is easy to fall into the buzz trap and embark on big data initiatives without a clear understanding of the business value they will bring.” (1) If organizations dig haphazardly into their mountain of big data, looking blindly for the good stuff, they can easily be buried in the rubble before striking it rich. For more information on how to develop a successful data mining program, partner with The Copley Consultant Group, and ensure your mountain of big data is transformed into actionable information vital to your company’s success.
- Marr, B. (2015, March 19). Why Only One of the 5 Vs of Big Data Really Matters. IBM Big Data & Analytics Hub
- Import.io. (2019, October 8). What is Exploratory Data Analysis and Why is it Important? How to use Data Exploration to Gain Insights for Your Organization.
- Committee on Technologies for the Mining Industries. (2002). Technologies in exploration, mining, and processing in evolutionary and revolutionary technologies for mining (pp. 19-45). National Academies Press.
- Lu, L., Pan, J., & Zhu, D. (2015). Quality requirements of iron ore for iron production. Iron Ore, 475-504.