We are living in an era that is technology driven and data is our competitive asset. It is present as raw information in heavy volumes by transaction system, search engines, social media and many more. Once analysed, this data can validate assumptions and help in decision-making.
According to business context, ‘dark’ means hidden or undigested. Dark analytics involves text-based data that is not scrutinized. It includes messages, documents, email, video and audio files etc. Dark analytics may also include exploration of inaccessible sites called the ‘dark web’ although it is impossible to calculate the deep web size but it is estimated to be 500 times more than the surface web.
It is estimated that 90% of present data was generated in the past five years. It clearly indicates that data is increasing at a very high rate. By the year 2020, the digital data will reach 44 zettabytes (44 trillion gigabytes). It is almost equal to as many bits as stars in the universe. The following figure explains this better:
The crucial aspect is that dark data should not remain hidden or unexplored. The moment dark data is taken to gain insights; it is converted into actionable data. Let us consider the following examples:
1) NETWORKING MACHINE DATA: A large amount of machine data is generated from firewalls, servers and network tools, etc. This information can be used to analyze and monitor network security and avoid dark data. It can also help in analyzing utilization of your network infrastructure.
2) CUSTOMER SUPPORT LOGS: Most businesses maintain their customer data that includes information such as customer contact data, communication channel used, duration of engagement, etc. This data is mistakenly left as dark data. Instead, it can be used to build analytics workflows to understand customer insights.
3) ‘LEGACY’ SYSTEM LOG: With the help of modern analytical tools, we can also scrutinize the old types of systems or mainframe. We can offload this data into an analytical tool such as Hadoop, thus not leaving any legacy data in the dark.
4) NON- TEXTUAL DATA: As mentioned earlier, data analytics can also be used to analyze audio and video files or other non- textual data. Moreover, audio files can be translated into text to gain more insights. The aim is that non- textual data should not remain in dark form. There are actionable ways to use it.
One should follow a strategy to discover value in unstructured data and help the organization by generating insights and greater opportunities in the future. To optimize dark data, following steps should be followed:
1) ASK THE RIGHT QUESTIONS: Specific questions should be identified and answered. For example: To boost equipment sales, analysts should focus on sales transaction, inventory and pricing in a particular geographic area.
2) LOOK OUTSIDE OF YOUR ORGANISATION: Data should be augmented with publically available information. For example: A physician recommending short-term solutions to an asthma patient like protection from pollen grains, etc.
3) BROAD THINKING: Looking for new capabilities and strategic implementations for using dark analytics for the betterment of organization, customers, business partners and vendors.
Dark data provides an opportunity to enlighten the hidden pattern and correlation into powerful insights. It leads to reduced risk, increased analytics ROI and innovative opportunities.
– Tanish Khandelwal (2017-2019)