Look up the term Pandora’s box and you’ll find descriptions like “A present which seems valuable, but which in reality is a curse” or “An unexpected source of great troubles”.
The legend dates back to ancient Greece, but the history of big data is littered with examples of failed projects that reflect the Pandora euphemism.
There are a bunch of famous big data stories that often get touted to make this point. Mitt Romney’s “Orca System“ that did his campaign more harm than good is one such example.
The good news is that we have moved on, and most big data projects deliver incredible insights.
However, another Pandora’s box scenario awaits. As data analytics becomes more critical to a company, the data that gets stored and analysed can become increasingly subject to governance and compliance rules.
Your data scientists may be great in pulling information from data, but both they and the open-source tools they use to analyse data at scale do not have data compliance and governance built into their DNA.
For companies that build data platforms that do not have governance architected into them, the gift of big data insights can quickly become a curse. Eventually, they expand their systems to incorporate regulated data. Worse, some may find that they are already out of compliance and potentially already breaching governance regulations like PDPA or RMiT.
Cloudera saw this problem coming a long time ago. It provides a cloud data platform that leverages the open-source technologies of the Apache Foundation but supplemented with cast iron governance capabilities.
Cloudera SDX is a complete data governance solution for Hadoop. It layers capabilities such as data discovery, data audit, metadata management, and policy enforcement on top of the framework data scientists know and love.
Without these governance capabilities, companies are literally flying blind. You can’t govern what you can’t see. For instance, if you have no audit capability, you are very likely to fall foul of the regulators in the event they start asking you compliance questions.
Where Cloudera has cracked it, is to give the data scientists and analysts the tools they need and love. At the same time, it ensures those responsible for compliance and audit can rest easy knowing that they have complete visibility across the entire data platform with a unified, searchable audit log dashboard.
You can’t expect data scientists to build compliance into their systems. They need to focus on investigating your data; that’s why Cloudera has built the compliance functions for you.