Authored by: Ravi Shankar, senior vice president and chief marketing officer at Denodo
Business organisations have had to accelerate and strategise to identify, capture and optimise the use of data in business decision-making in tandem with the rise of the digital economy. The current global health crisis has proven to be a catalyst in putting data front and centre in the way businesses operate today, and processes and systems are fast streamlining and becoming data-dependent, data-driven.
However, with valuable data existing in varied formats and quality levels, the rise of data sprawl – the overwhelming amount and variety of data produced – has posed a real challenge for many organisations. That is, organisations often struggle to efficiently convert and make full potential use of their data, stuck in disparate silos, and turning it into intelligible insights. According to a recent IDC report, more than 80% of IT leaders reported data sprawl as one of the most critical problems they face today.
Nonetheless, data management has evolved significantly since the 1970s, when we first stored and accessed data by interpreting it in different spindles and punch cards. Here, I introduce the concept of logical data fabrics as the modern solution to data sprawl. The technique that allows data to be more efficiently converted and used for business purposes.
Logical Data Fabric: Knitting a virtual view of data
Computer scientists have created centralised repositories for storing data, from the concept of data warehouses in the 1990s to data lakes in the past decade. Mainly, for business organisations to achieve a unified view of information across applications and databases.
This approach, while attempting to go against data gravity – which pulls the data towards its sources and away from central repository – works only for some period of time. Computer scientists realised that data quickly moved apart into decentralised components as new data continuously sprouted in other locations.
Rather than forcefully consolidating the data into a central physical repository, a new data management paradigm called logical data fabric has emerged. Much akin to the weaving of textile fibres to produce cloth, logical data fabric knits a virtual view of data across applications by leaving data at the sources where they are created, while allowing a unified view of all enterprise data.
What makes the data fabric successful is that data virtualisation forms its core technology and many of its capabilities are automated through Artificial Intelligence (AI) and Machine Learning (ML).
The concept of a data fabric has increased in importance as organisations increasingly store data in multiple cloud-based platforms, which can add to existing on-premises data silo problems.
Stop collecting and start connecting: Why Logical Data Fabric is a better solution
Specifically, logical data fabric enables organisations to stop ‘collecting’ their data into a central repository and start ‘connecting’ to the data at the sources through data virtualisation. Regardless of the data’s location – on-premises or in the cloud; format – structured or unstructured; or latency – data in motion or data in rest, data fabrics free up the data to be innovated at its sources while bringing it together in a virtual fashion for the benefits of data discovery, management, and governance. It hence operates according to the concept of data gravity.
This way, logical data fabric greatly improves the efficacy of business data users. For one, without needing to move data from its source into a temporary repository, IT teams no longer need to program ETL scripts for the purposes of converting data before loading it into target systems. Data virtualisation can perform transformations on the fly, which saves on storage costs.
Furthermore, data virtualisation’s low- to no-code approach significantly reduces the manpower and effort involved to technologically develop the unified views that organisations demand.
More importantly, it serves as a virtual catalogue for all data assets within the enterprise, including the sources of data origins, its format, and its relationship and association with other data assets. Through this catalogue, business users are not required to access different systems to perform actions such as data discovery or documenting business definitions for the purposes of data governance.
In addition, logical data fabric possesses powerful data preparation capabilities to normalise data formats and simplify them for business consumption. Business users will then be able to easily access the data within their favourite analytical, operational, web, or mobile applications.
The future of Logical Data Fabrics? Embracing AI/ML
As mentioned above, logical data fabric is already embracing AI and ML to automate some routine tasks, and this is set to intensify moving forward. At most business organisations, data is constantly evolving, new data sources are routinely added, and new forms of data are being innovated. AI and ML analyse changing data patterns and automatically integrate new data for unified views, delivering them in the most appropriate formats to business users. The incorporation of AI and ML technologies allow organisations to better understand consumption behaviours of certain users and share the availability of new data sets for further analysis and collaboration.
There is no doubt that the benefits of AI and ML include raising efficiency and productivity levels among data scientists working on related projects. These data fabrics will enable them to optimise data models for deeper analytics and insights that benefit the organisations, thereby improving their performance.
Data Fabrics are already in use with more to come
The logical data fabric is no longer just a concept. Their adoption has been gaining speed, thanks to the tremendous benefits outlined above. By adhering to the principles of data gravity, providing business friendly views of enterprise data in its entirety, and automating using AI and ML, data fabrics will be one of the hottest data management trends in 2020 and beyond.