Databricks, the Data and AI company and pioneer of the data lakehouse paradigm, announced the general availability of Delta Live Tables (DLT), the first ETL framework to use a simple declarative approach to build reliable data pipelines and to automatically manage data infrastructure at scale. Often, converting SQL queries into production ETL pipelines necessitates a great deal of time-consuming, sophisticated operational effort. Data engineers and analysts can focus on delivering data rather than operating and maintaining pipelines by employing current software engineering principles to automate the most time-consuming portions of data engineering.

Many organisations will engage expensive, highly trained data engineers - a resource that is already scarce - to avoid delays and failed projects as they design methods to extract the most value from their data. What's less commonly known is that a key issue is at the root of many of the delays and project failures: it's difficult to design trustworthy data pipelines that work automatically without a lot of operational rigour to keep them going. As a result, even on a modest scale, the majority of a data practitioner's work is spent on tools and infrastructure management to ensure that these data pipelines don't fail.

Delta Live Tables is the first and only ETL framework to solve this problem by combining both modern engineering practices and automatic management of infrastructure, whereas past efforts in the market have only tackled one aspect or the other. It simplifies ETL development by allowing engineers to simply describe the outcomes of data transformations. Delta Live Tables then understands the full data pipeline's dependencies and automates away virtually all of the manual complexity. It also enables data engineers to treat their data as code and apply modern software engineering best practices like testing, error-handling, monitoring, and documentation to more easily deploy reliable pipelines at scale. Delta Live Tables fully supports Python and SQL and is tailored to work with streaming and batch workloads. 

Delta Live Tables is already powering production use cases at leading companies around the globe like JLL, Shell, Jumbo, Bread Finance, and ADP. "At ADP, we are migrating our human resource management data to an integrated data store on the lakehouse. Delta Live Tables has helped our team build in quality controls, and because of the declarative APIs, support for batch and real-time using only SQL, it has enabled our team to save time and effort in managing our data," said Jack Berkowitz, Chief Data Officer, ADP. 

“The power of DLT comes from something no one else can do - combine modern software engineering practices and automatically manage infrastructure. It’s game-changing technology that will allow data engineers and analysts to be more productive than ever,” said Ali Ghodsi, CEO and Co-Founder at Databricks. “It also broadens Databricks’ reach; DLT supports any type of data workload with a single API, eliminating the need for advanced data engineering skills.”