Delta Lake

About Delta Lake

Delta Lake is an open-source storage framework that provides ACID transactions and a unified format for data lake tables. It offers a family of libraries for reading and writing Delta tables from different engines and languages, including Delta Spark for Apache Spark, Delta Kernel for building connectors without needing to understand the underlying Delta protocol, Delta Rust for low-level access from Rust (with Python bindings) for use with frameworks like DataFusion and Ballista, and Delta Flink for reading and writing Delta tables from Apache Flink applications. Delta Standalone, a JVM library for reading and writing Delta tables without Spark, is deprecated in favor of Delta Kernel.

The project is aimed at developers building data processing applications or connectors that need to interact with Delta tables, whether through a distributed engine like Spark, Flink, or Trino, or directly from custom applications. API documentation is provided in Scala, Java, and Python depending on the library.

Key features

• Reads and writes Delta tables using Apache Spark via Delta Spark
• Delta Kernel provides simple APIs for reading and writing Delta tables without needing to understand the Delta protocol
• Rust library (with Python bindings) for low-level access to Delta tables
• Delta Standalone JVM library reads and writes Delta tables without requiring a Spark cluster
• Delta Flink connector reads and writes data from Apache Flink applications to Delta tables

Frequently asked questions