This is the full developer documentation for Delta Lake # Welcome to the Delta Lake documentation > Learn how to use Delta Lake [Delta Lake](https://delta.io) is an [open source project](https://github.com/delta-io/delta) that enables building a [Lakehouse architecture](https://www.databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html) on top of [data lakes](https://www.databricks.com/discover/data-lakes). Delta Lake provides [ACID transactions](/concurrency-control), scalable metadata handling, and unifies [streaming](/delta-streaming) and [batch](/delta-batch) data processing on top of existing data lakes, such as S3, ADLS, GCS, and HDFS. Specifically, Delta Lake offers: * [ACID transactions](/concurrency-control) on Spark: Serializable isolation levels ensure that readers never see inconsistent data. * Scalable metadata handling: Leverages Spark distributed processing power to handle all the metadata for petabyte-scale tables with billions of files at ease. * [Streaming](/delta-streaming) and [batch](/delta-batch) unification: A table in Delta Lake is a batch table as well as a streaming source and sink. Streaming data ingest, batch historic backfill, interactive queries all just work out of the box. * Schema enforcement: Automatically handles schema variations to prevent insertion of bad records during ingestion. * [Time travel](/delta-batch#query-an-older-snapshot-of-a-table-time-travel): Data versioning enables rollbacks, full historical audit trails, and reproducible machine learning experiments. * [Upserts](/delta-update#upsert-into-a-table-using-merge) and [deletes](/delta-update#delete-from-a-table): Supports merge, update and delete operations to enable complex use cases like change-data-capture, slowly-changing-dimension (SCD) operations, streaming upserts, and so on. * Vibrant connector ecosystem: Delta Lake has connectors read and write Delta tables from various data processing engines like Apache Spark, Apache Flink, Apache Hive, Apache Trino, AWS Athena, and more. To get started follow the [quickstart guide](/quick-start) to learn how to use Delta Lake with Apache Spark. # Best practices > Learn best practices when using Delta Lake. ## Choose the right partition column [Section titled “Choose the right partition column”](#choose-the-right-partition-column) You can partition a Delta table by a column. The most commonly used partition column is `date`. Follow these two rules of thumb for deciding on what column to partition by: * If the cardinality of a column will be very high, do not use that column for partitioning. For example, if you partition by a column `userId` and if there can be 1M distinct user IDs, then that is a bad partitioning strategy. * Amount of data in each partition: You can partition by a column if you expect data in that partition to be at least 1 GB. ## Compact files [Section titled “Compact files”](#compact-files) If you continuously write data to a Delta table, it will over time accumulate a large number of files, especially if you add data in small batches. This can have an adverse effect on the efficiency of table reads, and it can also affect the performance of your file system. Ideally, a large number of small files should be rewritten into a smaller number of larger files on a regular basis. This is known as compaction. You can compact a table by repartitioning it to smaller number of files. In addition, you can specify the option `dataChange` to be `false` indicates that the operation does not change the data, only rearranges the data layout. This would ensure that other concurrent operations are minimally affected due to this compaction operation. For example, you can compact a table into 16 files: * Scala ```scala val path = "..." val numFiles = 16 spark.read .format("delta") .load(path) .repartition(numFiles) .write .option("dataChange", "false") .format("delta") .mode("overwrite") .save(path) ``` * Python ```python path = "..." numFiles = 16 (spark.read .format("delta") .load(path) .repartition(numFiles) .write .option("dataChange", "false") .format("delta") .mode("overwrite") .save(path)) ``` If your table is partitioned and you want to repartition just one partition based on a predicate, you can read only the partition using `where` and write back to that using `replaceWhere`: * Scala ```scala val path = "..." val partition = "year = '2019'" val numFilesPerPartition = 16 spark.read .format("delta") .load(path) .where(partition) .repartition(numFilesPerPartition) .write .option("dataChange", "false") .format("delta") .mode("overwrite") .option("replaceWhere", partition) .save(path) ``` * Python ```python path = "..." partition = "year = '2019'" numFilesPerPartition = 16 (spark.read .format("delta") .load(path) .where(partition) .repartition(numFilesPerPartition) .write .option("dataChange", "false") .format("delta") .mode("overwrite") .option("replaceWhere", partition) .save(path)) ``` Caution Using `dataChange = false` on an operation that changes data can corrupt the data in the table. Note This operation does not remove the old files. To remove them, run the [VACUUM](/delta-utility/#remove-files-no-longer-referenced-by-a-delta-table) command. ## Replace the content or schema of a table [Section titled “Replace the content or schema of a table”](#replace-the-content-or-schema-of-a-table) Sometimes you may want to replace a Delta table. For example: * You discover the data in the table is incorrect and want to replace the content. * You want to rewrite the whole table to do incompatible schema changes (such as changing column types). While you can delete the entire directory of a Delta table and create a new table on the same path, it’s *not recommended* because: * Deleting a directory is not efficient. A directory containing very large files can take hours or even days to delete. * You lose all of content in the deleted files; it’s hard to recover if you delete the wrong table. * The directory deletion is not atomic. While you are deleting the table a concurrent query reading the table can fail or see a partial table. If you don’t need to change the table schema, you can [delete](/delta-update/#delete-from-a-table) data from a Delta table and insert your new data, or [update](/delta-update/#update-a-table) the table to fix the incorrect values. If you want to change the table schema, you can replace the whole table atomically. For example: * Python ```python dataframe.write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .partitionBy() \ .saveAsTable("") # Managed table dataframe.write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .option("path", "") \ .partitionBy() \ .saveAsTable("") # External table ``` * SQL ```sql REPLACE TABLE USING DELTA PARTITIONED BY () AS SELECT ... -- Managed table REPLACE TABLE USING DELTA PARTITIONED BY () LOCATION "" AS SELECT ... -- External table ``` * Scala ```scala dataframe.write .format("delta") .mode("overwrite") .option("overwriteSchema", "true") .partitionBy() .saveAsTable("") // Managed table dataframe.write .format("delta") .mode("overwrite") .option("overwriteSchema", "true") .option("path", "") .partitionBy() .saveAsTable("") // External table ``` There are multiple benefits with this approach: * Overwriting a table is much faster because it doesn’t need to list the directory recursively or delete any files. * The old version of the table still exists. If you delete the wrong table you can easily retrieve the old data using [Time Travel](/delta-batch/#query-an-older-snapshot-of-a-table-time-travel). * It’s an atomic operation. Concurrent queries can still read the table while you are deleting the table. * Because of Delta Lake ACID transaction guarantees, if overwriting the table fails, the table will be in its previous state. In addition, if you want to delete old files to save storage cost after overwriting the table, you can use [VACUUM](/delta-utility/#remove-files-no-longer-referenced-by-a-delta-table) to delete them. It’s optimized for file deletion and usually faster than deleting the entire directory. ## Spark caching [Section titled “Spark caching”](#spark-caching) You should not use Spark caching for the following reasons: * You lose any data skipping that can come from additional filters added on top of the cached `DataFrame`. * The data that gets cached may not be updated if the table is accessed using a different identifier (for example, you do `spark.table(x).cache()` but then write to the table using `spark.write.save(/some/path)`. # Google BigQuery connector > Learn how to read Delta Lake tables from Google BigQuery. Google BigQuery supports reading Delta Lake (reader version 3 with [Deletion Vectors](/delta-deletion-vectors) and [Column Mapping](/delta-column-mapping/)). Please refer to [Delta Lake BigLake tables documentation](https://cloud.google.com/bigquery/docs/create-delta-lake-table) for more details. # Concurrency control > Learn about the ACID transaction guarantees between reads and writes provided by Delta Lake. Delta Lake provides ACID transaction guarantees between reads and writes. This means that: * For supported [storage systems](/delta-storage), multiple writers across multiple clusters can simultaneously modify a table partition and see a consistent snapshot view of the table and there will be a serial order for these writes. * Readers continue to see a consistent snapshot view of the table that the Apache Spark job started with, even when a table is modified during a job. ## Optimistic concurrency control [Section titled “Optimistic concurrency control”](#optimistic-concurrency-control) Delta Lake uses [optimistic concurrency control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) to provide transactional guarantees between writes. Under this mechanism, writes operate in three stages: 1. **Read**: Reads (if needed) the latest available version of the table to identify which files need to be modified (that is, rewritten). 2. **Write**: Stages all the changes by writing new data files. 3. **Validate and commit**: Before committing the changes, checks whether the proposed changes conflict with any other changes that may have been concurrently committed since the snapshot that was read. If there are no conflicts, all the staged changes are committed as a new versioned snapshot, and the write operation succeeds. However, if there are conflicts, the write operation fails with a concurrent modification exception rather than corrupting the table as would happen with the write operation on a Parquet table. ## Write conflicts [Section titled “Write conflicts”](#write-conflicts) The following table describes which pairs of write operations can conflict. Compaction refers to [file compaction operation](/best-practices/#compact-files) written with the option `dataChange` set to `false`. | | INSERT | UPDATE, DELETE, MERGE INTO | COMPACTION | | ------------------------------ | --------------- | -------------------------- | ------------ | | **INSERT** | Cannot conflict | | | | **UPDATE, DELETE, MERGE INTO** | Can conflict | Can conflict | | | **COMPACTION** | Cannot conflict | Can conflict | Can conflict | ## Avoid conflicts using partitioning and disjoint command conditions [Section titled “Avoid conflicts using partitioning and disjoint command conditions”](#avoid-conflicts-using-partitioning-and-disjoint-command-conditions) In all cases marked “can conflict”, whether the two operations will conflict depends on whether they operate on the same set of files. You can make the two sets of files disjoint by partitioning the table by the same columns as those used in the conditions of the operations. For example, the two commands `UPDATE table WHERE date > '2010-01-01' ...` and `DELETE table WHERE date < '2010-01-01'` will conflict if the table is not partitioned by date, as both can attempt to modify the same set of files. Partitioning the table by `date` will avoid the conflict. Hence, partitioning a table according to the conditions commonly used on the command can reduce conflicts significantly. However, partitioning a table by a column that has high cardinality can lead to other performance issues due to large number of subdirectories. ## Conflict exceptions [Section titled “Conflict exceptions”](#conflict-exceptions) When a transaction conflict occurs, you will observe one of the following exceptions: ### ConcurrentAppendException [Section titled “ConcurrentAppendException”](#concurrentappendexception) This exception occurs when a concurrent operation adds files in the same partition (or anywhere in an unpartitioned table) that your operation reads. The file additions can be caused by `INSERT`, `DELETE`, `UPDATE`, or `MERGE` operations. This exception is often thrown during concurrent `DELETE`, `UPDATE`, or `MERGE` operations. While the concurrent operations may be physically updating different partition directories, one of them may read the same partition that the other one concurrently updates, thus causing a conflict. You can avoid this by making the separation explicit in the operation condition. Consider the following example. * Scala ```scala // Target 'deltaTable' is partitioned by date and country deltaTable.as("t") .merge( source.as("s"), "s.user_id = t.user_id AND s.date = t.date AND s.country = t.country" ) .whenMatched() .updateAll() .whenNotMatched() .insertAll() .execute() ``` Suppose you run the above code concurrently for different dates or countries. Since each job is working on an independent partition on the target Delta table, you don’t expect any conflicts. However, the condition is not explicit enough and can scan the entire table and can conflict with concurrent operations updating any other partitions. Instead, you can rewrite your statement to add specific date and country to the merge condition, as shown in the following example. * Scala ```scala // Target 'deltaTable' is partitioned by date and country deltaTable.as("t") .merge( source.as("s"), "s.user_id = t.user_id AND s.date = t.date AND s.country = t.country AND t.date = '" + + "' AND t.country = '" + + "'" ) .whenMatched() .updateAll() .whenNotMatched() .insertAll() .execute() ``` This operation is now safe to run concurrently on different dates and countries. ### ConcurrentDeleteReadException [Section titled “ConcurrentDeleteReadException”](#concurrentdeletereadexception) This exception occurs when a concurrent operation deleted a file that your operation read. Common causes are a `DELETE`, `UPDATE`, or `MERGE` operation that rewrites files. ### ConcurrentDeleteDeleteException [Section titled “ConcurrentDeleteDeleteException”](#concurrentdeletedeleteexception) This exception occurs when a concurrent operation deleted a file that your operation also deletes. This could be caused by two concurrent compaction operations rewriting the same files. ### MetadataChangedException [Section titled “MetadataChangedException”](#metadatachangedexception) This exception occurs when a concurrent transaction updates the metadata of a Delta table. Common causes are `ALTER TABLE` operations or writes to your Delta table that update the schema of the table. ### ConcurrentTransactionException [Section titled “ConcurrentTransactionException”](#concurrenttransactionexception) If a streaming query using the same checkpoint location is started multiple times concurrently and tries to write to the Delta table at the same time. You should never have two streaming queries use the same checkpoint location and run at the same time. ### ProtocolChangedException [Section titled “ProtocolChangedException”](#protocolchangedexception) This exception can occur in the following cases: * When your Delta table is upgraded to a new version. For future operations to succeed you may need to upgrade your Delta Lake version. * When multiple writers are creating or replacing a table at the same time. * When multiple writers are writing to an empty path at the same time. # Delta Lake APIs > Learn about the APIs provided by Delta Lake. Note Some Delta Lake APIs are still evolving and are indicated with the **Evolving** qualifier or annotation in the API docs. ## Delta Spark [Section titled “Delta Spark”](#delta-spark) Delta Spark is a library for reading and writing Delta tables using Apache Spark™. For most read and write operations on Delta tables, you can use Apache Spark reader and writer APIs. For examples, see [Table batch reads and writes](/delta-batch/) and [Table streaming reads and writes](/delta-streaming/). However, there are some operations that are specific to Delta Lake and you must use Delta Lake APIs. For examples, see [Table utility commands](/delta-utility/). * [Scala API docs](/api/latest/scala/spark/io/delta/tables/index.html) * [Java API docs](/api/latest/java/spark/index.html) * [Python API docs](/api/latest/python/spark/index.html) ## Delta Kernel [Section titled “Delta Kernel”](#delta-kernel) Delta Kernel is a library for operating on Delta tables. Specifically, it provides simple and narrow APIs for reading and writing to Delta tables without the need to understand the [Delta protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) details. You can use this library to do the following: * Read Delta tables from your applications. * Build a connector for a distributed engine like Apache Spark™, Apache Flink, or Trino for reading massive Delta tables. More details refer [here](https://github.com/delta-io/delta/blob/branch-3.0/kernel/USER_GUIDE.md). * [Java API docs](/api/latest/java/kernel/index.html) ## Delta Rust [Section titled “Delta Rust”](#delta-rust) This [library](https://docs.rs/deltalake/latest/deltalake/) allows Rust (with Python bindings) low level access to Delta tables and is intended to be used with data processing frameworks like `datafusion`, `ballista`, `rust-dataframe`, `vega`, etc. ## Delta Standalone [Section titled “Delta Standalone”](#delta-standalone) Caution The Delta Standalone is deprecated in favor of [Delta Kernel](/delta-kernel/) which has support for reading from or writing into Delta tables with advanced features. Delta Standalone, formerly known as the Delta Standalone Reader (DSR), is a JVM library to read and write Delta tables. Unlike Delta-Spark, this library doesn’t use Spark to read or write tables and it has only a few transitive dependencies. It can be used by any application that cannot use a Spark cluster. More details refer [here](https://github.com/delta-io/delta/blob/master/connectors/README.md). * [Java API docs](/api/3.3.2/java/standalone/index.html) ## Delta Flink [Section titled “Delta Flink”](#delta-flink) Flink/Delta Connector is a JVM library to read and write data from Apache Flink applications to Delta tables utilizing the Delta Standalone JVM library. More details refer [here](https://github.com/delta-io/delta/blob/master/connectors/flink/README.md). * [Java API docs](/api/3.3.2/java/flink/index.html) # AWS Athena Delta Connector > Learn how to set up an integration to enable you to read Delta tables from AWS Athena. # AWS Athena Delta Connector [Section titled “AWS Athena Delta Connector”](#aws-athena-delta-connector) Since Athena [version 3](https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html), Athena natively supports reading Delta Lake tables. For details on using the native Delta Lake connector, see [Querying Delta Lake tables](https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html). For Athena versions lower than [version 3](https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html), you can use the manifest-based approach detailed in [Presto, Trino, and Athena to Delta Lake integration using manifests](/presto-integration). # Table batch reads and writes > Learn how to perform batch reads and writes on Delta tables. Delta Lake supports most of the options provided by Apache Spark DataFrame read and write APIs for performing batch reads and writes on tables. For many Delta Lake operations on tables, you enable integration with Apache Spark DataSourceV2 and Catalog APIs (since 3.0) by setting configurations when you create a new `SparkSession`. See [Configure SparkSession](#configure-sparksession). ## Create a table [Section titled “Create a table”](#create-a-table) Delta Lake supports creating two types of tables—tables defined in the metastore and tables defined by path. To work with metastore-defined tables, you must enable integration with Apache Spark DataSourceV2 and Catalog APIs by setting configurations when you create a new `SparkSession`. See [Configure SparkSession](#configure-sparksession). You can create tables in the following ways: * **SQL DDL commands**: You can use standard SQL DDL commands supported in Apache Spark (for example, `CREATE TABLE` and `REPLACE TABLE`) to create Delta tables. * SQL ```sql CREATE TABLE IF NOT EXISTS default.people10m ( id INT, firstName STRING, middleName STRING, lastName STRING, gender STRING, birthDate TIMESTAMP, ssn STRING, salary INT ) USING DELTA CREATE OR REPLACE TABLE default.people10m ( id INT, firstName STRING, middleName STRING, lastName STRING, gender STRING, birthDate TIMESTAMP, ssn STRING, salary INT ) USING DELTA ``` SQL also supports creating a table at a path, without creating an entry in the Hive metastore. * SQL ```sql -- Create or replace table with path CREATE OR REPLACE TABLE delta.`/tmp/delta/people10m` ( id INT, firstName STRING, middleName STRING, lastName STRING, gender STRING, birthDate TIMESTAMP, ssn STRING, salary INT ) USING DELTA ``` * **`DataFrameWriter` API**: If you want to simultaneously create a table and insert data into it from Spark DataFrames or Datasets, you can use the Spark `DataFrameWriter` ([Scala or Java](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriter.html) and [Python](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/io.html)). * Python ```python # Create table in the metastore using DataFrame's schema and write data to it df.write.format("delta").saveAsTable("default.people10m") # Create or replace partitioned table with path using DataFrame's schema and write/overwrite data to it df.write.format("delta").mode("overwrite").save("/tmp/delta/people10m") ``` * Scala ```scala // Create table in the metastore using DataFrame's schema and write data to it df.write.format("delta").saveAsTable("default.people10m") // Create table with path using DataFrame's schema and write data to it df.write.format("delta").mode("overwrite").save("/tmp/delta/people10m") ``` You can also create Delta tables using the Spark `DataFrameWriterV2` API. * **`DeltaTableBuilder` API**: You can also use the `DeltaTableBuilder` API in Delta Lake to create tables. Compared to the DataFrameWriter APIs, this API makes it easier to specify additional information like column comments, table properties, and [generated columns](#use-generated-columns). Note This feature is new and is in Preview. * Python ```python # Create table in the metastore DeltaTable.createIfNotExists(spark) \ .tableName("default.people10m") \ .addColumn("id", "INT") \ .addColumn("firstName", "STRING") \ .addColumn("middleName", "STRING") \ .addColumn("lastName", "STRING", comment = "surname") \ .addColumn("gender", "STRING") \ .addColumn("birthDate", "TIMESTAMP") \ .addColumn("ssn", "STRING") \ .addColumn("salary", "INT") \ .execute() # Create or replace table with path and add properties DeltaTable.createOrReplace(spark) \ .addColumn("id", "INT") \ .addColumn("firstName", "STRING") \ .addColumn("middleName", "STRING") \ .addColumn("lastName", "STRING", comment = "surname") \ .addColumn("gender", "STRING") \ .addColumn("birthDate", "TIMESTAMP") \ .addColumn("ssn", "STRING") \ .addColumn("salary", "INT") \ .property("description", "table with people data") \ .location("/tmp/delta/people10m") \ .execute() ``` * Scala ```scala // Create table in the metastore DeltaTable.createOrReplace(spark) .tableName("default.people10m") .addColumn("id", "INT") .addColumn("firstName", "STRING") .addColumn("middleName", "STRING") .addColumn( DeltaTable.columnBuilder("lastName") .dataType("STRING") .comment("surname") .build()) .addColumn("lastName", "STRING", comment = "surname") .addColumn("gender", "STRING") .addColumn("birthDate", "TIMESTAMP") .addColumn("ssn", "STRING") .addColumn("salary", "INT") .execute() // Create or replace table with path and add properties DeltaTable.createOrReplace(spark) .addColumn("id", "INT") .addColumn("firstName", "STRING") .addColumn("middleName", "STRING") .addColumn( DeltaTable.columnBuilder("lastName") .dataType("STRING") .comment("surname") .build()) .addColumn("lastName", "STRING", comment = "surname") .addColumn("gender", "STRING") .addColumn("birthDate", "TIMESTAMP") .addColumn("ssn", "STRING") .addColumn("salary", "INT") .property("description", "table with people data") .location("/tmp/delta/people10m") .execute() ``` See the [API documentation](/delta-apidoc/) for details. ### Partition data [Section titled “Partition data”](#partition-data) You can partition data to speed up queries or DML that have predicates involving the partition columns. To partition data when you create a Delta table, specify a partition by columns. The following example partitions by gender. * SQL ```sql -- Create table in the metastore CREATE TABLE default.people10m ( id INT, firstName STRING, middleName STRING, lastName STRING, gender STRING, birthDate TIMESTAMP, ssn STRING, salary INT ) USING DELTA PARTITIONED BY (gender) ``` * Python ```python df.write.format("delta").partitionBy("gender").saveAsTable("default.people10m") DeltaTable.create(spark) \ .tableName("default.people10m") \ .addColumn("id", "INT") \ .addColumn("firstName", "STRING") \ .addColumn("middleName", "STRING") \ .addColumn("lastName", "STRING", comment = "surname") \ .addColumn("gender", "STRING") \ .addColumn("birthDate", "TIMESTAMP") \ .addColumn("ssn", "STRING") \ .addColumn("salary", "INT") \ .partitionedBy("gender") \ .execute() ``` * Scala ```scala df.write.format("delta").partitionBy("gender").saveAsTable("default.people10m") DeltaTable.createOrReplace(spark) .tableName("default.people10m") .addColumn("id", "INT") .addColumn("firstName", "STRING") .addColumn("middleName", "STRING") .addColumn( DeltaTable.columnBuilder("lastName") .dataType("STRING") .comment("surname") .build()) .addColumn("lastName", "STRING", comment = "surname") .addColumn("gender", "STRING") .addColumn("birthDate", "TIMESTAMP") .addColumn("ssn", "STRING") .addColumn("salary", "INT") .partitionedBy("gender") .execute() ``` To determine whether a table contains a specific partition, use the statement `SELECT COUNT(*) > 0 FROM WHERE = `. If the partition exists, `true` is returned. For example: * SQL ```sql SELECT COUNT(*) > 0 AS `Partition exists` FROM default.people10m WHERE gender = "M" ``` * Python ```python display(spark.sql("SELECT COUNT(*) > 0 AS `Partition exists` FROM default.people10m WHERE gender = 'M'")) ``` * Scala ```scala display(spark.sql("SELECT COUNT(*) > 0 AS `Partition exists` FROM default.people10m WHERE gender = 'M'")) ``` ### Control data location [Section titled “Control data location”](#control-data-location) For tables defined in the metastore, you can optionally specify the `LOCATION` as a path. Tables created with a specified `LOCATION` are considered unmanaged by the metastore. Unlike a managed table, where no path is specified, an unmanaged table’s files are not deleted when you `DROP` the table. When you run `CREATE TABLE` with a `LOCATION` that *already* contains data stored using Delta Lake, Delta Lake does the following: * If you specify *only the table name and location*, for example: * SQL ```sql CREATE TABLE default.people10m USING DELTA LOCATION '/tmp/delta/people10m' ``` the table in the metastore automatically inherits the schema, partitioning, and table properties of the existing data. This functionality can be used to “import” data into the metastore. * If you specify *any configuration* (schema, partitioning, or table properties), Delta Lake verifies that the specification exactly matches the configuration of the existing data. Important If the specified configuration does not *exactly* match the configuration of the data, Delta Lake throws an exception that describes the discrepancy. Note The metastore is not the source of truth about the latest information of a Delta table. In fact, the table definition in the metastore may not contain all the metadata like schema and properties. It contains the location of the table, and the table’s transaction log at the location is the source of truth. If you query the metastore from a system that is not aware of this Delta-specific customization, you may see incomplete or stale table information. ### Use generated columns [Section titled “Use generated columns”](#use-generated-columns) Note This feature is new and is in Preview. Delta Lake supports generated columns which are a special type of columns whose values are automatically generated based on a user-specified function over other columns in the Delta table. When you write to a table with generated columns and you do not explicitly provide values for them, Delta Lake automatically computes the values. For example, you can automatically generate a date column (for partitioning the table by date) from the timestamp column; any writes into the table need only specify the data for the timestamp column. However, if you explicitly provide values for them, the values must satisfy the [constraint](/delta-constraints/) `( <=> ) IS TRUE` or the write will fail with an error. Important Tables created with generated columns have a higher table writer protocol version than the default. See [How does Delta Lake manage feature compatibility?](/versioning/) to understand table protocol versioning and what it means to have a higher version of a table protocol version. The following example shows how to create a table with generated columns: * Python ```python DeltaTable.create(spark) \ .tableName("default.people10m") \ .addColumn("id", "INT") \ .addColumn("firstName", "STRING") \ .addColumn("middleName", "STRING") \ .addColumn("lastName", "STRING", comment = "surname") \ .addColumn("gender", "STRING") \ .addColumn("birthDate", "TIMESTAMP") \ .addColumn("dateOfBirth", DateType(), generatedAlwaysAs="CAST(birthDate AS DATE)") \ .addColumn("ssn", "STRING") \ .addColumn("salary", "INT") \ .partitionedBy("gender") \ .execute() ``` * Scala ```scala DeltaTable.create(spark) .tableName("default.people10m") .addColumn("id", "INT") .addColumn("firstName", "STRING") .addColumn("middleName", "STRING") .addColumn( DeltaTable.columnBuilder("lastName") .dataType("STRING") .comment("surname") .build()) .addColumn("lastName", "STRING", comment = "surname") .addColumn("gender", "STRING") .addColumn("birthDate", "TIMESTAMP") .addColumn( DeltaTable.columnBuilder("dateOfBirth") .dataType(DateType) .generatedAlwaysAs("CAST(dateOfBirth AS DATE)") .build()) .addColumn("ssn", "STRING") .addColumn("salary", "INT") .partitionedBy("gender") .execute() ``` Generated columns are stored as if they were normal columns. That is, they occupy storage. The following restrictions apply to generated columns: * A generation expression can use any SQL functions in Spark that always return the same result when given the same argument values, except the following types of functions: * User-defined functions. * Aggregate functions. * Window functions. * Functions returning multiple rows. * For Delta Lake 1.1.0 and above, `MERGE` operations support generated columns when you set `spark.databricks.delta.schema.autoMerge.enabled` to true. Delta Lake may be able to generate partition filters for a query whenever a partition column is defined by one of the following expressions: * `CAST(col AS DATE)` and the type of `col` is `TIMESTAMP`. * `YEAR(col)` and the type of `col` is `TIMESTAMP`. * Two partition columns defined by `YEAR(col), MONTH(col)` and the type of `col` is `TIMESTAMP`. * Three partition columns defined by `YEAR(col), MONTH(col), DAY(col)` and the type of `col` is `TIMESTAMP`. * Four partition columns defined by `YEAR(col), MONTH(col), DAY(col), HOUR(col)` and the type of `col` is `TIMESTAMP`. * `SUBSTRING(col, pos, len)` and the type of `col` is `STRING` * `DATE_FORMAT(col, format)` and the type of `col` is `TIMESTAMP`. * `DATE_TRUNC(format, col)` and the type of the `col` is `TIMESTAMP` or `DATE`. * `TRUNC(col, format)` and type of the `col` is either `TIMESTAMP` or `DATE`. If a partition column is defined by one of the preceding expressions, and a query filters data using the underlying base column of a generation expression, Delta Lake looks at the relationship between the base column and the generated column, and populates partition filters based on the generated partition column if possible. For example, given the following table: * Python ```python DeltaTable.create(spark) \ .tableName("default.events") \ .addColumn("eventId", "BIGINT") \ .addColumn("data", "STRING") \ .addColumn("eventType", "STRING") \ .addColumn("eventTime", "TIMESTAMP") \ .addColumn("eventDate", "DATE", generatedAlwaysAs="CAST(eventTime AS DATE)") \ .partitionedBy("eventType", "eventDate") \ .execute() ``` If you then run the following query: * Python ```python spark.sql('SELECT * FROM default.events WHERE eventTime >= "2020-10-01 00:00:00" <= "2020-10-01 12:00:00"') ``` Delta Lake automatically generates a partition filter so that the preceding query only reads the data in partition `date=2020-10-01` even if a partition filter is not specified. As another example, given the following table: * Python ```python DeltaTable.create(spark) \ .tableName("default.events") \ .addColumn("eventId", "BIGINT") \ .addColumn("data", "STRING") \ .addColumn("eventType", "STRING") \ .addColumn("eventTime", "TIMESTAMP") \ .addColumn("year", "INT", generatedAlwaysAs="YEAR(eventTime)") \ .addColumn("month", "INT", generatedAlwaysAs="MONTH(eventTime)") \ .addColumn("day", "INT", generatedAlwaysAs="DAY(eventTime)") \ .partitionedBy("eventType", "year", "month", "day") \ .execute() ``` If you then run the following query: * Python ```python spark.sql('SELECT * FROM default.events WHERE eventTime >= "2020-10-01 00:00:00" <= "2020-10-01 12:00:00"') ``` Delta Lake automatically generates a partition filter so that the preceding query only reads the data in partition `year=2020/month=10/day=01` even if a partition filter is not specified. You can use an [EXPLAIN](https://spark.apache.org/docs/latest/sql-ref-syntax-qry-explain.html) clause and check the provided plan to see whether Delta Lake automatically generates any partition filters. ### Use identity columns [Section titled “Use identity columns”](#use-identity-columns) Important Declaring an identity column on a Delta table disables concurrent transactions. Only use identity columns in use cases where concurrent writes to the target table are not required. Delta Lake identity columns are supported in Delta Lake 3.3 and above. They are a type of generated column that assigns unique values for each record inserted into a table. The following example shows how to declare an identity column during a create table command: * Python ```python from delta.tables import DeltaTable, IdentityGenerator from pyspark.sql.types import LongType DeltaTable.create() .tableName("table_name") .addColumn("id_col1", dataType=LongType(), generatedAlwaysAs=IdentityGenerator()) .addColumn("id_col2", dataType=LongType(), generatedAlwaysAs=IdentityGenerator(start=-1, step=1)) .addColumn("id_col3", dataType=LongType(), generatedByDefaultAs=IdentityGenerator()) .addColumn("id_col4", dataType=LongType(), generatedByDefaultAs=IdentityGenerator(start=-1, step=1)) .execute() ``` * Scala ```scala import io.delta.tables.DeltaTable import org.apache.spark.sql.types.LongType DeltaTable.create(spark) .tableName("table_name") .addColumn( DeltaTable.columnBuilder(spark, "id_col1") .dataType(LongType) .generatedAlwaysAsIdentity().build()) .addColumn( DeltaTable.columnBuilder(spark, "id_col2") .dataType(LongType) .generatedAlwaysAsIdentity(start = -1L, step = 1L).build()) .addColumn( DeltaTable.columnBuilder(spark, "id_col3") .dataType(LongType) .generatedByDefaultAsIdentity().build()) .addColumn( DeltaTable.columnBuilder(spark, "id_col4") .dataType(LongType) .generatedByDefaultAsIdentity(start = -1L, step = 1L).build()) .execute() ``` Note SQL APIs for identity columns are not supported yet. You can optionally specify the following: * A starting value. * A step size, which can be positive or negative. Both the starting value and step size default to `1`. You cannot specify a step size of `0`. Values assigned by identity columns are unique and increment in the direction of the specified step, and in multiples of the specified step size, but are not guaranteed to be contiguous. For example, with a starting value of `0` and a step size of `2`, all values are positive even numbers but some even numbers might be skipped. When the identity column is specified to be `generated by default as identity`, insert operations can specify values for the identity column. Specify it to be `generated always as identity` to override the ability to manually set values. Identity columns only support `LongType`, and operations fail if the assigned value exceeds the range supported by `LongType`. You can use `ALTER TABLE table_name ALTER COLUMN column_name SYNC IDENTITY` to synchronize the metadata of an identity column with the actual data. When you write your own values to an identity column, it might not comply with the metadata. This option evaluates the state and updates the metadata to be consistent with the actual data. After this command, the next automatically assigned identity value will start from `start + (n + 1) * step`, where `n` is the smallest value that satisfies `start + n * step >= max()` (for a positive step). #### CTAS and identity columns [Section titled “CTAS and identity columns”](#ctas-and-identity-columns) You cannot define schema, identity column constraints, or any other table specifications when using a `CREATE TABLE table_name AS SELECT` (CTAS) statement. To create a new table with an identity column and populate it with existing data, do the following: 1. Create a table with the correct schema, including the identity column definition and other table properties. 2. Run an insertion operation. The following example define the identity column to be `generated by default as identity`. If data inserted into the table includes valid values for the identity column, these values are used. * Python ```python from delta.tables import DeltaTable, IdentityGenerator from pyspark.sql.types import LongType, DateType DeltaTable.create(spark) .tableName("new_table") .addColumn("id", dataType=LongType(), generatedByDefaultAs=IdentityGenerator(start=5, step=1)) .addColumn("event_date", dataType=DateType()) .addColumn("some_value", dataType=LongType()) .execute() # Insert records including existing IDs old_table_df = spark.table("old_table").select("id", "event_date", "some_value") old_table_df.write .format("delta") .mode("append") .saveAsTable("new_table") # Insert records and generate new IDs new_records_df = spark.table("new_records").select("event_date", "some_value") new_records_df.write .format("delta") .mode("append") .saveAsTable("new_table") ``` * Scala ```scala import org.apache.spark.sql.types._ import io.delta.tables.DeltaTable DeltaTable.createOrReplace(spark) .tableName("new_table") .addColumn( DeltaTable.columnBuilder(spark, "id") .dataType(LongType) .generatedByDefaultAsIdentity(start = 5L, step = 1L) .build()) .addColumn( DeltaTable.columnBuilder(spark, "event_date") .dataType(DateType) .nullable(true) .build()) .addColumn( DeltaTable.columnBuilder(spark, "some_value") .dataType(LongType) .nullable(true) .build()) .execute() // Insert records including existing IDs val oldTableDF = spark.table("old_table").select("id", "event_date", "some_value") oldTableDF.write .format("delta") .mode("append") .saveAsTable("new_table") // Insert records and generate new IDs val newRecordsDF = spark.table("new_records").select("event_date", "some_value") newRecordsDF.write .format("delta") .mode("append") .saveAsTable("new_table") ``` #### Identity column limitations [Section titled “Identity column limitations”](#identity-column-limitations) The following limitations exist when working with identity columns: * Concurrent transactions are not supported on tables with identity columns enabled. * You cannot partition a table by an identity column. * You cannot `ADD`, `REPLACE`, or `CHANGE` an identity column. * You cannot update the value of an identity column for an existing record. Note To change the `IDENTITY` value for an existing record, you must delete the record and `INSERT` it as a new record. ### Specify default values for columns [Section titled “Specify default values for columns”](#specify-default-values-for-columns) Delta enables the specification of [default expressions](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#default-columns) for columns in Delta tables. When users write to these tables without explicitly providing values for certain columns, or when they explicitly use the `DEFAULT` SQL keyword for a column, Delta automatically generates default values for those columns. For more information, please refer to the dedicated documentation page. ### Use special characters in column names [Section titled “Use special characters in column names”](#use-special-characters-in-column-names) By default, special characters such as spaces and any of the characters `,;{}()\n\t=` are not supported in table column names. To include these special characters in a table’s column name, enable column mapping. ### Default table properties [Section titled “Default table properties”](#default-table-properties) Delta Lake configurations set in the SparkSession override the default [table properties](/table-properties/) for new Delta Lake tables created in the session. The prefix used in the SparkSession is different from the configurations used in the table properties. | Delta Lake conf | SparkSession conf | | --------------- | --------------------------------------------------- | | `delta.` | `spark.databricks.delta.properties.defaults.` | For example, to set the `delta.appendOnly = true` property for all new Delta Lake tables created in a session, set the following: ```sql SET spark.databricks.delta.properties.defaults.appendOnly = true ``` ## Read a table [Section titled “Read a table”](#read-a-table) You can load a Delta table as a DataFrame by specifying a table name or a path: * SQL ```sql SELECT * FROM default.people10m -- query table in the metastore SELECT * FROM delta.`/tmp/delta/people10m` -- query table by path ``` * Python ```python spark.table("default.people10m") # query table in the metastore spark.read.format("delta").load("/tmp/delta/people10m") # query table by path ``` * Scala ```scala spark.table("default.people10m") // query table in the metastore spark.read.format("delta").load("/tmp/delta/people10m") // create table by path import io.delta.implicits._ spark.read.delta("/tmp/delta/people10m") ``` The DataFrame returned automatically reads the most recent snapshot of the table for any query; you never need to run `REFRESH TABLE`. Delta Lake automatically uses partitioning and statistics to read the minimum amount of data when there are applicable predicates in the query. ## Query an older snapshot of a table (time travel) [Section titled “Query an older snapshot of a table (time travel)”](#query-an-older-snapshot-of-a-table-time-travel) Delta Lake time travel allows you to query an older snapshot of a Delta table. Time travel has many use cases, including: * Re-creating analyses, reports, or outputs (for example, the output of a machine learning model). This could be useful for debugging or auditing, especially in regulated industries. * Writing complex temporal queries. * Fixing mistakes in your data. * Providing snapshot isolation for a set of queries for fast changing tables. This section describes the supported methods for querying older versions of tables, data retention concerns, and provides examples. Note The timestamp of each version N depends on the timestamp of the log file corresponding to the version N in Delta table log. Hence, time travel by timestamp can break if you copy the entire Delta table directory to a new location. Time travel by version will be unaffected. ### Syntax [Section titled “Syntax”](#syntax) This section shows how to query an older version of a Delta table. #### SQL `AS OF` syntax [Section titled “SQL AS OF syntax”](#sql-as-of-syntax) * SQL ```sql SELECT * FROM table_name TIMESTAMP AS OF timestamp_expression SELECT * FROM table_name VERSION AS OF version ``` - `timestamp_expression` can be any one of: * `'2018-10-18T22:15:12.013Z'`, that is, a string that can be cast to a timestamp * `cast('2018-10-18 13:36:32 CEST' as timestamp)` * `'2018-10-18'`, that is, a date string * `current_timestamp() - interval 12 hours` * `date_sub(current_date(), 1)` * Any other expression that is or can be cast to a timestamp - `version` is a long value that can be obtained from the output of `DESCRIBE HISTORY table_spec`. Neither `timestamp_expression` nor `version` can be subqueries. ##### Example [Section titled “Example”](#example) * SQL ```sql SELECT * FROM default.people10m TIMESTAMP AS OF '2018-10-18T22:15:12.013Z' SELECT * FROM delta.`/tmp/delta/people10m` VERSION AS OF 123 ``` #### DataFrameReader options [Section titled “DataFrameReader options”](#dataframereader-options) DataFrameReader options allow you to create a DataFrame from a Delta table that is fixed to a specific version of the table. * Python ```python df1 = spark.read.format("delta").option("timestampAsOf", timestamp_string).load("/tmp/delta/people10m") df2 = spark.read.format("delta").option("versionAsOf", version).load("/tmp/delta/people10m") ``` For `timestamp_string`, only date or timestamp strings are accepted. For example, `"2019-01-01"` and `"2019-01-01T00:00:00.000Z"`. A common pattern is to use the latest state of the Delta table throughout the execution of a job to update downstream applications. Because Delta tables auto update, a DataFrame loaded from a Delta table may return different results across invocations if the underlying data is updated. By using time travel, you can fix the data returned by the DataFrame across invocations: * Python ```python history = spark.sql("DESCRIBE HISTORY delta.`/tmp/delta/people10m`") latest_version = history.selectExpr("max(version)").collect() df = spark.read.format("delta").option("versionAsOf", latest_version[0][0]).load("/tmp/delta/people10m") ``` ##### Examples [Section titled “Examples”](#examples) * Fix accidental deletes to a table for the user 111: * Python ```python yesterday = spark.sql("SELECT CAST(date_sub(current_date(), 1) AS STRING)").collect()[0][0] df = spark.read.format("delta").option("timestampAsOf", yesterday).load("/tmp/delta/events") df.where("userId = 111").write.format("delta").mode("append").save("/tmp/delta/events") ``` * Fix accidental incorrect updates to a table: * Python ```python yesterday = spark.sql("SELECT CAST(date_sub(current_date(), 1) AS STRING)").collect()[0][0] df = spark.read.format("delta").option("timestampAsOf", yesterday).load("/tmp/delta/events") df.createOrReplaceTempView("my_table_yesterday") spark.sql(''' MERGE INTO delta.`/tmp/delta/events` target USING my_table_yesterday source ON source.userId = target.userId WHEN MATCHED THEN UPDATE SET * ''') ``` * Query the number of new customers added over the last week: * Python ```python last_week = spark.sql("SELECT CAST(date_sub(current_date(), 7) AS STRING)").collect()[0][0] df = spark.read.format("delta").option("timestampAsOf", last_week).load("/tmp/delta/events") last_week_count = df.select("userId").distinct().count() count = spark.read.format("delta").load("/tmp/delta/events").select("userId").distinct().count() new_customers_count = count - last_week_count ``` ### Data retention [Section titled “Data retention”](#data-retention) To time travel to a previous version, you must retain *both* the log and the data files for that version. The data files backing a Delta table are *never* deleted automatically; data files are deleted only when you run [VACUUM](/delta-utility#remove-files-no-longer-referenced-by-a-delta-table). `VACUUM` *does not* delete Delta log files; log files are automatically cleaned up after checkpoints are written. By default you can time travel to a Delta table up to 30 days old unless you have: * Run `VACUUM` on your Delta table. * Changed the data or log file retention periods using the following [table properties](/table-properties/): * `delta.logRetentionDuration = "interval "`: controls how long the history for a table is kept. The default is `interval 30 days`. Each time a checkpoint is written, Delta automatically cleans up log entries older than the retention interval. If you set this config to a large enough value, many log entries are retained. This should not impact performance as operations against the log are constant time. Operations on history are parallel but will become more expensive as the log size increases. * `delta.deletedFileRetentionDuration = "interval "`: controls how long ago a file must have been deleted *before being a candidate for* `VACUUM`. The default is `interval 7 days`. To access 30 days of historical data even if you run `VACUUM` on the Delta table, set `delta.deletedFileRetentionDuration = "interval 30 days"`. This setting may cause your storage costs to go up. Note Due to log entry cleanup, instances can arise where you cannot time travel to a version that is less than the retention interval. Delta Lake requires all consecutive log entries since the previous checkpoint to time travel to a particular version. For example, with a table initially consisting of log entries for versions \[0, 19] and a checkpoint at verison 10, if the log entry for version 0 is cleaned up, then you cannot time travel to versions \[1, 9]. Increasing the table property `delta.logRetentionDuration` can help avoid these situations. ## Write to table [Section titled “Write to table”](#write-to-table) ### Append [Section titled “Append”](#append) To atomically add new data to an existing Delta table, use `append` mode: * SQL ```sql INSERT INTO default.people10m SELECT * FROM morePeople ``` * Python ```python df.write.format("delta").mode("append").save("/tmp/delta/people10m") df.write.format("delta").mode("append").saveAsTable("default.people10m") ``` * Scala ```scala df.write.format("delta").mode("append").save("/tmp/delta/people10m") df.write.format("delta").mode("append").saveAsTable("default.people10m") import io.delta.implicits._ df.write.mode("append").delta("/tmp/delta/people10m") ``` ### Overwrite [Section titled “Overwrite”](#overwrite) To atomically replace all the data in a table, use `overwrite` mode: * SQL ```sql INSERT OVERWRITE TABLE default.people10m SELECT * FROM morePeople ``` * Python ```python df.write.format("delta").mode("overwrite").save("/tmp/delta/people10m") df.write.format("delta").mode("overwrite").saveAsTable("default.people10m") ``` * Scala ```scala df.write.format("delta").mode("overwrite").save("/tmp/delta/people10m") df.write.format("delta").mode("overwrite").saveAsTable("default.people10m") import io.delta.implicits._ df.write.mode("overwrite").delta("/tmp/delta/people10m") ``` Delta Lake provides several options to selectively overwrite only part of a table. The following table summarizes them. For most use cases, we recommend using `replaceUsing` or `replaceWhere`. Use `replaceOn` only when your use case requires complex or null-safe matching conditions. | Option | Use case | Available since | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | | `replaceWhere` | Atomically overwrite rows that match a predicate. Use for replacements with a fixed matching condition, such as `colA = 5` or `int_col IN (1, 2, 3)`. | Delta Lake 1.1.0 (DataFrames); 2.4.0 (SQL) | | `replaceUsing` | Dynamic data overwrite by column equality. Replaces all rows that match the specified columns, based on equality comparison of the column values, in the provided data set. | Delta Lake 4.3.0 | | `replaceOn` | Dynamic data overwrite by Boolean expression. Use for replacements with a complex or null-safe matching condition, such as `s.colA <=> t.colA AND s.colB <=> t.colB`. | Delta Lake 4.3.0 | | `partitionOverwriteMode` | Dynamic partition overwrite, which overwrites all the existing data in each partition for which the write will commit new data. Legacy since Delta Lake 4.3.0; not recommended for new workloads. | Delta Lake 2.0 | #### Overwrite rows matching a predicate with `replaceWhere` [Section titled “Overwrite rows matching a predicate with replaceWhere”](#overwrite-rows-matching-a-predicate-with-replacewhere) `replaceWhere` atomically overwrites the rows that match a predicate, leaving all other rows unchanged. Use it for replacements with a fixed matching condition, such as `colA = 5` or `int_col IN (1, 2, 3)`. This feature is available with DataFrames in Delta Lake 1.1.0 and above and supported in SQL in Delta Lake 2.4.0 and above. The following command atomically replaces events in January in the target table, which is partitioned by `start_date`, with the data in `replace_data`: * SQL ```sql INSERT INTO TABLE events REPLACE WHERE start_data >= '2017-01-01' AND end_date <= '2017-01-31' SELECT * FROM replace_data ``` * Python ```python replace_data.write \ .format("delta") \ .mode("overwrite") \ .option("replaceWhere", "start_date >= '2017-01-01' AND end_date <= '2017-01-31'") \ .save("/tmp/delta/events") ``` * Scala ```scala replace_data.write .format("delta") .mode("overwrite") .option("replaceWhere", "start_date >= '2017-01-01' AND end_date <= '2017-01-31'") .save("/tmp/delta/events") ``` This sample code writes out the data in `replace_data`, validates that it all matches the predicate, and performs an atomic replacement. If you want to write out data that doesn’t all match the predicate, to replace the matching rows in the target table, you can disable the constraint check by setting `spark.databricks.delta.replaceWhere.constraintCheck.enabled` to false: * SQL ```sql SET spark.databricks.delta.replaceWhere.constraintCheck.enabled=false ``` * Python ```python spark.conf.set("spark.databricks.delta.replaceWhere.constraintCheck.enabled", False) ``` * Scala ```scala spark.conf.set("spark.databricks.delta.replaceWhere.constraintCheck.enabled", false) ``` In Delta Lake 1.0.0 and below, `replaceWhere` overwrites data matching a predicate over partition columns only. The following command atomically replaces the month in January in the target table, which is partitioned by `date`, with the data in `df`: * Python ```python df.write \ .format("delta") \ .mode("overwrite") \ .option("replaceWhere", "birthDate >= '2017-01-01' AND birthDate <= '2017-01-31'") \ .save("/tmp/delta/people10m") ``` * Scala ```scala df.write .format("delta") .mode("overwrite") .option("replaceWhere", "birthDate >= '2017-01-01' AND birthDate <= '2017-01-31'") .save("/tmp/delta/people10m") ``` In Delta Lake 1.1.0 and above, if you want to fall back to the old behavior, you can disable the `spark.databricks.delta.replaceWhere.dataColumns.enabled` flag: * SQL ```sql SET spark.databricks.delta.replaceWhere.dataColumns.enabled=false ``` * Python ```python spark.conf.set("spark.databricks.delta.replaceWhere.dataColumns.enabled", False) ``` * Scala ```scala spark.conf.set("spark.databricks.delta.replaceWhere.dataColumns.enabled", false) ``` #### Dynamically overwrite data with `replaceUsing` and `replaceOn` [Section titled “Dynamically overwrite data with replaceUsing and replaceOn”](#dynamically-overwrite-data-with-replaceusing-and-replaceon) Dynamic data overwrites selectively replace the data that matches the specified key columns or Boolean expression, leaving all other data unchanged. The `replaceUsing` option matches on key columns, while `replaceOn` matches on a Boolean expression. Partitioned tables, unpartitioned tables, and tables with liquid clustering are all supported. Both options are available with DataFrames in Delta Lake 4.3.0 and above. Dynamic partition overwrites (the legacy [`partitionOverwriteMode`](#dynamic-partition-overwrites-with-partitionoverwritemode-legacy-since-delta-lake-430) option) are a subset of dynamic data overwrite behavior. Dynamic partition overwrites replace all the existing data in each partition for which the write will commit new data, and leave all other partitions unchanged. Only partitioned tables are supported in dynamic partition overwrite mode. ##### replaceUsing [Section titled “replaceUsing”](#replaceusing) `replaceUsing` atomically deletes the rows in the target table that match a row in the incoming DataFrame on the specified columns, then inserts the incoming DataFrame rows. Matching is based on equality comparison of the specified column values. Provide a comma-separated list of columns. Each column must exist in both the table and the incoming DataFrame, at the same ordinal position. The following command replaces rows whose `event_id` and `start_date` match between the incoming DataFrame and the target table: * Python ```python source_data.write \ .format("delta") \ .mode("overwrite") \ .option("replaceUsing", "event_id, start_date") \ .saveAsTable("events") ``` * Scala ```scala source_data.write .format("delta") .mode("overwrite") .option("replaceUsing", "event_id, start_date") .saveAsTable("events") ``` If the source query is empty, no data is deleted. Like a `JOIN USING`, `replaceUsing` matches rows with regular equality, where `NULL` is not equal to anything. Rows with `NULL` values in the specified columns don’t match and aren’t removed from the target table. For null-safe matching, or other logic that equality cannot express, use `replaceOn` instead. For example, suppose the `students` table is clustered by `country`: | name | country | | ----- | ------- | | Dylan | US | | Doug | UK | | Julia | IT | | Adam | `NULL` | and the `new_students` DataFrame holds the replacement rows: | name | country | | ------ | ------- | | Peter | FR | | Jennie | UK | | Eva | `NULL` | * Python ```python # Replace rows in students whose country matches a row in new_students. new_students.write \ .format("delta") \ .mode("overwrite") \ .option("replaceUsing", "country") \ .saveAsTable("students") ``` * Scala ```scala // Replace rows in students whose country matches a row in new_students. new_students.write .format("delta") .mode("overwrite") .option("replaceUsing", "country") .saveAsTable("students") ``` After the write, `students` contains: | name | country | | ------ | ------- | | Dylan | US | | Julia | IT | | Jennie | UK | | Peter | FR | | Adam | `NULL` | | Eva | `NULL` | `Doug` is removed because `UK` matches a row in the incoming DataFrame. `Adam` keeps his row because his `NULL` country never matches, and `Eva`’s `NULL` row is appended for the same reason. ##### replaceOn [Section titled “replaceOn”](#replaceon) `replaceOn` replaces rows when they match a user-defined condition, unlike `replaceUsing`, which replaces rows when the specified columns compare equal under equality. Use `replaceOn` when you need matching logic that `replaceUsing` does not support, such as treating `NULL` values as equal with the `<=>` operator. Optionally, use the `targetAlias` option to specify an alias for the target table and the `.as()` or `.alias()` APIs to specify an alias for the source data. The following command replaces rows whose `event_id` and `start_date` match between the incoming DataFrame and the target table: * Python ```python source_data.alias("s").write \ .format("delta") \ .mode("overwrite") \ .option("targetAlias", "t") \ .option("replaceOn", "s.event_id <=> t.event_id AND s.start_date <=> t.start_date") \ .saveAsTable("events") ``` * Scala ```scala source_data.as("s").write .format("delta") .mode("overwrite") .option("targetAlias", "t") .option("replaceOn", "s.event_id <=> t.event_id AND s.start_date <=> t.start_date") .saveAsTable("events") ``` If the source query is empty, no data is deleted. For example, suppose the `students` table contains: | name | row\_origin | | ------ | ----------- | | Alice | table | | `NULL` | table | | Bob | table | and the `people` DataFrame holds the replacement rows: | name | row\_origin | | ------ | ----------- | | Alice | query | | `NULL` | query | | Delta | query | * Python ```python # Replace rows in students whose name matches a row in people, using a null-safe comparison. people.alias("s").write \ .format("delta") \ .mode("overwrite") \ .option("targetAlias", "t") \ .option("replaceOn", "t.name <=> s.name") \ .saveAsTable("students") ``` * Scala ```scala // Replace rows in students whose name matches a row in people, using a null-safe comparison. people.as("s").write .format("delta") .mode("overwrite") .option("targetAlias", "t") .option("replaceOn", "t.name <=> s.name") .saveAsTable("students") ``` After the write, `students` contains: | name | row\_origin | | ------ | ----------- | | Alice | query | | `NULL` | query | | Bob | table | | Delta | query | Because `<=>` is null-safe, the `NULL` row matches and is replaced. `Bob` has no match in the incoming DataFrame, so his row is kept, and `Delta` is appended. Note The `replaceOn` and `replaceUsing` options can’t be combined with `replaceWhere` or dynamic partition overwrite (`partitionOverwriteMode`) in the same write. #### Dynamic Partition Overwrites with `partitionOverwriteMode` (legacy since Delta Lake 4.3.0) [Section titled “Dynamic Partition Overwrites with partitionOverwriteMode (legacy since Delta Lake 4.3.0)”](#dynamic-partition-overwrites-with-partitionoverwritemode-legacy-since-delta-lake-430) Delta Lake 2.0 and above supports *dynamic* partition overwrite mode for partitioned tables. When in dynamic partition overwrite mode, we overwrite all existing data in each logical partition for which the write will commit new data. Any existing logical partitions for which the write does not contain data will remain unchanged. This mode is only applicable when data is being written in overwrite mode: either `INSERT OVERWRITE` in SQL, or a DataFrame write with `df.write.mode("overwrite")`. Configure dynamic partition overwrite mode by setting the Spark session configuration `spark.sql.sources.partitionOverwriteMode` to `dynamic`. You can also enable this by setting the `DataFrameWriter` option `partitionOverwriteMode` to `dynamic`. If present, the query-specific option overrides the mode defined in the session configuration. The default for `partitionOverwriteMode` is `static`. * SQL ```sql SET spark.sql.sources.partitionOverwriteMode=dynamic; INSERT OVERWRITE TABLE default.people10m SELECT * FROM morePeople; ``` * Python ```python df.write \ .format("delta") \ .mode("overwrite") \ .option("partitionOverwriteMode", "dynamic") \ .saveAsTable("default.people10m") ``` * Scala ```scala df.write .format("delta") .mode("overwrite") .option("partitionOverwriteMode", "dynamic") .saveAsTable("default.people10m") ``` Note Dynamic partition overwrite conflicts with the option `replaceWhere` for partitioned tables. * If dynamic partition overwrite is enabled in the Spark session configuration, and `replaceWhere` is provided as a `DataFrameWriter` option, then Delta Lake overwrites the data according to the `replaceWhere` expression (query-specific options override session configurations). * You’ll receive an error if the `DataFrameWriter` options have both dynamic partition overwrite and `replaceWhere` enabled. Important Validate that the data written with dynamic partition overwrite touches only the expected partitions. A single row in the incorrect partition can lead to unintentionally overwriting an entire partition. If a partition has been accidentally overwritten, you can use [Restore a Delta table to an earlier state](/delta-utility/#restore-a-delta-table-to-an-earlier-state) to undo the change. For Delta Lake support for updating tables, see [Table deletes, updates, and merges](/delta-update/). ### Limit rows written in a file [Section titled “Limit rows written in a file”](#limit-rows-written-in-a-file) You can use the SQL session configuration `spark.sql.files.maxRecordsPerFile` to specify the maximum number of records to write to a single file for a Delta Lake table. Specifying a value of zero or a negative value represents no limit. You can also use the DataFrameWriter option `maxRecordsPerFile` when using the DataFrame APIs to write to a Delta Lake table. When `maxRecordsPerFile` is specified, the value of the SQL session configuration `spark.sql.files.maxRecordsPerFile` is ignored. * Python ```python df.write.format("delta") \ .mode("append") \ .option("maxRecordsPerFile", "10000") \ .save("/tmp/delta/people10m") ``` * Scala ```scala df.write.format("delta") .mode("append") .option("maxRecordsPerFile", "10000") .save("/tmp/delta/people10m") ``` ### Idempotent writes [Section titled “Idempotent writes”](#idempotent-writes) Sometimes a job that writes data to a Delta table is restarted due to various reasons (for example, job encounters a failure). The failed job may or may not have written the data to Delta table before terminating. In the case where the data is written to the Delta table, the restarted job writes the same data to the Delta table which results in duplicate data. To address this, Delta tables support the following `DataFrameWriter` options to make the writes idempotent: * `txnAppId`: A unique string that you can pass on each `DataFrame` write. For example, this can be the name of the job. * `txnVersion`: A monotonically increasing number that acts as transaction version. This number needs to be unique for data that is being written to the Delta table(s). For example, this can be the epoch seconds of the instant when the query is attempted for the first time. Any subsequent restarts of the same job needs to have the same value for `txnVersion`. The above combination of options needs to be unique for each new data that is being ingested into the Delta table and the `txnVersion` needs to be higher than the last data that was ingested into the Delta table. For example: * Last successfully written data contains option values as `dailyETL:23423` (`txnAppId:txnVersion`). * Next write of data should have `txnAppId = dailyETL` and `txnVersion` as at least `23424` (one more than the last written data `txnVersion`). * Any attempt to write data with `txnAppId = dailyETL` and `txnVersion` as `23422` or less is ignored because the `txnVersion` is less than the last recorded `txnVersion` in the table. * Attempt to write data with `txnAppId:txnVersion` as `anotherETL:23424` is successful writing data to the table as it contains a different `txnAppId` compared to the same option value in last ingested data. You can also configure idempotent writes by setting the Spark session configuration `spark.databricks.delta.write.txnAppId` and `spark.databricks.delta.write.txnVersion`. In addition, you can set `spark.databricks.delta.write.txnVersion.autoReset.enabled` to true to automatically reset `spark.databricks.delta.write.txnVersion` after every write. When both the writer options and session configuration are set, we will use the writer option values. Warning This solution assumes that the data being written to Delta table(s) in multiple retries of the job is same. If a write attempt in a Delta table succeeds but due to some downstream failure there is a second write attempt with same txn options but different data, then that second write attempt will be ignored. This can cause unexpected results. #### Example [Section titled “Example”](#example-1) * SQL ```sql SET spark.databricks.delta.write.txnAppId = ...; SET spark.databricks.delta.write.txnVersion = ...; SET spark.databricks.delta.write.txnVersion.autoReset.enabled = true; -- if set to true, this will reset txnVersion after every write ``` * Python ```python app_id = ... # A unique string that is used as an application ID. version = ... # A monotonically increasing number that acts as transaction version. dataFrame.write.format(...).option("txnVersion", version).option("txnAppId", app_id).save(...) ``` * Scala ```scala val appId = ... // A unique string that is used as an application ID. version = ... // A monotonically increasing number that acts as transaction version. dataFrame.write.format(...).option("txnVersion", version).option("txnAppId", appId).save(...) ``` ### Set user-defined commit metadata [Section titled “Set user-defined commit metadata”](#set-user-defined-commit-metadata) You can specify user-defined strings as metadata in commits made by these operations, either using the DataFrameWriter option `userMetadata` or the SparkSession configuration `spark.databricks.delta.commitInfo.userMetadata`. If both of them have been specified, then the option takes preference. This user-defined metadata is readable in the [history](/delta-utility/#retrieve-delta-table-history) operation. * SQL ```sql SET spark.databricks.delta.commitInfo.userMetadata=overwritten-for-fixing-incorrect-data INSERT OVERWRITE default.people10m SELECT * FROM morePeople ``` * Python ```python df.write.format("delta") \ .mode("overwrite") \ .option("userMetadata", "overwritten-for-fixing-incorrect-data") \ .save("/tmp/delta/people10m") ``` * Scala ```scala df.write.format("delta") .mode("overwrite") .option("userMetadata", "overwritten-for-fixing-incorrect-data") .save("/tmp/delta/people10m") ``` ## Schema validation [Section titled “Schema validation”](#schema-validation) Delta Lake automatically validates that the schema of the DataFrame being written is compatible with the schema of the table. Delta Lake uses the following rules to determine whether a write from a DataFrame to a table is compatible: * All DataFrame columns must exist in the target table. If there are columns in the DataFrame not present in the table, an exception is raised. Columns present in the table but not in the DataFrame are set to null. * DataFrame column data types must match the column data types in the target table. If they don’t match, an exception is raised. * DataFrame column names cannot differ only by case. This means that you cannot have columns such as “Foo” and “foo” defined in the same table. While you can use Spark in case sensitive or insensitive (default) mode, Parquet is case sensitive when storing and returning column information. Delta Lake is case-preserving but insensitive when storing the schema and has this restriction to avoid potential mistakes, data corruption, or loss issues. Delta Lake support DDL to add new columns explicitly and the ability to update schema automatically. If you specify other options, such as `partitionBy`, in combination with append mode, Delta Lake validates that they match and throws an error for any mismatch. When `partitionBy` is not present, appends automatically follow the partitioning of the existing data. ## Update table schema [Section titled “Update table schema”](#update-table-schema) Delta Lake lets you update the schema of a table. The following types of changes are supported: * Adding new columns (at arbitrary positions) * Reordering existing columns You can make these changes explicitly using DDL or implicitly using DML. Important When you update a Delta table schema, streams that read from that table terminate. If you want the stream to continue you must restart it. ### Explicitly update schema [Section titled “Explicitly update schema”](#explicitly-update-schema) You can use the following DDL to explicitly change the schema of a table. #### Add columns [Section titled “Add columns”](#add-columns) * SQL ```sql ALTER TABLE table_name ADD COLUMNS (col_name data_type [COMMENT col_comment] [FIRST|AFTER colA_name], ...) ``` By default, nullability is `true`. To add a column to a nested field, use: * SQL ```sql ALTER TABLE table_name ADD COLUMNS (col_name.nested_col_name data_type [COMMENT col_comment] [FIRST|AFTER colA_name], ...) ``` ##### Example [Section titled “Example”](#example-2) If the schema before running `ALTER TABLE boxes ADD COLUMNS (colB.nested STRING AFTER field1)` is: ```plaintext - root | - colA | - colB | +-field1 | +-field2 ``` the schema after is: ```plaintext - root | - colA | - colB | +-field1 | +-nested | +-field2 ``` Note Adding nested columns is supported only for structs. Arrays and maps are not supported. #### Change column comment or ordering [Section titled “Change column comment or ordering”](#change-column-comment-or-ordering) * SQL ```sql ALTER TABLE table_name ALTER [COLUMN] col_name col_name data_type [COMMENT col_comment] [FIRST|AFTER colA_name] ``` To change a column in a nested field, use: * SQL ```sql ALTER TABLE table_name ALTER [COLUMN] col_name.nested_col_name nested_col_name data_type [COMMENT col_comment] [FIRST|AFTER colA_name] ``` ##### Example [Section titled “Example”](#example-3) If the schema before running `ALTER TABLE boxes CHANGE COLUMN colB.field2 field2 STRING FIRST` is: ```plaintext - root | - colA | - colB | +-field1 | +-field2 ``` the schema after is: ```plaintext - root | - colA | - colB | +-field2 | +-field1 ``` #### Replace columns [Section titled “Replace columns”](#replace-columns) * SQL ```sql ALTER TABLE table_name REPLACE COLUMNS (col_name1 col_type1 [COMMENT col_comment1], ...) ``` ##### Example [Section titled “Example”](#example-4) When running the following DDL: * SQL ```sql ALTER TABLE boxes REPLACE COLUMNS (colC STRING, colB STRUCT, colA STRING) ``` if the schema before is: ```plaintext - root | - colA | - colB | +-field1 | +-field2 ``` the schema after is: ```plaintext - root | - colC | - colB | +-field2 | +-nested | +-field1 | - colA ``` #### Rename columns [Section titled “Rename columns”](#rename-columns) Note This feature is available in Delta Lake 1.2.0 and above. This feature is currently experimental. To rename columns without rewriting any of the columns’ existing data, you must enable column mapping for the table. See [enable column mapping](/delta-column-mapping/). To rename a column: * SQL ```sql ALTER TABLE table_name RENAME COLUMN old_col_name TO new_col_name ``` To rename a nested field: * SQL ```sql ALTER TABLE table_name RENAME COLUMN col_name.old_nested_field TO new_nested_field ``` ##### Example [Section titled “Example”](#example-5) When you run the following command: ```sql ALTER TABLE boxes RENAME COLUMN colB.field1 TO field001 ``` If the schema before is: ```plaintext - root | - colA | - colB | +-field1 | +-field2 ``` Then the schema after is: ```plaintext - root | - colA | - colB | +-field001 | +-field2 ``` #### Drop columns [Section titled “Drop columns”](#drop-columns) Note This feature is available in Delta Lake 2.0 and above. This feature is currently experimental. To drop columns as a metadata-only operation without rewriting any data files, you must enable column mapping for the table. See [enable column mapping](/delta-column-mapping/). Important Dropping a column from metadata does not delete the underlying data for the column in files. To drop a column: * SQL ```sql ALTER TABLE table_name DROP COLUMN col_name ``` To drop multiple columns: * SQL ```sql ALTER TABLE table_name DROP COLUMNS (col_name_1, col_name_2) ``` #### Change column type or name [Section titled “Change column type or name”](#change-column-type-or-name) You can change a column’s type or name or drop a column by rewriting the table. To do this, use the `overwriteSchema` option: ##### Change a column type [Section titled “Change a column type”](#change-a-column-type) * Python ```python spark.read.table(...) \ .withColumn("birthDate", col("birthDate").cast("date")) \ .write \ .format("delta") \ .mode("overwrite") .option("overwriteSchema", "true") \ .saveAsTable(...) ``` ##### Change a column name [Section titled “Change a column name”](#change-a-column-name) * Python ```python spark.read.table(...) \ .withColumnRenamed("dateOfBirth", "birthDate") \ .write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .saveAsTable(...) ``` ### Automatic schema update [Section titled “Automatic schema update”](#automatic-schema-update) Delta Lake can automatically update the schema of a table as part of a DML transaction (either appending or overwriting), and make the schema compatible with the data being written. #### Add columns [Section titled “Add columns”](#add-columns-1) Columns present in the source data but missing from the table are automatically added during a write transaction when schema evolution is enabled. The added columns are appended to the end of the struct they are present in. The case is preserved when a new column is appended. ##### Enable schema evolution (Delta Lake 4.3 and above) [Section titled “Enable schema evolution (Delta Lake 4.3 and above)”](#enable-schema-evolution-delta-lake-43-and-above) On Delta Lake 4.3 and above, use `WITH SCHEMA EVOLUTION` in a SQL `INSERT` statement or `.withSchemaEvolution()` in the DataFrame API to enable automatic schema evolution for a single operation. When enabled, the schema of the target Delta Lake table is automatically updated to accommodate additional columns or widened types of the source data. For a list of the supported type changes, see [Supported type changes](https://docs.delta.io/delta-type-widening/). For example: * SQL ```sql INSERT WITH SCHEMA EVOLUTION INTO target_table SELECT * FROM source_table ``` * Python ```python (spark.read .table("source_table") .write .mode("append") .withSchemaEvolution() .saveAsTable("target_table") ) ``` * Scala ```scala spark.read .table("source_table") .write .mode("append") .withSchemaEvolution() .saveAsTable("target_table") ``` If the source data contains columns that don’t exist in the target table, those columns are automatically added to the `target_table` schema. Existing rows receive `NULL` values for the new columns. ##### Enable schema evolution in earlier versions [Section titled “Enable schema evolution in earlier versions”](#enable-schema-evolution-in-earlier-versions) In Delta Lake versions earlier than 4.3, enable schema evolution using one of the following alternatives. Both are also available in Delta Lake 4.3 and above. Set the `mergeSchema` option on an individual `write` or `writeStream` operation: * Python ```python (spark.read .table("source_table") .write .option("mergeSchema", "true") .mode("append") .saveAsTable("target_table") ) ``` * Scala ```scala spark.read .table("source_table") .write .option("mergeSchema", "true") .mode("append") .saveAsTable("target_table") ``` Alternatively, set the Spark configuration `spark.databricks.delta.schema.autoMerge.enabled` to `true` to enable schema evolution for all write operations in the current `SparkSession`: * Python ```python spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", True) ``` * Scala ```scala spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled", true) ``` * SQL ```sql SET spark.databricks.delta.schema.autoMerge.enabled=true ``` Important Enabling schema evolution session-wide is not recommended because it can lead to unintended schema changes across multiple operations and makes it harder to reason about which operations evolve the schema. Instead, enable schema evolution for each individual operation using `WITH SCHEMA EVOLUTION`, `.withSchemaEvolution()`, or the `mergeSchema` option. When both an operation-level setting (the `mergeSchema` option or `WITH SCHEMA EVOLUTION` / `.withSchemaEvolution()`) and the session-wide Spark configuration are specified, the operation-level setting takes precedence. #### `NullType` columns [Section titled “NullType columns”](#nulltype-columns) Because Parquet doesn’t support `NullType`, `NullType` columns are dropped from the DataFrame when writing into Delta tables, but are still stored in the schema. When a different data type is received for that column, Delta Lake merges the schema to the new data type. If Delta Lake receives a `NullType` for an existing column, the old schema is retained and the new column is dropped during the write. `NullType` in streaming is not supported. Since you must set schemas when using streaming this should be very rare. `NullType` is also not accepted for complex types such as `ArrayType` and `MapType`. ## Replace table schema [Section titled “Replace table schema”](#replace-table-schema) By default, overwriting the data in a table does not overwrite the schema. When overwriting a table using mode `overwrite` without `replaceWhere`, you may still want to overwrite the schema of the data being written. You replace the schema and partitioning of the table by setting the `overwriteSchema` option to `true`: * Python ```python df.write.option("overwriteSchema", "true") ``` ## Views on tables [Section titled “Views on tables”](#views-on-tables) Delta Lake supports the creation of views on top of Delta tables just like you might with a data source table. The core challenge when you operate with views is resolving the schemas. If you alter a Delta table schema, you must recreate derivative views to account for any additions to the schema. For instance, if you add a new column to a Delta table, you must make sure that this column is available in the appropriate views built on top of that base table. ## Table properties [Section titled “Table properties”](#table-properties) You can store your own metadata as a table property using `TBLPROPERTIES` in `CREATE` and `ALTER`. You can then `SHOW` that metadata. For example: * SQL ```sql ALTER TABLE default.people10m SET TBLPROPERTIES ('department' = 'accounting', 'delta.appendOnly' = 'true'); -- Show the table's properties. SHOW TBLPROPERTIES default.people10m; -- Show just the 'department' table property. SHOW TBLPROPERTIES default.people10m ('department'); ``` `TBLPROPERTIES` are stored as part of Delta table metadata. You cannot define new `TBLPROPERTIES` in a `CREATE` statement if a Delta table already exists in a given location. In addition, to tailor behavior and performance, Delta Lake supports certain Delta table properties: * Block deletes and updates in a Delta table: `delta.appendOnly=true`. * Configure the [time travel](#query-an-older-snapshot-of-a-table-time-travel) retention properties: `delta.logRetentionDuration=` and `delta.deletedFileRetentionDuration=`. For details, see [Data retention](#data-retention). * Configure the number of columns for which statistics are collected: `delta.dataSkippingNumIndexedCols=n`. This property indicates to the writer that statistics are to be collected only for the first `n` columns in the table. Also the data skipping code ignores statistics for any column beyond this column index. This property takes affect only for new data that is written out. Note Modifying a Delta table property is a write operation that will conflict with other [concurrent write operations](/concurrency-control), causing them to fail. We recommend that you modify a table property only when there are no concurrent write operations on the table. You can also set `delta.`-prefixed properties during the first commit to a Delta table using Spark configurations. For example, to initialize a Delta table with the property `delta.appendOnly=true`, set the Spark configuration `spark.databricks.delta.properties.defaults.appendOnly` to `true`. For example: * SQL ```sql spark.sql("SET spark.databricks.delta.properties.defaults.appendOnly = true") ``` * Python ```python spark.conf.set("spark.databricks.delta.properties.defaults.appendOnly", "true") ``` * Scala ```scala spark.conf.set("spark.databricks.delta.properties.defaults.appendOnly", "true") ``` See also the [Delta table properties reference](/table-properties/). ## Syncing table schema and properties to the Hive metastore [Section titled “Syncing table schema and properties to the Hive metastore”](#syncing-table-schema-and-properties-to-the-hive-metastore) You can enable asynchronous syncing of table schema and properties to the metastore by setting `spark.databricks.delta.catalog.update.enabled` to `true`. Whenever the Delta client detects that either of these two were changed due to an update, it will sync the changes to the metastore. The schema is stored in the table properties in HMS. If the schema is small, it will be stored directly under the key `spark.sql.sources.schema`: * JSON ```json { "spark.sql.sources.schema": "{'name':'col1','type':'string','nullable':true, 'metadata':{}},{'name':'col2','type':'string','nullable':true,'metadata':{}}" } ``` If Schema is large, the schema will be broken down into multiple parts. Appending them together should give the correct schema. For example: * JSON ```json { "spark.sql.sources.schema.numParts": "4", "spark.sql.sources.schema.part.1": "{'name':'col1','type':'string','nullable':tr", "spark.sql.sources.schema.part.2": "ue, 'metadata':{}},{'name':'co", "spark.sql.sources.schema.part.3": "l2','type':'string','nullable':true,'meta", "spark.sql.sources.schema.part.4": "data':{}}" } ``` ## Table metadata [Section titled “Table metadata”](#table-metadata) Delta Lake has rich features for exploring table metadata. It supports `SHOW COLUMNS` and `DESCRIBE TABLE`. It also provides the following unique commands: ### `DESCRIBE DETAIL` [Section titled “DESCRIBE DETAIL”](#describe-detail) Provides information about schema, partitioning, table size, and so on. For details, see [Retrieve Delta table details](/delta-utility/#retrieve-delta-table-history). ### `DESCRIBE HISTORY` [Section titled “DESCRIBE HISTORY”](#describe-history) Provides provenance information, including the operation, user, and so on, and operation metrics for each write to a table. Table history is retained for 30 days. For details, see [Retrieve Delta table history](/delta-utility/#retrieve-delta-table-history). ## Configure SparkSession [Section titled “Configure SparkSession”](#configure-sparksession) For many Delta Lake operations, you enable integration with Apache Spark DataSourceV2 and Catalog APIs (since 3.0) by setting the following configurations when you create a new `SparkSession`. * Python ```python from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName("...") \ .master("...") \ .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \ .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \ .getOrCreate() ``` * Scala ```scala import org.apache.spark.sql.SparkSession val spark = SparkSession .builder() .appName("...") .master("...") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate() ``` * Java ```java import org.apache.spark.sql.SparkSession; SparkSession spark = SparkSession .builder() .appName("...") .master("...") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate(); ``` Alternatively, you can add configurations when submitting your Spark application using `spark-submit` or when starting `spark-shell` or `pyspark` by specifying them as command-line parameters. ```bash spark-submit --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ... ``` ```bash pyspark --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ``` ## Configure storage credentials [Section titled “Configure storage credentials”](#configure-storage-credentials) Delta Lake uses Hadoop FileSystem APIs to access storage systems. The credentials for storage systems usually can be set through Hadoop configurations. Delta Lake provides multiple ways to set Hadoop configurations similar to Apache Spark. ### Spark configurations [Section titled “Spark configurations”](#spark-configurations) When you start a Spark application on a cluster, you can set the Spark configurations in the form of `spark.hadoop.*` to pass your custom Hadoop configurations. For example, setting a value for `spark.hadoop.a.b.c` will pass the value as a Hadoop configuration `a.b.c`, and Delta Lake will use it to access Hadoop FileSystem APIs. See [Spark documentation](http://spark.apache.org/docs/latest/configuration.html#custom-hadoophive-configuration) for more details. ### SQL session configurations [Section titled “SQL session configurations”](#sql-session-configurations) Spark SQL will pass all of the current [SQL session configurations](http://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration) to Delta Lake, and Delta Lake will use them to access Hadoop FileSystem APIs. For example, `SET a.b.c=x.y.z` will tell Delta Lake to pass the value `x.y.z` as a Hadoop configuration `a.b.c`, and Delta Lake will use it to access Hadoop FileSystem APIs. ### DataFrame options [Section titled “DataFrame options”](#dataframe-options) Besides setting Hadoop file system configurations through the Spark (cluster) configurations or SQL session configurations, Delta supports reading Hadoop file system configurations from `DataFrameReader` and `DataFrameWriter` options (that is, option keys that start with the `fs.` prefix) when the table is read or written, by using `DataFrameReader.load(path)` or `DataFrameWriter.save(path)`. For example, you can pass your storage credentials through DataFrame options: * Python ```python df1 = spark.read.format("delta") \ .option("fs.azure.account.key..dfs.core.windows.net", "") \ .read("...") df2 = spark.read.format("delta") \ .option("fs.azure.account.key..dfs.core.windows.net", "") \ .read("...") df1.union(df2).write.format("delta") \ .mode("overwrite") \ .option("fs.azure.account.key..dfs.core.windows.net", "") \ .save("...") ``` * Scala ```scala val df1 = spark.read.format("delta") .option("fs.azure.account.key..dfs.core.windows.net", "") .read("...") val df2 = spark.read.format("delta") .option("fs.azure.account.key..dfs.core.windows.net", "") .read("...") df1.union(df2).write.format("delta") .mode("overwrite") .option("fs.azure.account.key..dfs.core.windows.net", "") .save("...") ``` You can find the details of the Hadoop file system configurations for your storage in [Storage configuration](/delta-storage/). # Use catalog-managed tables > Learn how to enable and use catalog-managed commits in Delta Lake. Note This feature is available in Delta Lake 4.0.1 and above. [Catalog-Managed Tables](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#catalog-managed-tables) introduce a `catalogManaged` reader-writer table feature that changes how Delta Lake discovers and accesses tables. With this feature enabled, the catalog coordinates commit atomicity, allowing for features like multi-table transactions that are difficult to achieve with filesystem-only primitives. ## Overview [Section titled “Overview”](#overview) By default, Delta Lake relies entirely on the filesystem for read-time discovery and write-time commit atomicity. Each table manages its own transaction logs and conflict detection independently. Catalog-managed tables shift this responsibility to the managing catalog, which allows the catalog to orchestrate commits across multiple tables within a single transaction boundary while maintaining Delta Lake’s ACID guarantees. Important Filesystem-based access to catalog-managed tables is not supported. Delta clients must discover and access these tables through the managing catalog, not by direct path-based access. This ensures consistency across distributed environments. Users can use any catalog implementation that supports the Delta Catalog-Managed Table protocol. ## Requirements [Section titled “Requirements”](#requirements) * Catalog-managed tables requires the following Delta protocols: * Reader version 3 or above. * Writer version 7 or above. * The [In-Commit Timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps) table feature must be enabled, as commit publishing can occur asynchronously and file modification timestamps may not reflect actual commit times. * The [VACUUM Protocol Check](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#vacuum-protocol-check) table feature must be enabled to provide proper vacuum operations on catalog-managed tables. ## Enable catalog-managed commits [Section titled “Enable catalog-managed commits”](#enable-catalog-managed-commits) You can enable catalog-managed commits for new tables when using a catalog that supports this feature, such as [Unity Catalog](https://www.unitycatalog.io/). ### Enable catalog-managed commits for new tables [Section titled “Enable catalog-managed commits for new tables”](#enable-catalog-managed-commits-for-new-tables) Enable the `catalogManaged` table feature by setting the following table property when creating a table: * SQL ```sql CREATE TABLE sales_data ( sale_id BIGINT, amount DECIMAL(10,2), sale_date DATE ) TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); ``` Warning When you enable catalog-managed commits, the table protocol version is upgraded. After upgrading, the table will not be readable by Delta Lake clients that do not support catalog-managed tables. ## Check if catalog-managed commits are enabled [Section titled “Check if catalog-managed commits are enabled”](#check-if-catalog-managed-commits-are-enabled) To verify whether a table has catalog-managed commits enabled: * SQL ```sql DESCRIBE DETAIL sales_data; ``` If enabled, `catalogManaged` appears in the `tableFeatures` column. ## Limitations [Section titled “Limitations”](#limitations) * Catalog-managed tables cannot be enabled on existing tables. Once enabled, the feature cannot be disabled. * `CREATE OR REPLACE TABLE` is not supported for tables with catalog-managed commits enabled. # Change data feed > Learn how to get row-level change information from Delta tables using the Delta change data feed. Change Data Feed (CDF) feature allows Delta tables to track row-level changes between versions of a Delta table. When enabled on a Delta table, the runtime records “change events” for all the data written into the table. This includes the row data along with metadata indicating whether the specified row was inserted, deleted, or updated. You can read the change events in batch queries using DataFrame APIs (that is, `df.read`) and in streaming queries using DataFrame APIs (that is, `df.readStream`). ## Use cases [Section titled “Use cases”](#use-cases) Change Data Feed is not enabled by default. The following use cases should drive when you enable the change data feed. * **Silver and Gold tables**: Improve Delta performance by processing only row-level changes following initial `MERGE`, `UPDATE`, or `DELETE` operations to accelerate and simplify ETL and ELT operations. * **Transmit changes**: Send a change data feed to downstream systems such as Kafka or RDBMS that can use it to incrementally process in later stages of data pipelines. * **Audit trail table**: Capture the change data feed as a Delta table provides perpetual storage and efficient query capability to see all changes over time, including when deletes occur and what updates were made. ## Enable change data feed [Section titled “Enable change data feed”](#enable-change-data-feed) You must explicitly enable the change data feed option using one of the following methods: * **New table**: Set the table property `delta.enableChangeDataFeed = true` in the `CREATE TABLE` command. * SQL ```sql CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIES (delta.enableChangeDataFeed = true) ``` * **Existing table**: Set the table property `delta.enableChangeDataFeed = true` in the `ALTER TABLE` command. * SQL ```sql ALTER TABLE myDeltaTable SET TBLPROPERTIES (delta.enableChangeDataFeed = true) ``` * **All new tables**: * SQL ```sql set spark.databricks.delta.properties.defaults.enableChangeDataFeed = true; ``` Important Once you enable the change data feed option for a table, you can no longer write to the table using Delta Lake 1.2.1 or below. You can always read the table. Only changes made *after* you enable the change data feed are recorded; past changes to a table are not captured. ### Change data storage [Section titled “Change data storage”](#change-data-storage) Delta Lake records change data for `UPDATE`, `DELETE`, and `MERGE` operations in the `_change_data` folder under the Delta table directory. These records may be skipped when Delta Lake detects it can efficiently compute the change data feed directly from the transaction log. In particular, insert-only operations and full partition deletes will not generate data in the `_change_data` directory. The files in the `_change_data` folder follow the retention policy of the table. Therefore, if you run the [VACUUM](/delta-utility/#remove-files-no-longer-referenced-by-a-delta-table) command, change data feed data is also deleted. ## Read changes in batch queries [Section titled “Read changes in batch queries”](#read-changes-in-batch-queries) You can provide either version or timestamp for the start and end. The start and end versions and timestamps are inclusive in the queries. To read the changes from a particular start version to the *latest* version of the table, specify only the starting version or timestamp. You specify a version as an integer and a timestamps as a string in the format `yyyy-MM-dd[ HH:mm:ss[.SSS]]`. If you provide a version lower or timestamp older than one that has recorded change events, that is, when the change data feed was enabled, an error is thrown indicating that the change data feed was not enabled. * SQL ```sql -- version as ints or longs e.g. changes from version 0 to 10 SELECT * FROM table_changes('tableName', 0, 10) -- timestamp as string formatted timestamps SELECT * FROM table_changes('tableName', '2021-04-21 05:45:46', '2021-05-21 12:00:00') -- providing only the startingVersion/timestamp SELECT * FROM table_changes('tableName', 0) -- database/schema names inside the string for table name, with backticks for escaping dots and special characters SELECT * FROM table_changes('dbName.`dotted.tableName`', '2021-04-21 06:45:46' , '2021-05-21 12:00:00') -- path based tables SELECT * FROM table_changes_by_path('\path', '2021-04-21 05:45:46') ``` * Python ```python # version as ints or longs spark.read.format("delta") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .option("endingVersion", 10) \ .table("myDeltaTable") # timestamps as formatted timestamp spark.read.format("delta") \ .option("readChangeFeed", "true") \ .option("startingTimestamp", '2021-04-21 05:45:46') \ .option("endingTimestamp", '2021-05-21 12:00:00') \ .table("myDeltaTable") # providing only the startingVersion/timestamp spark.read.format("delta") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .table("myDeltaTable") # path based tables spark.read.format("delta") \ .option("readChangeFeed", "true") \ .option("startingTimestamp", '2021-04-21 05:45:46') \ .load("pathToMyDeltaTable") ``` * Scala ```scala // version as ints or longs spark.read.format("delta") .option("readChangeFeed", "true") .option("startingVersion", 0) .option("endingVersion", 10) .table("myDeltaTable") // timestamps as formatted timestamp spark.read.format("delta") .option("readChangeFeed", "true") .option("startingTimestamp", "2021-04-21 05:45:46") .option("endingTimestamp", "2021-05-21 12:00:00") .table("myDeltaTable") // providing only the startingVersion/timestamp spark.read.format("delta") .option("readChangeFeed", "true") .option("startingVersion", 0) .table("myDeltaTable") // path based tables spark.read.format("delta") .option("readChangeFeed", "true") .option("startingTimestamp", "2021-04-21 05:45:46") .load("pathToMyDeltaTable") ``` ## Read changes in streaming queries [Section titled “Read changes in streaming queries”](#read-changes-in-streaming-queries) * Python ```python # providing a starting version spark.readStream.format("delta") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .table("myDeltaTable") # providing a starting timestamp spark.readStream.format("delta") \ .option("readChangeFeed", "true") \ .option("startingTimestamp", "2021-04-21 05:35:43") \ .load("/pathToMyDeltaTable") # not providing a starting version/timestamp will result in the latest snapshot being fetched first spark.readStream.format("delta") \ .option("readChangeFeed", "true") \ .table("myDeltaTable") ``` * Scala ```scala // providing a starting version spark.readStream.format("delta") .option("readChangeFeed", "true") .option("startingVersion", 0) .table("myDeltaTable") // providing a starting timestamp spark.readStream.format("delta") .option("readChangeFeed", "true") .option("startingVersion", "2021-04-21 05:35:43") .load("/pathToMyDeltaTable") // not providing a starting version/timestamp will result in the latest snapshot being fetched first spark.readStream.format("delta") .option("readChangeFeed", "true") .table("myDeltaTable") ``` To get the change data while reading the table, set the option `readChangeFeed` to `true`. The `startingVersion` or `startingTimestamp` are optional and if not provided the stream returns the latest snapshot of the table at the time of streaming as an `INSERT` and future changes as change data. Options like rate limits (`maxFilesPerTrigger`, `maxBytesPerTrigger`) and `excludeRegex` are also supported when reading change data. Note Rate limiting can be atomic for versions other than the starting snapshot version. That is, the entire commit version will be rate limited or the entire commit will be returned. By default if a user passes in a version or timestamp exceeding the last commit on a table, the error `timestampGreaterThanLatestCommit` will be thrown. CDF can handle the out of range version case, if the user sets the following configuration to `true`. * Python ```sql set spark.databricks.delta.changeDataFeed.timestampOutOfRange.enabled = true; ``` If you provide a start version greater than the last commit on a table or a start timestamp newer than the last commit on a table, then when the preceding configuration is enabled, an empty read result is returned. If you provide an end version greater than the last commit on a table or an end timestamp newer than the last commit on a table, then when the preceding configuration is enabled in batch read mode, all changes between the start version and the last commit are be returned. ## What is the schema for the change data feed? [Section titled “What is the schema for the change data feed?”](#what-is-the-schema-for-the-change-data-feed) When you read from the change data feed for a table, the schema for the latest table version is used. Note Most schema change and evolution operations are fully supported. Tables with column mapping enabled do not support all use cases and demonstrate different behavior. See [Change data feed limitations for tables with column mapping enabled](#change-data-feed-limitations-for-tables-with-column-mapping-enabled). In addition to the data columns from the schema of the Delta table, change data feed contains metadata columns that identify the type of change event: | Column name | Type | Values | | :------------------ | :-------- | :-------------------------------------------------------------------- | | `_change_type` | String | `insert`, `update_preimage` , `update_postimage`, `delete` [(1)](#-1) | | `_commit_version` | Long | The Delta log or table version containing the change. | | `_commit_timestamp` | Timestamp | The timestamp associated when the commit was created. | []()**(1)** `preimage` is the value before the update, `postimage` is the value after the update. ## Change data feed limitations for tables with column mapping enabled [Section titled “Change data feed limitations for tables with column mapping enabled”](#change-data-feed-limitations-for-tables-with-column-mapping-enabled) With column mapping enabled on a Delta table, you can drop or rename columns in the table without rewriting data files for existing data. With column mapping enabled, change data feed has limitations after performing non-additive schema changes such as renaming or dropping a column, changing data type, or nullability changes. Important In Delta Lake 2.0 and before, tables with column mapping enabled do not support streaming reads or batch reads on change data feed. In Delta Lake 2.1, tables with column mapping enabled support batch reads on change data feed as long as there are no non-additive schema changes. Streaming reads of change data feed of tables with column mapping enabled is not supported. In Delta Lake 2.2, tables with column mapping enabled support both batch and streaming reads on change data feed as long as there are no non-additive schema changes. In Delta Lake 2.3 and above, you can perform batch reads on change data feed for tables with column mapping enabled that have experienced non-additive schema changes. Instead of using the schema of the latest version of the table, read operations use the schema of the end version of the table specified in the query. Queries still fail if the version range specified spans a non-additive schema change. In Delta Lake 3.0 and above, you can perform streaming read on change data feed for tables with column mapping enabled that have experienced non-additive schema changes by enabling [schema tracking](/delta-streaming/#tracking-non-additive-schema-changes). ## Frequently asked questions (FAQ) [Section titled “Frequently asked questions (FAQ)”](#frequently-asked-questions-faq) ### What is the overhead of enabling the change data feed? [Section titled “What is the overhead of enabling the change data feed?”](#what-is-the-overhead-of-enabling-the-change-data-feed) There is no significant impact. The change data records are generated in line during the query execution process, and are generally much smaller than the total size of rewritten files. ### What is the retention policy for change records? [Section titled “What is the retention policy for change records?”](#what-is-the-retention-policy-for-change-records) Change records follow the same retention policy as out-of-date table versions, and will be cleaned up through VACUUM if they are outside the specified retention period. ### When do new records become available in the change data feed? [Section titled “When do new records become available in the change data feed?”](#when-do-new-records-become-available-in-the-change-data-feed) Change data is committed along with the Delta Lake transaction, and will become available at the same time as the new data is available in the table. # Use liquid clustering for Delta tables > Learn about liquid clustering in Delta Lake. Liquid clustering improves the existing partitioning and `ZORDER` techniques by simplifying data layout decisions in order to optimize query performance. Liquid clustering provides flexibility to redefine clustering columns without rewriting existing data, allowing data layout to evolve alongside analytic needs over time. Note This feature is available in Delta Lake 3.1.0 and above. See [Limitations](#limitations). ## What is liquid clustering used for? [Section titled “What is liquid clustering used for?”](#what-is-liquid-clustering-used-for) The following are examples of scenarios that benefit from clustering: * Tables often filtered by high cardinality columns. * Tables with significant skew in data distribution. * Tables that grow quickly and require maintenance and tuning effort. * Tables with access patterns that change over time. * Tables where a typical partition column could leave the table with too many or too few partitions. ## Enable liquid clustering [Section titled “Enable liquid clustering”](#enable-liquid-clustering) You can enable liquid clustering on an existing table or during table creation. Clustering is not compatible with partitioning or `ZORDER`. Once enabled, run `OPTIMIZE` jobs as usual to incrementally cluster data. See [How to trigger clustering](#how-to-trigger-clustering). To enable liquid clustering, add the `CLUSTER BY` phrase to a table creation statement, as in the examples below: Note In Delta Lake 3.2 and above, you can use DeltaTable API in Python or Scala to enable liquid clustering. * SQL ```sql -- Create an empty table CREATE TABLE table1(col0 int, col1 string) USING DELTA CLUSTER BY (col0); -- Using a CTAS statement (Delta 3.3+) CREATE EXTERNAL TABLE table2 CLUSTER BY (col0) -- specify clustering after table name, not in subquery LOCATION 'table_location' AS SELECT * FROM table1; ``` * Python ```python # Create an empty table DeltaTable.create() .tableName("table1") .addColumn("col0", dataType = "INT") .addColumn("col1", dataType = "STRING") .clusterBy("col0") .execute() ``` * Scala ```scala // Create an empty table DeltaTable.create() .tableName("table1") .addColumn("col0", dataType = "INT") .addColumn("col1", dataType = "STRING") .clusterBy("col0") .execute() ``` Warning Tables created with liquid clustering have `Clustering` and `DomainMetadata` table features enabled (both writer features) and use Delta writer version 7 and reader version 1. Table protocol versions cannot be downgraded. See [How does Delta Lake manage feature compatibility?](/versioning/). In Delta Lake 3.3 and above you can enable liquid clustering on an existing unpartitioned Delta table using the following syntax: * SQL ```sql ALTER TABLE CLUSTER BY () ``` Important Default behavior does not apply clustering to previously written data. To force reclustering for all records, you must use `OPTIMIZE FULL`. See [Recluster entire table](#recluster-entire-table). ## Choose clustering columns [Section titled “Choose clustering columns”](#choose-clustering-columns) Clustering columns can be defined in any order. If two columns are correlated, you only need to add one of them as a clustering column. If you’re converting an existing table, consider the following recommendations: | Current data optimization technique | Recommendation for clustering columns | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | Hive-style partitioning | Use partition columns as clustering columns. | | Z-order indexing | Use the `ZORDER BY` columns as clustering columns. | | Hive-style partitioning and Z-order | Use both partition columns and `ZORDER BY` columns as clustering columns. | | Generated columns to reduce cardinality (for example, date for a timestamp) | Use the original column as a clustering column, and don’t create a generated column. | ## Write data to a clustered table [Section titled “Write data to a clustered table”](#write-data-to-a-clustered-table) You must use a Delta writer client that supports `Clustering` and `DomainMetadata` table features. ## How to trigger clustering [Section titled “How to trigger clustering”](#how-to-trigger-clustering) Use the `OPTIMIZE` command on your table, as in the following example: * SQL `sql OPTIMIZE table_name;` Liquid clustering is incremental, meaning that data is only rewritten as necessary to accommodate data that needs to be clustered. Already clustered data files with different clustering columns are not rewritten. ### Recluster entire table [Section titled “Recluster entire table”](#recluster-entire-table) In Delta Lake 3.3 and above, you can force reclustering of all records in a table with the following syntax: * SQL `sql OPTIMIZE table_name FULL;` Important Running `OPTIMIZE FULL` reclusters all existing data as necessary. For large tables that have not previously been clustered on the specified columns, this operation might take hours. Run `OPTIMIZE FULL` when you change clustering columns. If you have previously run `OPTIMIZE FULL` and there has been no change to clustering columns, `OPTIMIZE FULL` runs the same as `OPTIMIZE`. Always use `OPTIMIZE FULL` to ensure that data layout reflects the current clustering columns. ## Read data from a clustered table [Section titled “Read data from a clustered table”](#read-data-from-a-clustered-table) You can read data in a clustered table using any Delta Lake client. For best query results, include clustering columns in your query filters, as in the following example: * SQL ```sql ``` ## Change clustering columns [Section titled “Change clustering columns”](#change-clustering-columns) You can change clustering columns for a table at any time by running an `ALTER TABLE` command, as in the following example: * SQL `sql ALTER TABLE table_name CLUSTER BY (new_column1, new_column2);` When you change clustering columns, subsequent `OPTIMIZE` and write operations use the new clustering approach, but existing data is not rewritten. You can also turn off clustering by setting the columns to `NONE`, as in the following example: * SQL `sql ALTER TABLE table_name CLUSTER BY NONE;` Setting cluster columns to `NONE` does not rewrite data that has already been clustered, but prevents future `OPTIMIZE` operations from using clustering columns. ## See how table is clustered [Section titled “See how table is clustered”](#see-how-table-is-clustered) You can use `DESCRIBE DETAIL` commands to see the clustering columns for a table, as in the following examples: * SQL `sql DESCRIBE DETAIL table_name;` ## Limitations [Section titled “Limitations”](#limitations) The following limitations exist: * You can only specify columns with statistics collected for clustering columns. By default, the first 32 columns in a Delta table have statistics collected. * You can specify up to 4 clustering columns. Important In Delta Lake 3.1, users needs to enable the feature flag `spark.databricks.delta.clusteredTable.enableClusteringTablePreview` to use liquid clustering. The following features are not supported in this preview: * ZCube based incremental clustering * `ALTER TABLE ... CLUSTER BY` to change clustering columns * `DESCRIBE DETAIL` to inspect the current clustering columns In Delta Lake 3.2, the preview flag is removed and the above features are supported. # Delta column mapping > Learn about column mapping in Delta. Note This feature is available in Delta Lake 1.2.0 and above. This feature is currently experimental with [known limitations](#known-limitations). [Column mapping feature](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping) allows Delta table columns and the underlying Parquet file columns to use different names. This enables Delta schema evolution operations such as `RENAME COLUMN` and `DROP COLUMNS` on a Delta table without the need to rewrite the underlying Parquet files. It also allows users to name Delta table columns by using [characters that are not allowed](#supported-characters-in-column-names) by Parquet, such as spaces, so that users can directly ingest CSV or JSON data into Delta without the need to rename columns due to previous character constraints. ## How to enable Delta Lake column mapping [Section titled “How to enable Delta Lake column mapping”](#how-to-enable-delta-lake-column-mapping) Important Enabling column mapping for a table upgrades the Delta [table version](/versioning/#what-is-a-protocol-version). This protocol upgrade is irreversible. Tables with column mapping enabled can only be read in Delta Lake 1.2 and above. Column mapping requires the following Delta protocols: * Reader version 2 or above. * Writer version 5 or above. For a Delta table with the required protocol versions, you can enable column mapping by setting `delta.columnMapping.mode` to `name`. You can use the following command to upgrade the table version and enable column mapping: * SQL ```sql ALTER TABLE SET TBLPROPERTIES ( 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5', 'delta.columnMapping.mode' = 'name' ) ``` Note You cannot turn off column mapping after you enable it. If you try to set `'delta.columnMapping.mode' = 'none'`, you’ll get an error. ## Rename a column [Section titled “Rename a column”](#rename-a-column) When column mapping is enabled for a Delta table, you can rename a column: * SQL ```sql ALTER TABLE RENAME COLUMN old_col_name TO new_col_name ``` For more examples, see [Rename columns](/delta-batch/#rename-columns). ## Drop columns [Section titled “Drop columns”](#drop-columns) When column mapping is enabled for a Delta table, you can drop one or more columns: * SQL ```sql ALTER TABLE table_name DROP COLUMN col_name; ALTER TABLE table_name DROP COLUMNS (col_name_1, col_name_2, ...); ``` For more details, see [Drop columns](/delta-batch/#drop-columns). ## Supported characters in column names [Section titled “Supported characters in column names”](#supported-characters-in-column-names) When column mapping is enabled for a Delta table, you can include spaces as well as any of these characters in the table’s column names: `,;{}()\n\t=`. ## Known limitations [Section titled “Known limitations”](#known-limitations) * Enabling column mapping on tables might break downstream operations that rely on Delta change data feed. See [Change data feed limitations for tables with column mapping enabled](/delta-change-data-feed/#change-data-feed-limitations-for-tables-with-column-mapping-enabled). * In Delta Lake 2.1 and below, [Spark Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) reads are explicitly blocked on a column mapping enabled table. * In Delta Lake 2.2 and above, [Spark Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) reads are explicitly blocked on a column mapping enabled table that underwent column renaming or column dropping. * In Delta Lake 3.0 and above, [Spark Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) reads require schema tracking to be enabled on a column mapping enabled table that underwent column renaming or column dropping. See [Tracking non-additive schema changes](/delta-streaming/#schema-tracking) * The Delta table protocol specifies two modes of column mapping, by `name` and by `id`. Delta Lake 2.1 and below do not support `id` mode. ```plaintext ``` # Constraints > Learn how Delta tables apply constraints. Delta tables support standard SQL constraint management clauses that ensure that the quality and integrity of data added to a table is automatically verified. When a constraint is violated, Delta Lake throws an `InvariantViolationException` to signal that the new data can’t be added. Important Adding a constraint automatically upgrades the table writer protocol version. See [How does Delta Lake manage feature compatibility?](/versioning/) to understand table protocol versioning and what it means to upgrade the protocol version. Two types of constraints are supported: * `NOT NULL`: indicates that values in specific columns cannot be null. * `CHECK`: indicates that a specified Boolean expression must be true for each input row. ### `NOT NULL` constraint [Section titled “NOT NULL constraint”](#not-null-constraint) You specify `NOT NULL` constraints in the schema when you create a table and drop `NOT NULL` constraints using the `ALTER TABLE CHANGE COLUMN` command. * SQL ```sql CREATE TABLE default.people10m ( id INT NOT NULL, firstName STRING, middleName STRING NOT NULL, lastName STRING, gender STRING, birthDate TIMESTAMP, ssn STRING, salary INT ) USING DELTA; ALTER TABLE default.people10m CHANGE COLUMN middleName DROP NOT NULL; ``` If you specify a `NOT NULL` constraint on a column nested within a struct, the parent struct is also constrained to not be null. However, columns nested within array or map types do not accept `NOT NULL` constraints. ### `CHECK` constraint [Section titled “CHECK constraint”](#check-constraint) You manage `CHECK` constraints using the `ALTER TABLE ADD CONSTRAINT` and `ALTER TABLE DROP CONSTRAINT` commands. `ALTER TABLE ADD CONSTRAINT` verifies that all existing rows satisfy the constraint before adding it to the table. * SQL ```sql CREATE TABLE default.people10m ( id INT, firstName STRING, middleName STRING, lastName STRING, gender STRING, birthDate TIMESTAMP, ssn STRING, salary INT ) USING DELTA; ALTER TABLE default.people10m ADD CONSTRAINT dateWithinRange CHECK (birthDate > '1900-01-01'); ALTER TABLE default.people10m DROP CONSTRAINT dateWithinRange; ``` `CHECK` constraints are table properties in the output of the `DESCRIBE DETAIL` and `SHOW TBLPROPERTIES` commands. * SQL ```sql ALTER TABLE default.people10m ADD CONSTRAINT validIds CHECK (id > 1 and id < 99999999); DESCRIBE DETAIL default.people10m; SHOW TBLPROPERTIES default.people10m; ``` # Delta default column values > Learn about default column values in Delta. Note This feature is available in Delta Lake 3.1.0 and above and is enabled using the `allowColumnDefaults` writer [table feature](/versioning/#what-are-table-features). Delta enables the specification of [default expressions](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#default-columns) for columns in Delta tables. When users write to these tables without explicitly providing values for certain columns, or when they explicitly use the DEFAULT SQL keyword for a column, Delta automatically generates default values for those columns. This information is stored in the [StructField](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#struct-field) corresponding to the column of interest. ## How to enable Delta Lake default column values [Section titled “How to enable Delta Lake default column values”](#how-to-enable-delta-lake-default-column-values) Important Enabling default column values for a Delta table will upgrade its [protocol version](/versioning/#what-is-a-protocol-version) to support [table features](/versioning/#what-are-table-features). This protocol upgrade is irreversible. Tables with default column values enabled can only be written to in Delta Lake 3.1 and above. You can enable default column values for a table by setting `delta.feature.allowColumnDefaults` to `enabled`: * SQL ```sql ALTER TABLE SET TBLPROPERTIES ( 'delta.feature.allowColumnDefaults' = 'enabled' ) ``` ## How to use default columns in SQL commands [Section titled “How to use default columns in SQL commands”](#how-to-use-default-columns-in-sql-commands) * For SQL commands that perform table writes, such as `INSERT`, `UPDATE`, and `MERGE` commands, the `DEFAULT` keyword resolves to the most recently assigned default value for the corresponding column (or NULL if no default value exists). For instance, the following SQL command will use the default value for the second column in the table: `INSERT INTO t VALUES (16, DEFAULT);` * It is also possible for INSERT commands to specify lists of fewer columns than the target table, in which case the engine will assign default values for the remaining columns (or NULL for any columns where no defaults yet exist). Important The metadata discussed here apply solely to write operations, not read operations. * The `ALTER TABLE ... ADD COLUMN` command that introduces a new column to an existing table may not specify a default value for the new column. For instance, the following SQL command is not supported in Delta Lake: `ALTER TABLE t ADD COLUMN c INT DEFAULT 16;` * It is permissible, however, to assign or update default values for columns that were created in previous commands. For example, the following SQL command is valid: `ALTER TABLE t ALTER COLUMN c SET DEFAULT 16;` # What are deletion vectors? > Learn about deletion vectors in Delta Lake. Note This feature is available in Delta Lake 2.3.0 and above. This feature is in experimental support mode. Deletion vectors are a storage optimization feature that can be enabled on Delta Lake tables. By default, when a single row in a data file is deleted, the entire Parquet file containing the record must be rewritten. With deletion vectors enabled for the table, some Delta operations use deletion vectors to mark existing rows as removed without rewriting the Parquet file. Subsequent reads on the table resolve current table state by applying the deletions noted by deletion vectors to the most recent table version. Support for deletion vectors was incrementally added with each Delta Lake version. The table below depicts the supported operations for each Delta Lake version. | Operation | First available Delta Lake version | Enabled by default since Delta Lake version | | --------- | ---------------------------------- | ------------------------------------------- | | `SCAN` | 2.3.0 | 2.3.0 | | `DELETE` | 2.4.0 | 2.4.0 | | `UPDATE` | 3.0.0 | 3.1.0 | | `MERGE` | 3.1.0 | 3.1.0 | ## Enable deletion vectors [Section titled “Enable deletion vectors”](#enable-deletion-vectors) You enable support for deletion vectors on a Delta Lake table by setting a Delta Lake table property: * SQL ```sql ALTER TABLE SET TBLPROPERTIES('delta.enableDeletionVectors' = true); ``` Warning When you enable deletion vectors, the table protocol version is upgraded. After upgrading, the table will not be readable by Delta Lake clients that do not support deletion vectors. See [How does Delta Lake manage feature compatibility?](/versioning/). In Delta Lake 3.0 and above, you can drop the deletion vectors table feature to enable compatibility with other Delta clients. See [Drop Delta table features](/delta-drop-feature/). ## Apply changes to Parquet data files [Section titled “Apply changes to Parquet data files”](#apply-changes-to-parquet-data-files) Deletion vectors indicate changes to rows as soft-deletes that logically modify existing Parquet data files in the Delta Lake tables. These changes are applied physically when data files are rewritten, as triggered by one of the following events: * A DML command with deletion vectors disabled (by a command flag or a table property) is run on the table. * An `OPTIMIZE` command is run on the table. * `REORG TABLE ... APPLY (PURGE)` is run against the table. `UPDATE`, `MERGE`, and `OPTIMIZE` do not have strict guarantees for resolving changes recorded in deletion vectors, and some changes recorded in deletion vectors might not be applied if target data files contain no updated records, or would not otherwise be candidates for file compaction. `REORG TABLE ... APPLY (PURGE)` rewrites all data files containing records with modifications recorded using deletion vectors. See [Apply changes with REORG TABLE](#apply-changes-with-reorg-table) Note Modified data might still exist in the old files. You can run `VACUUM` to physically delete the old files. `REORG TABLE ... APPLY (PURGE)` creates a new version of the table at the time it completes, which is the timestamp you must consider for the retention threshold for your `VACUUM` operation to fully remove deleted files. ### Apply changes with REORG TABLE [Section titled “Apply changes with REORG TABLE”](#apply-changes-with-reorg-table) Reorganize a Delta Lake table by rewriting files to purge soft-deleted data, such as rows marked as deleted by deletion vectors with `REORG TABLE`: * SQL ```sql REORG TABLE events APPLY (PURGE); -- If you have a large amount of data and only want to purge a subset of it, you can specify an optional partition predicate using `WHERE`: REORG TABLE events WHERE date >= '2022-01-01' APPLY (PURGE); REORG TABLE events WHERE date >= current_timestamp() - INTERVAL '1' DAY APPLY (PURGE); ``` Note * `REORG TABLE` only rewrites files that contain soft-deleted data. - When resulting files of the purge are small, `REORG TABLE` will coalesce them into larger ones. See [OPTIMIZE](/optimizations-oss/) for more info. - `REORG TABLE` is *idempotent*, meaning that if it is run twice on the same dataset, the second run has no effect. - After running `REORG TABLE`, the soft-deleted data may still exist in the old files. You can run [VACUUM](/delta-utility/#vacuum) to physically delete the old files. # Drop Delta table features > Learn how to drop table features in Delta Lake to downgrade reader and writer protocol requirements and resolve compatibility issues. This article describes how to drop Delta Lake table features and downgrade protocol versions. Note This feature is available in Delta Lake 4.0.0 and above. A legacy implementation of the feature is available since Delta Lake 3.0.0. Not all Delta table features can be dropped. See [What Delta table features can be dropped?](#what-delta-table-features-can-be-dropped) You should only use this functionality to support compatibility with earlier Delta Lake versions, Delta Sharing, or other Delta Lake reader or writer clients. ## How can I drop a Delta table feature? [Section titled “How can I drop a Delta table feature?”](#how-can-i-drop-a-delta-table-feature) To remove a Delta table feature, you run an `ALTER TABLE DROP FEATURE ` command. ## What Delta table features can be dropped? [Section titled “What Delta table features can be dropped?”](#what-delta-table-features-can-be-dropped) You can drop the following Delta table features: * `deletionVectors`. See [What are deletion vectors?](/delta-deletion-vectors/). Drop support for deletion vectors is available in Delta Lake 4.0.0 and above. * `typeWidening-preview`. See [Delta type widening](/delta-type-widening/). Type widening is available in preview in Delta Lake 3.2.0 and above. * `typeWidening`. See [Delta type widening](/delta-type-widening/). Type widening is available in preview in Delta Lake 4.0.0 and above. * `v2Checkpoint`. See [V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec). Drop support for V2 Checkpoints is available in Delta Lake 3.1.0 and above. * `columnMapping`. See [Delta column mapping](/delta-column-mapping/). Drop support for column mapping is available in Delta Lake 3.3.0 and above. * `vacuumProtocolCheck`. See [Vacuum Protocol Check Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#vacuum-protocol-check). Drop support for vacuum protocol check is available in Delta Lake 3.3.0 and above. * `checkConstraints`. See [Constraints](/delta-constraints/). Drop support for check constraints is available in Delta Lake 3.3.0 and above. * `inCommitTimestamp`. See [In-Commit Timestamps](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#in-commit-timestamps). Drop support for In-Commit Timestamp is available in Delta Lake 3.3.0 and above. * `checkpointProtection`. See [Checkpoint Protection Spec](https://github.com/delta-io/delta/blob/master/protocol_rfcs/checkpoint-protection.md). Drop support for checkpoint protection is available in Delta Lake 4.0.0 and above. You cannot drop other [Delta table features](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#valid-feature-names-in-table-features). ## What happens when a table feature is dropped? [Section titled “What happens when a table feature is dropped?”](#what-happens-when-a-table-feature-is-dropped) When you drop a table feature, Delta Lake performs a series of atomic operations: * Disable table properties that use the table feature. * Rewrite data files as necessary to remove all traces of the table feature from the data files backing the table in the current version. * Create a set of protected checkpoints that allow reader clients to interpret table history correctly. * Add the writer table feature checkpointProtection to the table protocol. * Downgrade the table protocol to the lowest reader and writer versions that support all remaining table features. ## What is the checkpointProtection table feature? [Section titled “What is the checkpointProtection table feature?”](#what-is-the-checkpointprotection-table-feature) When you drop a feature, Delta Lake rewrites data and metadata in the table’s history as protected checkpoints to respect the protocol downgrade. After the downgrade, the table should always be readable by more clients. This is because the protocol for the table now reflects that support for the dropped feature is no longer required to read the table. The protected checkpoints and the checkpointProtection feature accomplish the following: * Reader clients that understand the dropped table feature can access all available table history. * Reader clients that do not support the dropped table feature only need to read the table history starting from the protocol downgrade version. * Writer clients do not rewrite checkpoints prior to the protocol downgrade. * Table maintenance operations respect requirements set by `checkpointProtection`, which mark protocol downgrade checkpoints as protected. * While you can only drop one table feature with each DROP FEATURE command, a table can have multiple protected checkpoints and dropped features in its table history. The table feature `checkpointProtection` should not block read-only access from Delta Lake clients. To fully downgrade the table and remove the `checkpointProtection` table feature, you must use TRUNCATE HISTORY. The recommendation is to only use this pattern if you need to write to tables with external Delta clients that do not support checkpointProtection. ## Fully downgrade table protocols for legacy clients [Section titled “Fully downgrade table protocols for legacy clients”](#fully-downgrade-table-protocols-for-legacy-clients) If integrations with external Delta Lake clients require writes that don’t support the checkpointProtection table feature, you must use TRUNCATE HISTORY to fully remove all traces of the disabled table features and fully downgrade the table protocol. It is recommended to always test the default behavior for DROP FEATURE before proceeding with TRUNCATE HISTORY. Running TRUNCATE HISTORY removes all table history greater than 24 hours. Full table downgrade occurs in two steps that must occur at least 24 hours apart. ### Step 1: Prepare to drop a table feature [Section titled “Step 1: Prepare to drop a table feature”](#step-1-prepare-to-drop-a-table-feature) During the first stage, the user prepares to drop the table feature. The following describes what happens during this stage: 1. You run the `ALTER TABLE DROP FEATURE TRUNCATE HISTORY` command. 2. Table properties that specifically enable a table feature have values set to disable the feature. 3. Table properties that control behaviors associated with the dropped feature have options set to default values before the feature was introduced. 4. As necessary, data and metadata files are rewritten respecting the updated table properties. 5. The command finishes running and returns an error message informing the user they must wait 24 hours to proceed with feature removal. After first disabling a feature, you can continue writing to the target table before completing the protocol downgrade, but you cannot use the table feature you are removing. Note If you leave the table in this state, operations against the table do not use the table feature, but the protocol still supports the table feature. Until you complete the final downgrade step, the table is not readable by Delta clients that do not understand the table feature. ### Step 2: Downgrade the protocol and drop a table feature [Section titled “Step 2: Downgrade the protocol and drop a table feature”](#step-2-downgrade-the-protocol-and-drop-a-table-feature) To fully remove all transaction history associated with the feature and downgrade the protocol: 1. After at least 24 hours have passed, you run the `ALTER TABLE DROP FEATURE TRUNCATE HISTORY` command. 2. The client confirms that no transactions in the specified retention threshold use the table feature, then truncates the table history to that threshold. 3. The protocol is downgraded, dropping the table feature. 4. If the table features that are present in the table can be represented by a legacy protocol version, the `minReaderVersion` and `minWriterVersion` for the table are downgraded to the lowest version that supports exactly all remaining features in use by the Delta table. Important Running `ALTER TABLE DROP FEATURE TRUNCATE HISTORY` removes all transaction log data older than 24 hours. After dropping a Delta table feature, you do not have access to table history or time travel. See [How does Delta Lake manage feature compatibility?](/versioning/). # Frequently asked questions (FAQ) > Find answers to commonly asked questions about Delta Lake. ## What is Delta Lake? [Section titled “What is Delta Lake?”](#what-is-delta-lake) [Delta Lake](https://delta.io/) is an [open source storage layer](https://github.com/delta-io/delta) that brings reliability to [data lakes](https://databricks.com/discover/data-lakes/introduction). Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs. ## How is Delta Lake related to Apache Spark? [Section titled “How is Delta Lake related to Apache Spark?”](#how-is-delta-lake-related-to-apache-spark) Delta Lake sits on top of Apache Spark. The format and the compute layer helps to simplify building big data pipelines and increase the overall efficiency of your pipelines. ## What format does Delta Lake use to store data? [Section titled “What format does Delta Lake use to store data?”](#what-format-does-delta-lake-use-to-store-data) Delta Lake uses versioned Parquet files to store your data in your cloud storage. Apart from the versions, Delta Lake also stores a transaction log to keep track of all the commits made to the table or blob store directory to provide ACID transactions. ## How can I read and write data with Delta Lake? [Section titled “How can I read and write data with Delta Lake?”](#how-can-i-read-and-write-data-with-delta-lake) You can use your favorite Apache Spark APIs to read and write data with Delta Lake. See [Read a table](/delta-batch/#read-a-table) and [Write to a table](/delta-batch/#write-to-table). ## Where does Delta Lake store the data? [Section titled “Where does Delta Lake store the data?”](#where-does-delta-lake-store-the-data) When writing data, you can specify the location in your cloud storage. Delta Lake stores the data in that location in Parquet format. ## Can I copy my Delta Lake table to another location? [Section titled “Can I copy my Delta Lake table to another location?”](#can-i-copy-my-delta-lake-table-to-another-location) Yes you can copy your Delta Lake table to another location. Remember to copy files without changing the timestamps to ensure that the time travel with timestamps will be consistent. ## Can I stream data directly into and from Delta tables? [Section titled “Can I stream data directly into and from Delta tables?”](#can-i-stream-data-directly-into-and-from-delta-tables) Yes, you can use Structured Streaming to directly write data into Delta tables and read from Delta tables. See [Stream data into Delta tables](/delta-streaming/#delta-table-as-a-sink) and [Stream data from Delta tables](/delta-streaming/#delta-table-as-a-source). ## Does Delta Lake support writes or reads using the Spark Streaming DStream API? [Section titled “Does Delta Lake support writes or reads using the Spark Streaming DStream API?”](#does-delta-lake-support-writes-or-reads-using-the-spark-streaming-dstream-api) Delta does not support the DStream API. We recommend [Table streaming reads and writes](/delta-streaming/). ## When I use Delta Lake, will I be able to port my code to other Spark platforms easily? [Section titled “When I use Delta Lake, will I be able to port my code to other Spark platforms easily?”](#when-i-use-delta-lake-will-i-be-able-to-port-my-code-to-other-spark-platforms-easily) Yes. When you use Delta Lake, you are using open Apache Spark APIs so you can easily port your code to other Spark platforms. To port your code, replace `delta` format with `parquet` format. ## Does Delta Lake support multi-table transactions? [Section titled “Does Delta Lake support multi-table transactions?”](#does-delta-lake-support-multi-table-transactions) Delta Lake does not support multi-table transactions and foreign keys. Delta Lake supports transactions at the *table* level. ## How can I change the type of a column? [Section titled “How can I change the type of a column?”](#how-can-i-change-the-type-of-a-column) Changing a column’s type or dropping a column requires rewriting the table. For an example, see [Change column type](/delta-batch/#change-column-type-or-name). # Delta Kernel > Learn how to build connectors to read and write Delta tables. The Delta Kernel project is a set of libraries ([Java](#kernel-java) and [Rust](#kernel-rust)) for building Delta connectors that can read from and write into Delta tables without the need to understand the [Delta protocol details](https://github.com/delta-io/delta/blob/master/PROTOCOL.md). You can use this library to do the following: * Read data from small Delta tables in a single thread in a single process. * Read data from large Delta tables using multiple threads in a single process. * Build a complex connector for a distributed processing engine and read very large Delta tables. * Insert data into a Delta table either from a single process or a complex distributed engine. Here is an example of a simple table scan with a filter: * Java ```java Engine myEngine = DefaultEngine.create() ; // define a engine (more details below) Table myTable = Table.forPath("/delta/table/path"); // define what table to scan Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); // define which version of table to scan Scan myScan = mySnapshot.getScanBuilder(myEngine) // specify the scan details .withFilters(myEngine, scanFilter) .build(); CloseableIterator physicalData = // read the Parquet data files .. read from Parquet data files ... Scan.transformPhysicalData(...) // returns the table data ``` A complete version of the above example program and more examples of reading from and writing into a Delta table are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). Notice that there are two sets of public APIs to build connectors. * **Table APIs** - Interfaces like [`Table`](https://delta-io.github.io/delta/snapshot/kernel-api/java/index.html?io/delta/kernel/Table.html) and [`Snapshot`](https://delta-io.github.io/delta/snapshot/kernel-api/java/index.html?io/delta/kernel/Snapshot.html) that allow you to read (and soon write to) Delta tables * **Engine APIs** - The [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java//index.html?io/delta/kernel/engine/Engine.html) interface allows you to plug in connector-specific optimizations to compute-intensive components in the Kernel. For example, Delta Kernel provides a *default* Parquet file reader via the `DefaultEngine`, but you may choose to replace that default with a custom `Engine` implementation that has a faster Parquet reader for your connector/processing engine. ## Kernel Java [Section titled “Kernel Java”](#kernel-java) ## What is Delta Kernel? [Section titled “What is Delta Kernel?”](#what-is-delta-kernel) Delta Kernel is a library for operating on Delta tables. Specifically, it provides simple and narrow APIs for reading and writing to Delta tables without the need to understand the [Delta protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) details. You can use this library to do the following: * Read and write Delta tables from your applications. * Build a connector for a distributed engine like [Apache Spark™](https://github.com/apache/spark), [Apache Flink](https://github.com/apache/flink), or [Trino](https://github.com/trinodb/trino) for reading or writing massive Delta tables. ## Set up Delta Kernel for your project [Section titled “Set up Delta Kernel for your project”](#set-up-delta-kernel-for-your-project) You need to `io.delta:delta-kernel-api` and `io.delta:delta-kernel-defaults` dependencies. Following is an example Maven `pom` file dependency list. The `delta-kernel-api` module contains the core of the Kernel that abstracts out the Delta protocol to enable reading and writing into Delta tables. It makes use of the `Engine` interface that is being passed to the Kernel API by the connector for heavy-lift operations such as reading/writing Parquet or JSON files, evaluating expressions or file system operations such as listing contents of the Delta Log directory, etc. Kernel supplies a default implementation of `Engine` in module `delta-kernel-defaults`. The connectors can implement their own version of `Engine` to make use of their native implementation of functionalities the `Engine` provides. For example: the connector can make use of their Parquet reader instead of using the reader from the `DefaultEngine`. More details on this [later](#step-2-build-your-own-engine). ```xml io.delta delta-kernel-api ${delta-kernel.version} io.delta delta-kernel-defaults ${delta-kernel.version} ``` If your connector is not using the [`DefaultEngine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) provided by the Kernel, the dependency `delta-kernel-defaults` from the above list can be skipped. ## Read a Delta table in a single process [Section titled “Read a Delta table in a single process”](#read-a-delta-table-in-a-single-process) In this section, we will walk through how to build a very simple single-process Delta connector that can read a Delta table using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. ### Step 1: Full scan on a Delta table [Section titled “Step 1: Full scan on a Delta table”](#step-1-full-scan-on-a-delta-table) The main entry point is [`io.delta.kernel.Table`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html) which is a programmatic representation of a Delta table. Say you have a Delta table at the directory `myTablePath`. You can create a `Table` object as follows: ```java import io.delta.kernel.*; import io.delta.kernel.defaults.*; import org.apache.hadoop.conf.Configuration; String myTablePath = ; // fully qualified table path. Ex: file:/user/tables/myTable Configuration hadoopConf = new Configuration(); Engine myEngine = DefaultEngine.create(hadoopConf); Table myTable = Table.forPath(myEngine, myTablePath); ``` Note the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) we are creating to bootstrap the `myTable` object. This object allows you to plug in your own libraries for computationally intensive operations like Parquet file reading, JSON parsing, etc. You can ignore it for now. We will discuss more about this later when we discuss how to build more complex connectors for distributed processing engines. From this `myTable` object you can create a [`Snapshot`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Snapshot.html) object which represents the consistent state (a.k.a. a snapshot consistency) in a specific version of the table. ```java Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); ``` Now that we have a consistent snapshot view of the table, we can query more details about the table. For example, you can get the version and schema of this snapshot. ```java long version = mySnapshot.getVersion(myEngine); StructType tableSchema = mySnapshot.getSchema(myEngine); ``` Next, to read the table data, we have to *build* a [`Scan`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html) object. In order to build a `Scan` object, create a [`ScanBuilder`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/ScanBuilder.html) object which optionally allows selecting a subset of columns to read or setting a query filter. For now, ignore these optional settings. ```java Scan myScan = mySnapshot.getScanBuilder(myEngine).build() // Common information about scanning for all data files to read. Row scanState = myScan.getScanState(myEngine) // Information about the list of scan files to read CloseableIterator scanFiles = myScan.getScanFiles(myEngine) ``` This [`Scan`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html) object has all the necessary metadata to start reading the table. There are two crucial pieces of information needed for reading data from a file in the table. * `myScan.getScanFiles(Engine)`: Returns scan files as columnar batches (represented as an iterator of `FilteredColumnarBatch`es, more on that later) where each selected row in the batch has information about a single file containing the table data. * `myScan.getScanState(Engine)`: Returns the snapshot-level information needed for reading any file. Note that this is a single row and common to all scan files. For each scan file the physical data must be read from the file. The columns to read are specified in the scan file state. Once the physical data is read, you have to call [`ScanFile.transformPhysicalData(…)`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html#transformPhysicalData-io.delta.kernel.engine.Engine-io.delta.kernel.data.Row-io.delta.kernel.data.Row-io.delta.kernel.utils.CloseableIterator-) with the scan state and the physical data read from scan file. This API takes care of transforming (e.g. adding partition columns) the physical data into logical data of the table. Here is an example of reading all the table data in a single thread. ```java CloserableIterator fileIter = scanObject.getScanFiles(myEngine); Row scanStateRow = scanObject.getScanState(myEngine); while(fileIter.hasNext()) { FilteredColumnarBatch scanFileColumnarBatch = fileIter.next(); // Get the physical read schema of columns to read from the Parquet data files StructType physicalReadSchema = ScanStateRow.getPhysicalDataReadSchema(engine, scanStateRow); try (CloseableIterator scanFileRows = scanFileColumnarBatch.getRows()) { while (scanFileRows.hasNext()) { Row scanFileRow = scanFileRows.next(); // From the scan file row, extract the file path, size and modification time metadata // needed to read the file. FileStatus fileStatus = InternalScanFileUtils.getAddFileStatus(scanFileRow); // Open the scan file which is a Parquet file using connector's own // Parquet reader or default Parquet reader provided by the Kernel (which // is used in this example). CloseableIterator physicalDataIter = engine.getParquetHandler().readParquetFiles( singletonCloseableIterator(fileStatus), physicalReadSchema, Optional.empty() /* optional predicate the connector can apply to filter data from the reader */ ); // Now the physical data read from the Parquet data file is converted to a table // logical data. Logical data may include the addition of partition columns and/or // subset of rows deleted try ( CloseableIterator transformedData = Scan.transformPhysicalData( engine, scanStateRow, scanFileRow, physicalDataIter)) { while (transformedData.hasNext()) { FilteredColumnarBatch logicalData = transformedData.next(); ColumnarBatch dataBatch = logicalData.getData(); // Not all rows in `dataBatch` are in the selected output. // An optional selection vector determines whether a row with a // specific row index is in the final output or not. Optional selectionVector = dataReadResult.getSelectionVector(); // access the data for the column at ordinal 0 ColumnVector column0 = dataBatch.getColumnVector(0); for (int rowIndex = 0; rowIndex < column0.getSize(); rowIndex++) { // check if the row is selected or not if (!selectionVector.isPresent() || // there is no selection vector, all records are selected (!selectionVector.get().isNullAt(rowId) && selectionVector.get().getBoolean(rowId))) { // Assuming the column type is String. // If it is a different type, call the relevant function on the `ColumnVector` System.out.println(column0.getString(rowIndex)); } } // access the data for column at ordinal 1 ColumnVector column1 = dataBatch.getColumnVector(1); for (int rowIndex = 0; rowIndex < column1.getSize(); rowIndex++) { // check if the row is selected or not if (!selectionVector.isPresent() || // there is no selection vector, all records are selected (!selectionVector.get().isNullAt(rowId) && selectionVector.get().getBoolean(rowId))) { // Assuming the column type is Long. // If it is a different type, call the relevant function on the `ColumnVector` System.out.println(column1.getLong(rowIndex)); } } // .. more .. } } } } } ``` A few working examples to read Delta tables within a single process are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). Important All the Delta protocol-level details are encoded in the rows returned by `Scan.getScanFiles` API, but you do not have to understand them in order to read the table data correctly. All you need is to get the Parquet file status from each scan file row and read the data from the Parquet file into the `ColumnarBatch` format. The physical data is converted into the logical data of the table using [`Scan.transformPhysicalData`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html#transformPhysicalData-io.delta.kernel.engine.Engine-io.delta.kernel.data.Row-io.delta.kernel.data.Row-io.delta.kernel.utils.CloseableIterator-). Transformation to logical data is dictated by the protocol and the metadata of the table and the scan file. As the Delta protocol evolves this transformation step will evolve with it and your code will not have to change to accommodate protocol changes. This is the major advantage of the abstractions provided by Delta Kernel. Note Observe that the same `Engine` instance `myEngine` is passed multiple times whenever a call to Delta Kernel API is made. The reason for passing this instance for every call is because it is the connector context, it should maintained outside of the Delta Kernel APIs to give the connector control over the `Engine`. ### Step 2: Improve scan performance with file skipping [Section titled “Step 2: Improve scan performance with file skipping”](#step-2-improve-scan-performance-with-file-skipping) We have explored how to do a full table scan. However, the real advantage of using the Delta format is that you can skip files using your query filters. To make this possible, Delta Kernel provides an [expression framework](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/expressions/package-summary.html) to encode your filters and provide them to Delta Kernel to skip files during the scan file generation. For example, say your table is partitioned by `columnX`, you want to query only the partition `columnX=1`. You can generate the expression and use it to build the scan as follows: * Java ```java import io.delta.kernel.expressions.*; import io.delta.kernel.defaults.engine.*; Engine myEngine = DefaultEngine.create(new Configuration()); Predicate filter = new Predicate( "=", Arrays.asList(new Column("columnX"), Literal.ofInt(1))); Scan myFilteredScan = mySnapshot.buildScan(engine) .withFilter(myEngine, filter) .build() // Subset of the given filter that is not guaranteed to be satisfied by // Delta Kernel when it returns data. This filter is used by Delta Kernel // to do data skipping as much as possible. The connector should use this filter // on top of the data returned by Delta Kernel in order for further filtering. Optional remainingFilter = myFilteredScan.getRemainingFilter(); ``` The scan files returned by `myFilteredScan.getScanFiles(myEngine)` will have rows representing files only of the required partition. Similarly, you can provide filters for non-partition columns, and if the data in the table is well clustered by those columns, then Delta Kernel will be able to skip files as much as possible. ## Create a Delta table [Section titled “Create a Delta table”](#create-a-delta-table) In this section, we will walk through how to build a Delta connector that can create a Delta table using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. The main entry point is [`io.delta.kernel.Table`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html) which is a programmatic representation of a Delta table. Say you want to create Delta table at the directory `myTablePath`. You can create a `Table` object as follows: * Java ```java package io.delta.kernel.examples; import io.delta.kernel.*; import io.delta.kernel.types.*; import io.delta.kernel.utils.CloseableIterable; String myTablePath = ; Configuration hadoopConf = new Configuration(); Engine myEngine = DefaultEngine.create(hadoopConf); Table myTable = Table.forPath(myEngine, myTablePath); ``` Note the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) we are creating to bootstrap the `myTable` object. This object allows you to plug in your own libraries for computationally intensive operations like Parquet file reading, JSON parsing, etc. You can ignore it for now. We will discuss more about this later when we discuss how to build more complex connectors for distributed processing engines. From this `myTable` object you can create a [`TransactionBuilder`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionBuilder.html) object which allows you to construct a [`Transaction`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Transaction.html) object ```java TransactionBuilder txnBuilder = myTable.createTransactionBuilder( myEngine, "Examples", /* engineInfo - connector can add its own identifier which is noted in the Delta Log */ Operation.CREATE_TABLE /* What is the operation we are trying to perform. This is noted in the Delta Log */ ); ``` Now that you have the `TransactionBuilder` object, you can set the table schema and partition columns of the table. ```java StructType mySchema = new StructType() .add("id", IntegerType.INTEGER) .add("name", StringType.STRING) .add("city", StringType.STRING) .add("salary", DoubleType.DOUBLE); // Partition columns are optional. Use it only if you are creating a partitioned table. List myPartitionColumns = Collections.singletonList("city"); // Set the schema of the new table on the transaction builder txnBuilder = txnBuilder .withSchema(engine, mySchema); // Set the partition columns of the new table only if you are creating // a partitioned table; otherwise, this step can be skipped. txnBuilder = txnBuilder .withPartitionColumns(engine, examplePartitionColumns); ``` `TransactionBuilder` allows setting additional properties of the table such as enabling a certain Delta feature or setting identifiers for idempotent writes. We will be visiting these in the next sections. The next step is to build `Transaction` out of the `TransactionBuilder` object. ```java // Build the transaction Transaction txn = txnBuilder.build(engine); ``` `Transaction` object allows the connector to optionally add any data and finally commit the transaction. A successful commit ensures that the table is created with the given schema. In this example, we are just creating a table and not adding any data as part of the table. ```java // Commit the transaction. // As we are just creating the table and not adding any data, the `dataActions` is empty. TransactionCommitResult commitResult = txn.commit( engine, CloseableIterable.emptyIterable() /* dataActions */ ); ``` The [`TransactionCommitResult`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionCommitResult.html) contains the what version the transaction is committed as and whether the table is ready for a checkpoint. As we are creating a table the version will be `0`. We will be discussing later on what a checkpoint is and what it means for the table to be ready for the checkpoint. A few working examples to create partitioned and un-partitioned Delta tables are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). ## Create a table and insert data into it [Section titled “Create a table and insert data into it”](#create-a-table-and-insert-data-into-it) In this section, we will walk through how to build a Delta connector that can create a Delta table and insert data into the table (similar to `CREATE TABLE AS ` construct in SQL) using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. The first step is to construct a `Transaction`. Below is the code for that. For more details on what each step of the code means, please read the [create table](#create-a-delta-table) section. ```plaintext package io.delta.kernel.examples; import io.delta.kernel.*; import io.delta.kernel.types.*; import io.delta.kernel.utils.CloseableIterable; String myTablePath = ; Configuration hadoopConf = new Configuration(); Engine myEngine = DefaultEngine.create(hadoopConf); Table myTable = Table.forPath(myEngine, myTablePath); StructType mySchema = new StructType() .add("id", IntegerType.INTEGER) .add("name", StringType.STRING) .add("city", StringType.STRING) .add("salary", DoubleType.DOUBLE); // Partition columns are optional. Use it only if you are creating a partitioned table. List myPartitionColumns = Collections.singletonList("city"); TransactionBuilder txnBuilder = myTable.createTransactionBuilder( myEngine, "Examples", /* engineInfo - connector can add its own identifier which is noted in the Delta Log */ Operation.WRITE /* What is the operation we are trying to perform? This is noted in the Delta Log */ ); // Set the schema of the new table on the transaction builder txnBuilder = txnBuilder .withSchema(engine, mySchema); // Set the partition columns of the new table only if you are creating // a partitioned table; otherwise, this step can be skipped. txnBuilder = txnBuilder .withPartitionColumns(engine, examplePartitionColumns); // Build the transaction Transaction txn = txnBuilder.build(engine); ``` Now that we have the [`Transaction`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Transaction.html) object, the next step is generating the data that confirms the table schema and partitioned according to the table partitions. ```java StructType dataSchema = txn.getSchema(engine) // Optional for un-partitioned tables List partitionColumnNames = txn.getPartitionColumns(engine) ``` Using the data schema and partition column names the connector can plan the query and generate data. At tasks that actually have the data to write to the table, the connector can ask the Kernel to transform the data given in the table schema into physical data that can actually be written to the Parquet data files. For partitioned tables, the data needs to be first partitioned by the partition columns, and then the connector should ask the Kernel to transform the data for each partition separately. The partitioning step is needed because any given data file in the Delta table contains data belonging to exactly one partition. Get the state of the transaction. The transaction state contains the information about how to convert the data in the table schema into physical data that needs to be written. The transformations depend on the protocol and features the table has. ```java Row txnState = txn.getTransactionState(engine); ``` Prepare the data. ```java // The data generated by the connector to write into a table CloseableIterator data = ... // Create partition value map Map partitionValues = Collections.singletonMap( "city", // partition column name // partition value. Depending upon the partition column type, the // partition value should be created. In this example, the partition // column is of type StringType, so we are creating a string literal. Literal.ofString(city) ); ``` The connector data is passed as an iterator of [`FilteredColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html). Each of the `FilteredColumnarBatch` contains a [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) which actually contains the data in columnar access format and an optional section vector that allows the connector to specify which rows from the `ColumnarBatch` to write to the table. Partition values are passed as a map of the partition column name to the partition value. For an un-partitioned table, the map should be empty as it has no partition columns. ```plaintext // Transform the logical data to physical data that needs to be written to the Parquet // files CloseableIterator physicalData = Transaction.transformLogicalData(engine, txnState, data, partitionValues); ``` The above code converts the given data for partitions into an iterator of `FilteredColumnarBatch` that needs to be written to the Parquet data files. In order to write the data files, the connector needs to get the [`WriteContext`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/DataWriteContext.html) from Kernel, which tells the connector where to write the data files and what columns to collect statistics from each data file. ```java // Get the write context DataWriteContext writeContext = Transaction.getWriteContext(engine, txnState, partitionValues); ``` Now, the connector has the physical data that needs to be written to Parquet data files, and where those files should be written, it can start writing the data files. ```java CloseableIterator dataFiles = engine.getParquetHandler() .writeParquetFiles( writeContext.getTargetDirectory(), physicalData, writeContext.getStatisticsColumns() ); ``` In the above code, the connector is making use of the `Engine` provided `ParquetHandler` to write the data, but the connector can choose its own Parquet file writer to write the data. Also note that the return of the above call is an iterator of [`DataFileStatus`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/DataFileStatus.html) for each data file written. It basically contains the file path, file metadata, and optional file-level statistics for columns specified by the [`WriteContext.getStatisticsColumns()`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/DataWriteContext.html#getStatisticsColumns--)) Convert each `DataFileStatus` into a Delta log action that can be written to the Delta table log. ```java CloseableIterator dataActions = Transaction.generateAppendActions(engine, txnState, dataFiles, writeContext); ``` The next step is constructing [`CloseableIterable`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/CloseableIterable.html) out of the all the Delta log actions generated above. The reason for constructing an `Iterable` is that the transaction committing involves accessing the list of Delta log actions more than one time (in order to resolve conflicts when there are multiple writes to the table). Kernel provides a [utility method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/CloseableIterable.html#inMemoryIterable-io.delta.kernel.utils.CloseableIterator-) to create an in-memory version of `CloseableIterable`. This interface also gives the connector an option to implement a custom implementation that spills the data actions to disk when the contents are too big to fit in memory. ```java // Create a iterable out of the data actions. If the contents are too big to fit in memory, // the connector may choose to write the data actions to a temporary file and return an // iterator that reads from the file. CloseableIterable dataActionsIterable = CloseableIterable.inMemoryIterable(dataActions); ``` The final step is committing the transaction! ```java TransactionCommitStatus commitStatus = txn.commit(engine, dataActionsIterable) ``` The [`TransactionCommitResult`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionCommitResult.html) contains the what version the transaction is committed as and whether the table is ready for a checkpoint. As we are creating a table the version will be `0`. We will be discussing later on what a checkpoint is and what it means for the table to be ready for the checkpoint. A few working examples to create and insert data into partitioned and un-partitioned Delta tables are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). ## Blind append into an existing Delta table [Section titled “Blind append into an existing Delta table”](#blind-append-into-an-existing-delta-table) In this section, we will walk through how to build a Delta connector that inserts data into an existing Delta table (similar to `INSERT INTO ` construct in SQL) using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. The steps are exactly similar to [Create table and insert data into it](#create-a-table-and-insert-data-into-it) except that we won’t be providing any schema or partition columns when building the `TransactionBuilder` ```java // Create a `Table` object with the given destination table path Table table = Table.forPath(engine, tablePath); // Create a transaction builder to build the transaction TransactionBuilder txnBuilder = table.createTransactionBuilder( engine, "Examples", /* engineInfo */ Operation.WRITE ); / Build the transaction - no need to provide the schema as the table already exists. Transaction txn = txnBuilder.build(engine); // Get the transaction state Row txnState = txn.getTransactionState(engine); List dataActions = new ArrayList<>(); // Generate the sample data for three partitions. Process each partition separately. // This is just an example. In a real-world scenario, the data may come from different // partitions. Connectors already have the capability to partition by partition values // before writing to the table // In the test data `city` is a partition column for (String city : Arrays.asList("San Francisco", "Campbell", "San Jose")) { FilteredColumnarBatch batch1 = generatedPartitionedDataBatch( 5 /* offset */, city /* partition value */); FilteredColumnarBatch batch2 = generatedPartitionedDataBatch( 5 /* offset */, city /* partition value */); FilteredColumnarBatch batch3 = generatedPartitionedDataBatch( 10 /* offset */, city /* partition value */); CloseableIterator data = toCloseableIterator(Arrays.asList(batch1, batch2, batch3).iterator()); // Create partition value map Map partitionValues = Collections.singletonMap( "city", // partition column name // partition value. Depending upon the parition column type, the // partition value should be created. In this example, the partition // column is of type StringType, so we are creating a string literal. Literal.ofString(city)); // First transform the logical data to physical data that needs to be written // to the Parquet // files CloseableIterator physicalData = Transaction.transformLogicalData(engine, txnState, data, partitionValues); // Get the write context DataWriteContext writeContext = Transaction.getWriteContext(engine, txnState, partitionValues); // Now write the physical data to Parquet files CloseableIterator dataFiles = engine.getParquetHandler() .writeParquetFiles( writeContext.getTargetDirectory(), physicalData, writeContext.getStatisticsColumns()); // Now convert the data file status to data actions that needs to be written to the Delta // table log CloseableIterator partitionDataActions = Transaction.generateAppendActions( engine, txnState, dataFiles, writeContext); // Now add all the partition data actions to the main data actions list. In a // distributed query engine, the partition data is written to files at tasks on executor // nodes. The data actions are collected at the driver node and then written to the // Delta table log using the `Transaction.commit` while (partitionDataActions.hasNext()) { dataActions.add(partitionDataActions.next()); } } // Create a iterable out of the data actions. If the contents are too big to fit in memory, // the connector may choose to write the data actions to a temporary file and return an // iterator that reads from the file. CloseableIterable dataActionsIterable = CloseableIterable.inMemoryIterable( toCloseableIterator(dataActions.iterator())); // Commit the transaction. TransactionCommitResult commitResult = txn.commit(engine, dataActionsIterable); ``` ## Idempotent Blind Appends to a Delta Table [Section titled “Idempotent Blind Appends to a Delta Table”](#idempotent-blind-appends-to-a-delta-table) Idempotent writes allow the connector to make sure the data belonging to a particular transaction version and application id is inserted into the table at most once. In incremental processing systems (e.g. streaming systems), track progress using their own application-specific versions need to record what progress has been made in order to avoid duplicating data in the face of failures and retries during writes. By setting the transaction identifier, the Delta table can ensure that the data with the same identifier is not written multiple times. For more information refer to the Delta protocol section [Transaction Identifiers](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#transaction-identifiers) To make the data append idempotent, set the transaction identifier on the [`TransactionBuilder`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionBuilder.html#withTransactionId-io.delta.kernel.engine.Engine-java.lang.String-long-) ```java // Set the transaction identifiers for idempotent writes // Delta/Kernel makes sure that there exists only one transaction in the Delta log // with the given application id and txn version txnBuilder = txnBuilder.withTransactionId( engine, "my app id", /* application id */ 100 /* monotonically increasing txn version with each new data insert */ ); ``` That’s all the connector need to do for idempotent blind appends. ## Checkpointing a Delta table [Section titled “Checkpointing a Delta table”](#checkpointing-a-delta-table) [Checkpoints](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoints) are an optimization in Delta Log in order to construct the state of the Delta table faster. It basically contains the state of the table at the version the checkpoint is created. Delta Kernel allows the connector to optionally make the checkpoints. It is created for every few commits (configurable table property) on the table. The result of `Transaction.commit` returns a `TransactionCommitResult` that contains the version the transaction is committed as and whether the table is [read for checkpoint](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionCommitResult.html#isReadyForCheckpoint--). Creating a checkpoint takes time as it needs to construct the entire state of the table. If the connector doesn’t want to checkpoint by itself but uses other connectors that are faster in creating a checkpoint, it can skip the checkpointing step. If it wants to checkpoint, the `Table` object has an [API](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html#checkpoint-io.delta.kernel.engine.Engine-long-) to checkpoint the table. ```java TransactionCommitResult commitResult = txn.commit(engine, dataActionsIterable); if (commitResult.isReadyForCheckpoint()) { // Checkpoint the table Table.forPath(engine, tablePath).checkpoint(engine, commitResult.getVersion()); } ``` ## Build a Delta connector for a distributed processing engine [Section titled “Build a Delta connector for a distributed processing engine”](#build-a-delta-connector-for-a-distributed-processing-engine) Unlike simple applications that just read the table in a single process, building a connector for complex processing engines like Apache Spark™ and Trino can require quite a bit of additional effort. For example, to build a connector for an SQL engine you have to do the following * Understand the APIs provided by the engine to build connectors and how Delta Kernel can be used to provide the information necessary for the connector + engine to operate on a Delta table. * Decide what libraries to use to do computationally expensive operations like reading Parquet files, parsing JSON, computing expressions, etc. Delta Kernel provides all the extension points to allow you to plug in any library without having to understand all the low-level details of the Delta protocol. * Deal with details specific to distributed engines. For example, * Serialization of Delta table metadata provided by Delta Kernel. * Efficiently transforming data read from Parquet into the engine in-memory processing format. In this section, we are going to outline the steps needed to build a connector. ### Step 0: Validate the prerequisites [Section titled “Step 0: Validate the prerequisites”](#step-0-validate-the-prerequisites) In the previous section showing how to read a simple table, we were briefly introduced to the [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html). This is the main extension point where you can plug in your implementations of computationally-expensive operations like reading Parquet files, parsing JSON, etc. For the simple case, we were using a default implementation of the helper that works in most cases. However, for building a high-performance connector for a complex processing engine, you will very likely need to provide your own implementation using the libraries that work with your engine. So before you start building your connector, it is important to understand these requirements and plan for building your own engine. Here are the libraries/capabilities you need to build a connector that can read the Delta table * Perform file listing and file reads from your storage/file system. * Read Parquet files in columnar data, preferably in an in-memory columnar format. * Parse JSON data * Read JSON files * Evaluate expressions on in-memory columnar batches For each of these capabilities, you can choose to build your own implementation or reuse the default implementation. ### Step 1: Set up Delta Kernel in your connector project [Section titled “Step 1: Set up Delta Kernel in your connector project”](#step-1-set-up-delta-kernel-in-your-connector-project) In the Delta Kernel project, there are multiple dependencies you can choose to depend on. 1. Delta Kernel core APIs - This is a must-have dependency, which contains all the main APIs like Table, Snapshot, and Scan that you will use to access the metadata and data of the Delta table. This has very few dependencies reducing the chance of conflicts with any dependencies in your connector and engine. This also provides the [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) interface which allows you to plug in your implementations of computationally expensive operations, but it does not provide any implementation of this interface. 2. Delta Kernel default- This has a default implementation called [`DefaultEngine`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultEngine.html) and additional dependencies such as `Hadoop`. If you wish to reuse all or parts of this implementation, then you can optionally depend on this. #### Set up Java projects [Section titled “Set up Java projects”](#set-up-java-projects) As discussed above, you can import one or both of the artifacts as follows: ```xml io.delta delta-kernel-api ${delta-kernel.version} io.delta delta-kernel-defaults ${delta-kernel.version} ``` ### Step 2: Build your own Engine [Section titled “Step 2: Build your own Engine”](#step-2-build-your-own-engine) In this section, we are going to explore the [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) interface and walk through how to implement your own implementation so that you can plug in your connector/engine-specific implementations of computationally-intensive operations, threading model, resource management, etc. > \[!IMPORTANT] During the validation process, if you believe that all the dependencies of the default `Engine` implementation can work with your connector and engine, then you can skip this step and jump to Step 3 of implementing your connector using the default engine. If later you have the need to customize the helper for your connector, you can revisit this step. #### Step 2.1: Implement the `Engine` interface [Section titled “Step 2.1: Implement the Engine interface”](#step-21-implement-the-engine-interface) The [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) interface combines a bunch of sub-interfaces each of which is designed for a specific purpose. Here is a brief overview of the subinterfaces. See the API docs (Java) for a more detailed view. ```java interface Engine { /** * Get the connector provided {@link ExpressionHandler}. * @return An implementation of {@link ExpressionHandler}. */ ExpressionHandler getExpressionHandler(); /** * Get the connector provided {@link JsonHandler}. * @return An implementation of {@link JsonHandler}. */ JsonHandler getJsonHandler(); /** * Get the connector provided {@link FileSystemClient}. * @return An implementation of {@link FileSystemClient}. */ FileSystemClient getFileSystemClient(); /** * Get the connector provided {@link ParquetHandler}. * @return An implementation of {@link ParquetHandler}. */ ParquetHandler getParquetHandler(); } ``` To build your own `Engine` implementation, you can choose to either use the default implementations of each sub-interface or completely build every one from scratch. ```java class MyEngine extends DefaultEngine { FileSystemClient getFileSystemClient() { // Build a new implementation from scratch return new MyFileSystemClient(); } // For all other sub-clients, use the default implementations provided by the `DefaultEngine`. } ``` Next, we will walk through how to implement each interface. #### Step 2.2: Implement `FileSystemClient` interface [Section titled “Step 2.2: Implement FileSystemClient interface”](#step-22-implement-filesystemclient-interface) The [`FileSystemClient`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/FileSystemClient.html) interface contains basic file system operations like listing directories, resolving paths into a fully qualified path and reading bytes from files. Implementation of this interface must take care of the following when interacting with storage systems such as S3, Hadoop, or ADLS: * Credentials and permissions: The connector must populate its `FileSystemClient` with the necessary configurations and credentials for the client to retrieve the necessary data from the storage system. For example, an implementation based on Hadoop’s FileSystem abstractions can be passed S3 credentials via the Hadoop configurations. * Decryption: If file system objects are encrypted, then the implementation must decrypt the data before returning the data. #### Step 2.3: Implement `ParquetHandler` [Section titled “Step 2.3: Implement ParquetHandler”](#step-23-implement-parquethandler) As the name suggests, this interface contains everything related to reading and writing Parquet files. It has been designed such that a connector can plug in a wide variety of implementations, from a simple single-threaded reader to a very advanced multi-threaded reader with pre-fetching and advanced connector-specific expression pushdown. Let’s explore the methods to implement, and the guarantees associated with them. ##### Method `readParquetFiles(CloseableIterator fileIter, StructType physicalSchema, java.util.Optional predicate)` [Section titled “Method readParquetFiles(CloseableIterator\ fileIter, StructType physicalSchema, java.util.Optional\ predicate)”](#method-readparquetfilescloseableiteratorfilestatus-fileiter-structtype-physicalschema-javautiloptionalpredicate-predicate) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ParquetHandler.html#readParquetFiles-io.delta.kernel.utils.CloseableIterator-io.delta.kernel.types.StructType-)) takes as input `FileStatus`s which contains metadata such as file path, size etc. of the Parquet file to read. The columns to be read from the Parquet file are defined by the physical schema. To implement this method, you may have to first implement your own [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) and [`ColumnVector`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnVector.html) which is used to represent the in-memory data generated from the Parquet files. When identifying the columns to read, note that there are multiple types of columns in the physical schema (represented as a [`StructType`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/StructType.html)). * Data columns: Columns that are expected to be read from the Parquet file. Based on the `StructField` object defining the column, read the column in the Parquet file that matches the same name or field id. If the column has a field id (stored as `parquet.field.id` in the `StructField` metadata) then the field id should be used to match the column in the Parquet file. Otherwise, the column name should be used for matching. * Metadata columns: These are special columns that must be populated using metadata about the Parquet file ([`StructField#isMetadataColumn`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/StructField.html#isMetadataColumn--) tells whether a column in `StructType` is a metadata column). To understand how to populate such a column, first match the column name against the set of standard metadata column name constants. For example, * `StructFileld#isMetadataColumn()` returns true and the column name is `StructField.METADATA_ROW_INDEX_COLUMN_NAME`, then you have to a generate column vector populated with the actual index of each row in the Parquet file (that is, not indexed by the possible subset of rows returned after Parquet data skipping). ##### Requirements and guarantees [Section titled “Requirements and guarantees”](#requirements-and-guarantees) Any implementation must adhere to the following guarantees. * The schema of the returned `ColumnarBatch`es must match the physical schema. * If a data column is not found and the `StructField.isNullable = true`, then return a `ColumnVector` of nulls. Throw an error if it is not nullable. * The output iterator must maintain ordering as the input iterator. That is, if `file1` is before `file2` in the input iterator, then columnar batches of `file1` must be before those of `file2` in the output iterator. ##### Method `writeParquetFiles(String directoryPath, CloseableIterator dataIter, java.util.List statsColumns)` [Section titled “Method writeParquetFiles(String directoryPath, CloseableIterator\ dataIter, java.util.List\ statsColumns)”](#method-writeparquetfilesstring-directorypath-closeableiteratorfilteredcolumnarbatch-dataiter-javautillistcolumn-statscolumns) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ParquetHandler.html#writeParquetFiles-java.lang.String-io.delta.kernel.utils.CloseableIterator-java.util.List-) takes given data writes it into one or more Parquet files into the given directory. The data is given as an iterator of [FilteredColumnarBatches](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html) which contains a [ColumnarBatch](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) and an optional selection vector containing one entry for each row in `ColumnarBatch` indicating whether a row is selected or not selected. The `ColumnarBatch` also contains the schema of the data. This schema should be converted to Parquet schema, including any field IDs present [`FieldMetadata`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/FieldMetadata.html) for each column `StructField`. There is also the parameter `statsColumns`, which is a hint to the Parquet writer on what set of columns to collect stats for each file. The statistics include `min`, `max` and `null_count` for each column in the `statsColumns` list. Statistics collection is optional, but when present it is used by Kernel to persist the stats as part of the Delta table commit. This will help read queries prune un-needed data files based on the query predicate. For each written data file, the caller is expecting a [`DataFileStatus`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/DataFileStatus.html) object. It contains the data file path, size, modification time, and optional column statistics. #### Method `writeParquetFileAtomically(String filePath, CloseableIterator data)` [Section titled “Method writeParquetFileAtomically(String filePath, CloseableIterator\ data)”](#method-writeparquetfileatomicallystring-filepath-closeableiteratorfilteredcolumnarbatch-data) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ParquetHandler.html#writeParquetFiles-java.lang.String-io.delta.kernel.utils.CloseableIterator-java.util.List-) writes the given `data` into Parquet file at location `filePath`. The write is an atomic write i.e., either a Parquet file is created with all given content or no Parquet file is created at all. This should not create a file with partial content in it. The default implementation makes use of [`LogStore`](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java) implementations from the [`delta-storage`](https://github.com/delta-io/delta/tree/master/storage) module to accomplish the atomicity. A connector that wants to implement their own version of `ParquetHandler` can take a look at the default implementation for details. ##### Performance suggestions [Section titled “Performance suggestions”](#performance-suggestions) * The representation of data as `ColumnVector`s and `ColumnarBatch`es can have a significant impact on the query performance and it’s best to read the Parquet file data directly into vectors and batches of the engine-native format to avoid potentially costly in-memory data format conversion. Create a Kernel `ColumnVector` and `ColumnarBatch` wrappers around the engine-native format equivalent classes. #### Step 2.4: Implement `ExpressionHandler` interface [Section titled “Step 2.4: Implement ExpressionHandler interface”](#step-24-implement-expressionhandler-interface) The [`ExpressionHandler`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html) interface has all the methods needed for handling expressions that may be applied on columnar data. ##### Method `getEvaluator(StructType batchSchema, Expression expresion, DataType outputType)` [Section titled “Method getEvaluator(StructType batchSchema, Expression expresion, DataType outputType)”](#method-getevaluatorstructtype-batchschema-expression-expresion-datatype-outputtype) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html#getEvaluator-io.delta.kernel.types.StructType-io.delta.kernel.expressions.Expression-io.delta.kernel.types.DataType-) generates an object of type [`ExpressionEvaluator`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/expressions/ExpressionEvaluator.html) that can evaluate the `expression` on a batch of row data to produce a result of a single column vector. To generate this function, the `getEvaluator()` method takes as input the expression and the schema of the `ColumnarBatch`es of data on which the expressions will be applied. The same object can be used to evaluate multiple columnar batches of input with the same schema and expression the evaluator is created for. ##### Method `getPredicateEvaluator(StructType inputSchema, Predicate predicate)` [Section titled “Method getPredicateEvaluator(StructType inputSchema, Predicate predicate)”](#method-getpredicateevaluatorstructtype-inputschema-predicate-predicate) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html#createSelectionVector-boolean:A-int-int-) is for creating an expression evaluator for `Predicate` type expressions. The `Predicate` type expressions return a boolean value as output. The returned object is of type [`PredicateEvaluator`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/expressions/PredicateEvaluator.html). This is a special interface for evaluating Predicate on input batch returns a selection vector containing one value for each row in input batch indicating whether the row has passed the predicate or not. Optionally it takes an existing selection vector along with the input batch for evaluation. The result selection vector is combined with the given existing selection vector and a new selection vector is returned. This mechanism allows running an input batch through several predicate evaluations without rewriting the input batch to remove rows that do not pass the predicate after each predicate evaluation. The new selection should be the same or more selective as the existing selection vector. For example, if a row is marked as unselected in the existing selection vector, then it should remain unselected in the returned selection vector even when the given predicate returns true for the row. ##### Method `createSelectionVector(boolean[] values, int from, int to)` [Section titled “Method createSelectionVector(boolean\[\] values, int from, int to)”](#method-createselectionvectorboolean-values-int-from-int-to) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html#createSelectionVector-boolean:A-int-int-) allows creating `ColumnVector` for boolean type values given as input. This allows the connector to maintain all `ColumnVector`s created in the desired memory format. ##### Requirements and guarantees [Section titled “Requirements and guarantees”](#requirements-and-guarantees-1) Any implementation must adhere to the following guarantees. * Implementation must handle all possible variations of expressions. If the implementation encounters an expression type that it does not know how to handle, then it must throw a specific language-dependent exception. * Java: [NotSupportedException](https://docs.oracle.com/javaee/7/api/javax/resource/NotSupportedException.html) * The `ColumnarBatch`es on which the generated `ExpressionEvaluator` is going to be used are guaranteed to have the schema provided during generation. Hence, it is safe to bind the expression evaluation logic to column ordinals instead of column names, thus making the actual evaluation faster. #### Step 2.5: Implement `JsonHandler` [Section titled “Step 2.5: Implement JsonHandler”](#step-25-implement-jsonhandler) [This](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html) engine interface allows the connector to use plug-in their own JSON handling code and expose it to the Delta Kernel. ##### Method `readJsonFiles(CloseableIterator fileIter, StructType physicalSchema, java.util.Optional predicate)` [Section titled “Method readJsonFiles(CloseableIterator\ fileIter, StructType physicalSchema, java.util.Optional\ predicate)”](#method-readjsonfilescloseableiteratorfilestatus-fileiter-structtype-physicalschema-javautiloptionalpredicate-predicate) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#readJsonFiles-io.delta.kernel.utils.CloseableIterator-io.delta.kernel.types.StructType-) takes as input `FileStatus`s of the JSON files and returns the data in a series of columnar batches. The columns to be read from the JSON file are defined by the physical schema, and the return batches must match that schema. To implement this method, you may have to first implement your own [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html)and [`ColumnVector`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnVector.html) which is used to represent the in-memory data generated from the JSON files. When identifying the columns to read, note that there are multiple types of columns in the physical schema (represented as a [`StructType`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/StructType.html)). ##### Method `parseJson(ColumnVector jsonStringVector, StructType outputSchema, java.util.Optional selectionVector)` [Section titled “Method parseJson(ColumnVector jsonStringVector, StructType outputSchema, java.util.Optional\ selectionVector)”](#method-parsejsoncolumnvector-jsonstringvector-structtype-outputschema-javautiloptionalcolumnvector-selectionvector) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#parseJson-io.delta.kernel.data.ColumnVector-io.delta.kernel.types.StructType-) allows parsing a `ColumnVector` of string values which are in JSON format into the output format specified by the `outputSchema`. If a given column in `outputSchema` is not found, then a null value is returned. It optionally takes a selection vector which indicates what entries in the input `ColumnVector` of strings to parse. If an entry is not selected then a `null` value is returned as parsed output for that particular entry in the output. ##### Method `deserializeStructType(String structTypeJson)` [Section titled “Method deserializeStructType(String structTypeJson)”](#method-deserializestructtypestring-structtypejson) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#deserializeStructType-java.lang.String-) allows parsing JSON encoded (according to [Delta schema serialization rules](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#schema-serialization-format)) `StructType` schema into a `StructType`. Most implementations of `JsonHandler` do not need to implement this method and instead use the one in the [default `JsonHandler`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultJsonHandler.html) implementation. #### Method `writeJsonFileAtomically(String filePath, CloseableIterator data, boolean overwrite)` [Section titled “Method writeJsonFileAtomically(String filePath, CloseableIterator\ data, boolean overwrite)”](#method-writejsonfileatomicallystring-filepath-closeableiteratorrow-data-boolean-overwrite) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#writeJsonFileAtomically-java.lang.String-io.delta.kernel.utils.CloseableIterator-boolean-) writes the given `data` into a JSON file at location `filePath`. The write is an atomic write i.e., either a JSON file is created with all given content or no Parquet file is created at all. This should not create a file with partial content in it. The default implementation makes use of [`LogStore`](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java) implementations from the [`delta-storage`](https://github.com/delta-io/delta/tree/master/storage) module to accomplish the atomicity. A connector that wants to implement their own version of `JsonHandler` can take a look at the default implementation for details. The implementation is expected to handle the serialization rules (converting the `Row` object to JSON string) as described in the [API Javadoc](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#writeJsonFileAtomically-java.lang.String-io.delta.kernel.utils.CloseableIterator-boolean-). #### Step 2.6: Implement `ColumnarBatch` and `ColumnVector` [Section titled “Step 2.6: Implement ColumnarBatch and ColumnVector”](#step-26-implement-columnarbatch-and-columnvector) [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) and [`ColumnVector`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnVector.html) are two interfaces to represent the data read into memory from files. This representation can have a significant impact on query performance. Each engine likely has a native representation of in-memory data with which it applies data transformation operations. For example, in Apache Spark™, the row data is internally represented as `UnsafeRow` for efficient processing. So it’s best to read the Parquet file data directly into vectors and batches of the native format to avoid potentially costly in-memory data format conversions. So the recommended approach is to build wrapper classes that extend the two interfaces but internally use engine-native classes to store the data. When the connector has to forward the columnar batches received from the kernel to the engine, it has to be smart enough to skip converting vectors and batches that are already in the engine-native format. ### Step 3: Build read support in your connector [Section titled “Step 3: Build read support in your connector”](#step-3-build-read-support-in-your-connector) In this section, we are going to walk through the likely sequence of Kernel API calls your connector will have to make to read a table. The exact timing of making these calls in your connector in the context of connector-engine interactions depends entirely on the engine-connector APIs and is therefore beyond the scope of this guide. However, we will try to provide broad guidelines that are likely (but not guaranteed) to apply to your connector-engine setup. For this purpose, we are going to assume that the engine goes through the following phases when processing a read/scan query - logical plan analysis, physical plan generation, and physical plan execution. Based on these broad characterizations, a typical control and data flow for reading a Delta table is going to be as follows: | Step | Typical query phase when this step occurs | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Resolve the table snapshot to query | Logical plan analysis phase when the plan’s schema and other details need to be resolved and validated | | Resolve files to scan based on query parameters | Physical plan generation, when the final parameters of the scan are available. For example: Schema of data to read after pruning away unused columns. Query filters to apply after filter rearrangement | | Distribute the file information to workers | Physical plan execution, only if it is a distributed engine. | | Read the columnar data using the file information | Physical plan execution, when the data is being processed by the engine | Let’s understand the details of each step. #### Step 3.1: Resolve the table snapshot to query [Section titled “Step 3.1: Resolve the table snapshot to query”](#step-31-resolve-the-table-snapshot-to-query) The first step is to resolve the consistent snapshot and the schema associated with it. This is often required by the connector/ engine to resolve and validate the logical plan of the scan query (if the concept of logical plan exists in your engine). To achieve this, the connector has to do the following. * Resolve the table path from the query: If the path is directly available, then this is easy. Otherwise, if it is a query based on a catalog table (for example, a Delta table defined in Hive Metastore), then the connector has to resolve the table path from the catalog. * Initialize the `Engine` object: Create a new instance of the `Engine` that you have chosen in Step 2. * Initialize the Kernel objects and get the schema: Assuming the query is on the latest available version/snapshot of the table, you can get the table schema as follows: ```java import io.delta.kernel.*; import io.delta.kernel.defaults.engine.*; Engine myEngine = new MyEngine(); Table myTable = Table.forPath(myTablePath); Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); StructType mySchema = mySnapshot.getSchema(myEngine); ``` If you want to query a specific version of the table (that is, not the schema), then you can get the required snapshot as `myTable.getSnapshot(version)`. #### Step 3.2: Resolve files to scan [Section titled “Step 3.2: Resolve files to scan”](#step-32-resolve-files-to-scan) Next, we need to build a Scan object using more information from the query. Here we are going to assume that the connector/engine has been able to extract the following details from the query (say, after optimizing the logical plan): * Read schema: The columns in the table that the query needs to read. This may be the full set of columns or a subset of columns. * Query filters: The filters on partitions or data columns that can be used skip reading table data. To provide this information to Kernel, you have to do the following: * Convert the engine-specific schema and filter expressions to Kernel schema and expressions: For schema, you have to create a `StructType` object. For the filters, you have to create an `Expression` object using all the available subclasses of `Expression`. * Build the scan with the converted information: Build the scan as follows: ```java import io.delta.kernel.expressions.*; import io.delta.kernel.types.*; StructType readSchema = ... ; // convert engine schema Predicate filterExpr = ... ; // convert engine filter expression Scan myScan = mySnapshot.buildScan(engine) .withFilter(myEngine, filterExpr) .withReadSchema(myEngine, readSchema) .build() ``` * Resolve the information required to file reads: The generated Scan object has two sets of information. * Scan files: `myScan.getScanFiles()` returns an iterator of `ColumnarBatch`es. Each batch in the iterator contains rows and each row has information about a single file that has been selected based on the query filter. * Scan state: `myScan.getScanState()` returns a `Row` that contains all the information that is common across all the files that need to be read. ````java Row myScanStateRow = myScan.getScanState(); CloseableIterator myScanFilesAsBatches = myScan.getScanFiles(); ```java Row myScanStateRow = myScan.getScanState(); CloseableIterator myScanFilesAsBatches = myScan.getScanFiles(); while (myScanFilesAsBatches.hasNext()) { FilteredColumnarBatch scanFileBatch = myScanFilesAsBatches.next(); CloseableIterator myScanFilesAsRows = scanFileBatch.getRows(); } ```` As we will soon see, reading the columnar data from a selected file will need to use both, the scan state row, and a scan file row with the file information. ##### Requirements and guarantees [Section titled “Requirements and guarantees”](#requirements-and-guarantees-2) Here are the details you need to ensure when defining this scan. * The provided `readSchema` must be the exact schema of the data that the engine will expect when executing the query. Any mismatch in the schema defined during this query planning and the query execution will result in runtime failures. Hence you must build the scan with the readSchema only after the engine has finalized the logical plan after any optimizations like column pruning. * When applicable (for example, with Java Kernel APIs), you have to make sure to call the close() method as you consume the `ColumnarBatch`es of scan files (that is, either serialize the rows or use them to read the table data). #### Step 3.3: Distribute the file information to the workers [Section titled “Step 3.3: Distribute the file information to the workers”](#step-33-distribute-the-file-information-to-the-workers) If you are building a connector for a distributed engine like Spark/Presto/Trino/Flink, then your connector has to send all the scan metadata from the query planning machine (henceforth called the driver) to task execution machines (henceforth called the workers). You will have to serialize and deserialize the scan state and scan file rows. It is the connector job to implement serialization and deserialization utilities for a [`Row`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/Row.html). If the connector wants to split reading one scan file into multiple tasks, it can add additional connector specific split context to the task. At the task, the connector can use its own Parquet reader to read the specific part of the file indicated by the split info. ##### Custom `Row` Serializer/Deserializer [Section titled “Custom Row Serializer/Deserializer”](#custom-row-serializerdeserializer) Here are steps on how to build your own serializer/deserializer such that it will work with any [`Row`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/Row.html) of any schema. * Serializing * First serialize the row schema, that is, `StructType` object. * Then, use the schema to identify types of each column/ordinal in the `Row` and use that to serialize all the values one by one. * Deserializing * Define your own class that extends the Row interface. It must be able to handle complex types like arrays, nested structs and maps. * First deserialize the schema. * Then, use the schema to deserialize the values and put them in an instance of your custom Row class. ```java import io.delta.kernel.utils.*; // In the driver where query planning is being done Byte[] scanStateRowBytes = RowUtils.serialize(scanStateRow); Byte[] scanFileRowBytes = RowUtils.serialize(scanFileRow); // Optionally the connector adds a split info to the task (scan file, scan state) to // split reading of a Parquet file into multiple tasks. The task gets split info // along with the scan file row and scan state row. Split split = ...; // connector specific class, not related to Kernel // Send these over to the worker // In the worker when data will be read, after rowBytes have been sent over Row scanStateRow = RowUtils.deserialize(scanStateRowBytes); Row scanFileRow = RowUtils.deserialize(scanFileRowBytes); Split split = ... deserialize split info ...; ``` #### Step 3.4: Read the columnar data [Section titled “Step 3.4: Read the columnar data”](#step-34-read-the-columnar-data) Finally, we are ready to read the columnar data. You will have to do the following: * Read the physical data from Parquet file as indicated by the scan file row, scan state, and optionally the split info * Convert the physical data into logical data of the table using the Kernel’s APIs. ```java Row scanStateRow = ... ; Row scanFileRow = ... ; Split split = ...; // Additional option predicate such as dynamic filters the connector wants to // pass to the reader when reading files. Predicate optPredicate = ...; // Get the physical read schema of columns to read from the Parquet data files StructType physicalReadSchema = ScanStateRow.getPhysicalDataReadSchema(engine, scanStateRow); // From the scan file row, extract the file path, size and modification metadata // needed to read the file. FileStatus fileStatus = InternalScanFileUtils.getAddFileStatus(scanFileRow); // Open the scan file which is a Parquet file using connector's own // Parquet reader which supports reading specific parts (split) of the file. // If the connector doesn't have its own Parquet reader, it can use the // default Parquet reader provider which at the moment doesn't support reading // a specific part of the file, but reads the entire file from the beginning. CloseableIterator physicalDataIter = connectParquetReader.readParquetFile( fileStatus physicalReadSchema, split, // what part of the Parquet file to read data from optPredicate /* additional predicate the connector can apply to filter data from the reader */ ); // Now the physical data read from the Parquet data file is converted to logical data // the table represents. // Logical data may include the addition of partition columns and/or // subset of rows deleted CloseableIterator transformedData = Scan.transformPhysicalData( engine, scanState, scanFileRow, physicalDataIter)); ``` * Resolve the data in the batches: Each [`FilteredColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html) has two components: * Columnar batch (returned by `FilteredColumnarBatch.getData()`): This is the data read from the files having the schema matching the readSchema provided when the Scan object was built in the earlier step. * Optional selection vector (returned by `FilteredColumnarBatch.getSelectionVector()`): Optionally, a boolean vector that will define which rows in the batch are valid and should be consumed by the engine. If the selection vector is present, then you will have to apply it to the batch to resolve the final consumable data. * Convert to engine-specific data format: Each connector/engine has its own native row / columnar batch formats and interfaces. To return the read data batches to the engine, you have to convert them to fit those engine-specific formats and/or interfaces. Here are a few tips that you can follow to make this efficient. * Matching the engine-specific format: Some engines may expect the data in an in-memory format that may be different from the data produced by `getData()`. So you will have to do the data conversion for each column vector in the batch as needed. * Matching the engine-specific interfaces: You may have to implement wrapper classes that extend the engine-specific interfaces and appropriately encapsulate the row data. For best performance, you can implement your own Parquet reader and other `Engine` implementations to make sure that every `ColumnVector` generated is already in the engine-native format thus eliminating any need to convert. Now you should be able to read the Delta table correctly. ### Step 4: Build append support in your connector [Section titled “Step 4: Build append support in your connector”](#step-4-build-append-support-in-your-connector) In this section, we are going to walk through the likely sequence of Kernel API calls your connector will have to make to append data to a table. The exact timing of making these calls in your connector in the context of connector-engine interactions depends entirely on the engine-connector APIs and is, therefore, beyond the scope of this guide. However, we will try to provide broad guidelines that are likely (but not guaranteed) to apply to your connector-engine setup. For this purpose, we are going to assume that the engine goes through the following phases when processing a write query - logical plan analysis, physical plan generation, and physical plan execution. Based on these broad characterizations, a typical control and data flow for reading a Delta table is going to be as follows: | Step | Typical query phase when this step occurs | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Determine the schema of the data that needs to be written to the table. Schema is derived from the existing table or from the parent operation of the `write` operator in the query plan when the table doesn’t exist yet. | Logical plan analysis phase when the plan’s schema (`write` operator schema matches the table schema, etc.) and other details need to be resolved and validated. | | Determine the physical partitioning of the data based on the table schema and partition columns either from the existing table or from the query plan (for new tables) | Physical plan generation, where the number of writer tasks, data schema and partitioning is determined | | Distribute the writer tasks definitions (which include the transaction state) to workers. | Physical plan execution, only if it is a distributed engine. | | Tasks write the data to data files and send the data file info to the driver. | Physical plan execution, when the data is actually written to the table location | | Finalize the query. Here, all the info of the data files written by the tasks is aggregated and committed to the transaction created at the beginning of the physical execution. | Finalize the query. This happens on the driver where the query has started. | Let’s understand the details of each step. #### Step 4.1: Determine the schema of the data that needs to be written to the table [Section titled “Step 4.1: Determine the schema of the data that needs to be written to the table”](#step-41-determine-the-schema-of-the-data-that-needs-to-be-written-to-the-table) The first step is to resolve the output data schema. This is often required by the connector/ engine to resolve and validate the logical plan of the query (if the concept of logical plan exists in your engine). To achieve this, the connector has to do the following. At a high level query plan is a tree of operators where the leaf-level operators generate or read data from storage/tables and feed it upwards towards the parent operator nodes. This data transfer happens until it reaches the root operator node where the query is finalized (either the results are sent to the client or data is written to another table). * Create the `Table` object * From the `Table` object try to get the schema. * If the table is not found * the query includes creating the table (e.g., `CREATE TABLE AS` SQL query); * the schema is derived from the operator above the `write` that feeds the data to the `write` operator. * the query doesn’t include creating new table, an exception is thrown saying the table is not found * If the table already exists * get the schema from the table and check if it matches the schema of the `write` operator. If not throw an exception. * Create a `TransactionBuilder` - this basically begins the steps of transaction construction. ```java import io.delta.kernel.*; import io.delta.kernel.defaults.engine.*; Engine myEngine = new MyEngine(); Table myTable = Table.forPath(myTablePath); StructType writeOperatorSchema = // ... derived from the query operator tree ... StructType dataSchema; boolean isNewTable = false; try { Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); dataSchema = mySnapshot.getSchema(myEngine); // .. check dataSchema and writeOperatorSchema match ... } catch(TableNotFoundException e) { isNewTable = true; dataSchema = writeOperatorSchema; } TransactionBuilder txnBuilder = myTable.createTransactionBuilder( myEngine, "Examples", /* engineInfo - connector can add its own identifier which is noted in the Delta Log */ Operation /* What is the operation we are trying to perform? This is noted in the Delta Log */ ); if (isNewTable) { // For a new table set the table schema in the transaction builder txnBuilder = txnBuilder.withSchema(engine, dataSchema) } ``` #### Step 4.2: Determine the physical partitioning of the data based on the table schema and partition columns [Section titled “Step 4.2: Determine the physical partitioning of the data based on the table schema and partition columns”](#step-42-determine-the-physical-partitioning-of-the-data-based-on-the-table-schema-and-partition-columns) Partition columns are found either from the query (for new tables, the query defines the partition columns) or from the existing table. ```java TransactionBuilder txnBuilder = ... from the last step ... Transaction txn; List partitionColumns = ... if (newTable) { partitionColumns = ... derive from the query parameters (ex. PARTITION BY clause in SQL) ... txnBuilder = txnBuilder.withPartitionColumns(engine, partitionColumns); txn = txnBuilder.build(engine); } else { txn = txnBuilder.build(engine); partitionColumns = txn.getPartitionColumns(engine); } ``` At the end of this step, we have the `Transaction` and schema of the data to generate and its partitioning. #### Step 4.3: Distribute the writer tasks definitions (which include the transaction state) to workers [Section titled “Step 4.3: Distribute the writer tasks definitions (which include the transaction state) to workers”](#step-43-distribute-the-writer-tasks-definitions-which-include-the-transaction-state-to-workers) If you are building a connector for a distributed engine like Spark/Presto/Trino/Flink, then your connector has to send all the writer metadata from the query planning machine (henceforth called the driver) to task execution machines (henceforth called the workers). You will have to serialize and deserialize the transaction state. It is the connector job to implement serialization and deserialization utilities for a [`Row`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/Row.html). More details on a custom `Row` SerDe are found [here](#custom-row-serializerdeserializer). ```java Row txnState = txn.getState(engine); String jsonTxnState = serializeToJson(txnState); ``` #### Step 4.4: Tasks write the data to data files and send the data file info to the driver [Section titled “Step 4.4: Tasks write the data to data files and send the data file info to the driver”](#step-44-tasks-write-the-data-to-data-files-and-send-the-data-file-info-to-the-driver) In this step (which is executed on the worker nodes inside each task): * Deserialize the transaction state * Writer operator within the task gets the data from its parent operator. * The data is converted into a `FilteredColumnarBatch`. Each [`FilteredColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html) has two components: * Columnar batch (returned by `FilteredColumnarBatch.getData()`): This is the data read from the files having the schema matching the readSchema provided when the Scan object was built in the earlier step. * Optional selection vector (returned by `FilteredColumnarBatch.getSelectionVector()`): Optionally, a boolean vector that will define which rows in the batch are valid and should be consumed by the engine. * The connector can create `FilteredColumnBatch` wrapper around data in its own in-memory format. * Check if the data is partitioned or not. If not partitioned, partition the data by partition values. * For each partition generate the map of the partition column to the partition value * Use Kernel to convert the partitioned data into physical data that should go into the data files * Write the physical data into one or more data files. * Convert data file statues into a Delta log actions * Serialize the Delta log action `Row` objects and send them to the driver node ```plaintext Row txnState = ... deserialize from JSON string sent by the driver ... CloseableIterator data = ... generate data ... // If the table is un-partitioned then this is an empty map Map partitionValues = ... prepare the partition values ... // First transform the logical data to physical data that needs to be written // to the Parquet files CloseableIterator physicalData = Transaction.transformLogicalData(engine, txnState, data, partitionValues); // Get the write context DataWriteContext writeContext = Transaction.getWriteContext(engine, txnState, partitionValues); // Now write the physical data to Parquet files CloseableIterator dataFiles = engine.getParquetHandler() .writeParquetFiles( writeContext.getTargetDirectory(), physicalData, writeContext.getStatisticsColumns()); // Now convert the data file status to data actions that needs to be written to the Delta table log CloseableIterator partitionDataActions = Transaction.generateAppendActions( engine, txnState, dataFiles, writeContext); .... serialize `partitionDataActions` and send them to driver node ``` #### Step 4.5: Finalize the query [Section titled “Step 4.5: Finalize the query”](#step-45-finalize-the-query) At the driver node, the delta log actions from all the tasks are received and committed to the transaction. The tasks send the Delta log actions as a serialized JSON and deserialize them back to `Row` objects. ```plaintext // Create a iterable out of the data actions. If the contents are too big to fit in memory, // the connector may choose to write the data actions to a temporary file and return an // iterator that reads from the file. CloseableIterable dataActionsIterable = CloseableIterable.inMemoryIterable( toCloseableIterator(dataActions.iterator())); // Commit the transaction. TransactionCommitResult commitResult = txn.commit(engine, dataActionsIterable); // Optional step if (commitResult.isReadyForCheckpoint()) { // Checkpoint the table Table.forPath(engine, tablePath).checkpoint(engine, commitResult.getVersion()); } ``` Thats it. Now you should be able to append data to Delta tables using the Kernel APIs. ## Migration guide [Section titled “Migration guide”](#migration-guide) Kernel APIs are still evolving and new features are being added. Kernel authors try to make the API changes backward compatible as much as they can with each new release, but sometimes it is hard to maintain the backward compatibility for a project that is evolving rapidly. This section provides guidance on how to migrate your connector to the latest version of Delta Kernel. With each new release the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) are kept up-to-date with the latest API changes. You can refer to the examples to understand how to use the new APIs. ### Migration from Delta Lake version 3.1.0 to 3.2.0 [Section titled “Migration from Delta Lake version 3.1.0 to 3.2.0”](#migration-from-delta-lake-version-310-to-320) Following are API changes in Delta Kernel 3.2.0 that may require changes in your connector. #### Rename `TableClient` to [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) [Section titled “Rename TableClient to Engine”](#rename-tableclient-to-engine) The `TableClient` interface has been renamed to `Engine`. This is the most significant API change in this release. The `TableClient` interface name is not exactly representing the functionality it provides. At a high level it provides capabilities such as reading Parquet files, JSON files, evaluating expressions on data and file system functionality. These are basically the heavy lift operations that Kernel depends on as a separate interface to allow the connectors to substitute their own custom implementation of the same functionality (e.g. custom Parquet reader). Essentially, these functionalities are the core of the `engine` functionalities. By renaming to `Engine`, we are representing the interface functionality with a proper name that is easy to understand. The `DefaultTableClient` has been renamed to [`DefaultEngine`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultEngine.html). #### [`Table.forPath(Engine engine, String tablePath)`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html#forPath-io.delta.kernel.engine.Engine-java.lang.String-) behavior change [Section titled “Table.forPath(Engine engine, String tablePath) behavior change”](#tableforpathengine-engine-string-tablepath-behavior-change) Earlier when a non-existent table path is passed, the API used to throw `TableNotFoundException`. Now it doesn’t throw the exception. Instead, it returns a `Table` object. When trying to get a `Snapshot` from the table object it throws the `TableNotFoundException`. #### [`FileSystemClient.resolvePath`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultFileSystemClient.html#resolvePath-java.lang.String-) behavior change [Section titled “FileSystemClient.resolvePath behavior change”](#filesystemclientresolvepath-behavior-change) Earlier when a non-existent path is passed, the API used to throw `FileNotFoundException`. Now it doesn’t throw the exception. It still resolves the given path into a fully qualified path. ## Kernel Rust [Section titled “Kernel Rust”](#kernel-rust) The Rust Kernel is a set of libraries for building Delta connectors in native languages. Work in progress. ## More Information [Section titled “More Information”](#more-information) * [Talk](https://www.youtube.com/watch?v=KVUMFv7470I) explaining the rationale behind Kernel and the API design (slides are available [here](https://docs.google.com/presentation/d/1PGSSuJ8ndghucSF9GpYgCi9oeRpWolFyehjQbPh92-U/edit) which are kept up-to-date with the changes). * [User guide](https://github.com/delta-io/delta/blob/master/kernel/USER_GUIDE.md) on the step-by-step process of using Kernel in a standalone Java program or in a distributed processing connector for reading and writing to Delta tables. * Example [Java programs](https://github.com/delta-io/delta/tree/master/kernel/examples) that illustrate how to read and write Delta tables using the Kernel APIs. * Table and default Engine API Java [documentation](/api/latest/java/kernel/index.html) * [Migration guide](https://github.com/delta-io/delta/blob/master/kernel/USER_GUIDE.md#migration-guide) # Delta Kernel Java User Guide > Learn how to build connectors to read and write Delta tables using Delta Kernel Java. ## What is Delta Kernel? [Section titled “What is Delta Kernel?”](#what-is-delta-kernel) Delta Kernel is a library for operating on Delta tables. Specifically, it provides simple and narrow APIs for reading and writing to Delta tables without the need to understand the [Delta protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) details. You can use this library to do the following: * Read and write Delta tables from your applications. * Build a connector for a distributed engine like [Apache Spark™](https://github.com/apache/spark), [Apache Flink](https://github.com/apache/flink), or [Trino](https://github.com/trinodb/trino) for reading or writing massive Delta tables. ## Set up Delta Kernel for your project [Section titled “Set up Delta Kernel for your project”](#set-up-delta-kernel-for-your-project) You need to `io.delta:delta-kernel-api` and `io.delta:delta-kernel-defaults` dependencies. Following is an example Maven `pom` file dependency list. The `delta-kernel-api` module contains the core of the Kernel that abstracts out the Delta protocol to enable reading and writing into Delta tables. It makes use of the `Engine` interface that is being passed to the Kernel API by the connector for heavy-lift operations such as reading/writing Parquet or JSON files, evaluating expressions or file system operations such as listing contents of the Delta Log directory, etc. Kernel supplies a default implementation of `Engine` in module `delta-kernel-defaults`. The connectors can implement their own version of `Engine` to make use of their native implementation of functionalities the `Engine` provides. For example: the connector can make use of their Parquet reader instead of using the reader from the `DefaultEngine`. More details on this [later](#step-2-build-your-own-engine). ```xml io.delta delta-kernel-api ${delta-kernel.version} io.delta delta-kernel-defaults ${delta-kernel.version} ``` If your connector is not using the [`DefaultEngine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) provided by the Kernel, the dependency `delta-kernel-defaults` from the above list can be skipped. ## Read a Delta table in a single process [Section titled “Read a Delta table in a single process”](#read-a-delta-table-in-a-single-process) In this section, we will walk through how to build a very simple single-process Delta connector that can read a Delta table using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. ### Step 1: Full scan on a Delta table [Section titled “Step 1: Full scan on a Delta table”](#step-1-full-scan-on-a-delta-table) The main entry point is [`io.delta.kernel.Table`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html) which is a programmatic representation of a Delta table. Say you have a Delta table at the directory `myTablePath`. You can create a `Table` object as follows: ```java import io.delta.kernel.*; import io.delta.kernel.defaults.*; import org.apache.hadoop.conf.Configuration; String myTablePath = ; // fully qualified table path. Ex: file:/user/tables/myTable Configuration hadoopConf = new Configuration(); Engine myEngine = DefaultEngine.create(hadoopConf); Table myTable = Table.forPath(myEngine, myTablePath); ``` Note the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) we are creating to bootstrap the `myTable` object. This object allows you to plug in your own libraries for computationally intensive operations like Parquet file reading, JSON parsing, etc. You can ignore it for now. We will discuss more about this later when we discuss how to build more complex connectors for distributed processing engines. From this `myTable` object you can create a [`Snapshot`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Snapshot.html) object which represents the consistent state (a.k.a. a snapshot consistency) in a specific version of the table. ```java Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); ``` Now that we have a consistent snapshot view of the table, we can query more details about the table. For example, you can get the version and schema of this snapshot. ```java long version = mySnapshot.getVersion(); StructType tableSchema = mySnapshot.getSchema(); ``` Next, to read the table data, we have to *build* a [`Scan`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html) object. In order to build a `Scan` object, create a [`ScanBuilder`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/ScanBuilder.html) object which optionally allows selecting a subset of columns to read or setting a query filter. For now, ignore these optional settings. ```java Scan myScan = mySnapshot.getScanBuilder().build() // Common information about scanning for all data files to read. Row scanState = myScan.getScanState(myEngine) // Information about the list of scan files to read CloseableIterator scanFiles = myScan.getScanFiles(myEngine) ``` This [`Scan`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html) object has all the necessary metadata to start reading the table. There are two crucial pieces of information needed for reading data from a file in the table. * `myScan.getScanFiles(Engine)`: Returns scan files as columnar batches (represented as an iterator of `FilteredColumnarBatch`es, more on that later) where each selected row in the batch has information about a single file containing the table data. * `myScan.getScanState(Engine)`: Returns the snapshot-level information needed for reading any file. Note that this is a single row and common to all scan files. For each scan file the physical data must be read from the file. The columns to read are specified in the scan file state. Once the physical data is read, you have to call [`ScanFile.transformPhysicalData(...)`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html#transformPhysicalData-io.delta.kernel.engine.Engine-io.delta.kernel.data.Row-io.delta.kernel.data.Row-io.delta.kernel.utils.CloseableIterator-) with the scan state and the physical data read from scan file. This API takes care of transforming (e.g. adding partition columns) the physical data into logical data of the table. Here is an example of reading all the table data in a single thread. ```java CloserableIterator fileIter = scanObject.getScanFiles(myEngine); Row scanStateRow = scanObject.getScanState(myEngine); while(fileIter.hasNext()) { FilteredColumnarBatch scanFileColumnarBatch = fileIter.next(); // Get the physical read schema of columns to read from the Parquet data files StructType physicalReadSchema = ScanStateRow.getPhysicalDataReadSchema(engine, scanStateRow); try (CloseableIterator scanFileRows = scanFileColumnarBatch.getRows()) { while (scanFileRows.hasNext()) { Row scanFileRow = scanFileRows.next(); // From the scan file row, extract the file path, size and modification time metadata // needed to read the file. FileStatus fileStatus = InternalScanFileUtils.getAddFileStatus(scanFileRow); // Open the scan file which is a Parquet file using connector's own // Parquet reader or default Parquet reader provided by the Kernel (which // is used in this example). CloseableIterator physicalDataIter = engine.getParquetHandler().readParquetFiles( singletonCloseableIterator(fileStatus), physicalReadSchema, Optional.empty() /* optional predicate the connector can apply to filter data from the reader */ ); // Now the physical data read from the Parquet data file is converted to a table // logical data. Logical data may include the addition of partition columns and/or // subset of rows deleted try ( CloseableIterator transformedData = Scan.transformPhysicalData( engine, scanStateRow, scanFileRow, physicalDataIter)) { while (transformedData.hasNext()) { FilteredColumnarBatch logicalData = transformedData.next(); ColumnarBatch dataBatch = logicalData.getData(); // Not all rows in `dataBatch` are in the selected output. // An optional selection vector determines whether a row with a // specific row index is in the final output or not. Optional selectionVector = dataReadResult.getSelectionVector(); // access the data for the column at ordinal 0 ColumnVector column0 = dataBatch.getColumnVector(0); for (int rowIndex = 0; rowIndex < column0.getSize(); rowIndex++) { // check if the row is selected or not if (!selectionVector.isPresent() || // there is no selection vector, all records are selected (!selectionVector.get().isNullAt(rowId) && selectionVector.get().getBoolean(rowId))) { // Assuming the column type is String. // If it is a different type, call the relevant function on the `ColumnVector` System.out.println(column0.getString(rowIndex)); } } // access the data for column at ordinal 1 ColumnVector column1 = dataBatch.getColumnVector(1); for (int rowIndex = 0; rowIndex < column1.getSize(); rowIndex++) { // check if the row is selected or not if (!selectionVector.isPresent() || // there is no selection vector, all records are selected (!selectionVector.get().isNullAt(rowId) && selectionVector.get().getBoolean(rowId))) { // Assuming the column type is Long. // If it is a different type, call the relevant function on the `ColumnVector` System.out.println(column1.getLong(rowIndex)); } } // .. more .. } } } } } ``` A few working examples to read Delta tables within a single process are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). important All the Delta protocol-level details are encoded in the rows returned by `Scan.getScanFiles` API, but you do not have to understand them in order to read the table data correctly. All you need is to get the Parquet file status from each scan file row and read the data from the Parquet file into the `ColumnarBatch` format. The physical data is converted into the logical data of the table using [`Scan.transformPhysicalData`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Scan.html#transformPhysicalData-io.delta.kernel.engine.Engine-io.delta.kernel.data.Row-io.delta.kernel.data.Row-io.delta.kernel.utils.CloseableIterator-). Transformation to logical data is dictated by the protocol and the metadata of the table and the scan file. As the Delta protocol evolves this transformation step will evolve with it and your code will not have to change to accommodate protocol changes. This is the major advantage of the abstractions provided by Delta Kernel. Note Observe that the same `Engine` instance `myEngine` is passed multiple times whenever a call to Delta Kernel API is made. The reason for passing this instance for every call is because it is the connector context; it should be maintained outside of the Delta Kernel APIs to give the connector control over the `Engine`. ### Step 2: Improve scan performance with file skipping [Section titled “Step 2: Improve scan performance with file skipping”](#step-2-improve-scan-performance-with-file-skipping) We have explored how to do a full table scan. However, the real advantage of using the Delta format is that you can skip files using your query filters. To make this possible, Delta Kernel provides an [expression framework](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/expressions/package-summary.html) to encode your filters and provide them to Delta Kernel to skip files during the scan file generation. For example, say your table is partitioned by `columnX`, you want to query only the partition `columnX=1`. You can generate the expression and use it to build the scan as follows: ```java import io.delta.kernel.expressions.*; import io.delta.kernel.defaults.engine.*; Engine myEngine = DefaultEngine.create(new Configuration()); Predicate filter = new Predicate( "=", Arrays.asList(new Column("columnX"), Literal.ofInt(1))); Scan myFilteredScan = mySnapshot.getScanBuilder().withFilter(filter).build() // Subset of the given filter that is not guaranteed to be satisfied by // Delta Kernel when it returns data. This filter is used by Delta Kernel // to do data skipping as much as possible. The connector should use this filter // on top of the data returned by Delta Kernel in order for further filtering. Optional remainingFilter = myFilteredScan.getRemainingFilter(); ``` The scan files returned by `myFilteredScan.getScanFiles(myEngine)` will have rows representing files only of the required partition. Similarly, you can provide filters for non-partition columns, and if the data in the table is well clustered by those columns, then Delta Kernel will be able to skip files as much as possible. ## Create a Delta table [Section titled “Create a Delta table”](#create-a-delta-table) In this section, we will walk through how to build a Delta connector that can create a Delta table using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. The main entry point is [`io.delta.kernel.Table`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html) which is a programmatic representation of a Delta table. Say you want to create Delta table at the directory `myTablePath`. You can create a `Table` object as follows: ```java package io.delta.kernel.examples; import io.delta.kernel.*; import io.delta.kernel.types.*; import io.delta.kernel.utils.CloseableIterable; String myTablePath = ; Configuration hadoopConf = new Configuration(); Engine myEngine = DefaultEngine.create(hadoopConf); Table myTable = Table.forPath(myEngine, myTablePath); ``` Note the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) we are creating to bootstrap the `myTable` object. This object allows you to plug in your own libraries for computationally intensive operations like Parquet file reading, JSON parsing, etc. You can ignore it for now. We will discuss more about this later when we discuss how to build more complex connectors for distributed processing engines. From this `myTable` object you can create a [`TransactionBuilder`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionBuilder.html) object which allows you to construct a [`Transaction`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Transaction.html) object ```java TransactionBuilder txnBuilder = myTable.createTransactionBuilder( myEngine, "Examples", /* engineInfo - connector can add its own identifier which is noted in the Delta Log */ Operation.CREATE_TABLE /* What is the operation we are trying to perform. This is noted in the Delta Log */ ); ``` Now that you have the `TransactionBuilder` object, you can set the table schema and partition columns of the table. ```java StructType mySchema = new StructType() .add("id", IntegerType.INTEGER) .add("name", StringType.STRING) .add("city", StringType.STRING) .add("salary", DoubleType.DOUBLE); // Partition columns are optional. Use it only if you are creating a partitioned table. List myPartitionColumns = Collections.singletonList("city"); // Set the schema of the new table on the transaction builder txnBuilder = txnBuilder .withSchema(engine, mySchema); // Set the partition columns of the new table only if you are creating // a partitioned table; otherwise, this step can be skipped. txnBuilder = txnBuilder .withPartitionColumns(engine, examplePartitionColumns); ``` `TransactionBuilder` allows setting additional properties of the table such as enabling a certain Delta feature or setting identifiers for idempotent writes. We will be visiting these in the next sections. The next step is to build `Transaction` out of the `TransactionBuilder` object. ```java // Build the transaction Transaction txn = txnBuilder.build(engine); ``` `Transaction` object allows the connector to optionally add any data and finally commit the transaction. A successful commit ensures that the table is created with the given schema. In this example, we are just creating a table and not adding any data as part of the table. ```java // Commit the transaction. // As we are just creating the table and not adding any data, the `dataActions` is empty. TransactionCommitResult commitResult = txn.commit( engine, CloseableIterable.emptyIterable() /* dataActions */ ); ``` The [`TransactionCommitResult`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionCommitResult.html) contains the what version the transaction is committed as and whether the table is ready for a checkpoint. As we are creating a table the version will be `0`. We will be discussing later on what a checkpoint is and what it means for the table to be ready for the checkpoint. A few working examples to create partitioned and un-partitioned Delta tables are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). ## Create a table and insert data into it [Section titled “Create a table and insert data into it”](#create-a-table-and-insert-data-into-it) In this section, we will walk through how to build a Delta connector that can create a Delta table and insert data into the table (similar to `CREATE TABLE AS ` construct in SQL) using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. The first step is to construct a `Transaction`. Below is the code for that. For more details on what each step of the code means, please read the [create table](#create-a-delta-table) section. ```plaintext package io.delta.kernel.examples; import io.delta.kernel.*; import io.delta.kernel.types.*; import io.delta.kernel.utils.CloseableIterable; String myTablePath = ; Configuration hadoopConf = new Configuration(); Engine myEngine = DefaultEngine.create(hadoopConf); Table myTable = Table.forPath(myEngine, myTablePath); StructType mySchema = new StructType() .add("id", IntegerType.INTEGER) .add("name", StringType.STRING) .add("city", StringType.STRING) .add("salary", DoubleType.DOUBLE); // Partition columns are optional. Use it only if you are creating a partitioned table. List myPartitionColumns = Collections.singletonList("city"); TransactionBuilder txnBuilder = myTable.createTransactionBuilder( myEngine, "Examples", /* engineInfo - connector can add its own identifier which is noted in the Delta Log */ Operation.WRITE /* What is the operation we are trying to perform? This is noted in the Delta Log */ ); // Set the schema of the new table on the transaction builder txnBuilder = txnBuilder .withSchema(engine, mySchema); // Set the partition columns of the new table only if you are creating // a partitioned table; otherwise, this step can be skipped. txnBuilder = txnBuilder .withPartitionColumns(engine, examplePartitionColumns); // Build the transaction Transaction txn = txnBuilder.build(engine); ``` Now that we have the [`Transaction`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Transaction.html) object, the next step is generating the data that confirms the table schema and partitioned according to the table partitions. ```java StructType dataSchema = txn.getSchema(engine) // Optional for un-partitioned tables List partitionColumnNames = txn.getPartitionColumns(engine) ``` Using the data schema and partition column names the connector can plan the query and generate data. At tasks that actually have the data to write to the table, the connector can ask the Kernel to transform the data given in the table schema into physical data that can actually be written to the Parquet data files. For partitioned tables, the data needs to be first partitioned by the partition columns, and then the connector should ask the Kernel to transform the data for each partition separately. The partitioning step is needed because any given data file in the Delta table contains data belonging to exactly one partition. Get the state of the transaction. The transaction state contains the information about how to convert the data in the table schema into physical data that needs to be written. The transformations depend on the protocol and features the table has. ```java Row txnState = txn.getTransactionState(engine); ``` Prepare the data. ```java // The data generated by the connector to write into a table CloseableIterator data = ... // Create partition value map Map partitionValues = Collections.singletonMap( "city", // partition column name // partition value. Depending upon the partition column type, the // partition value should be created. In this example, the partition // column is of type StringType, so we are creating a string literal. Literal.ofString(city) ); ``` The connector data is passed as an iterator of [`FilteredColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html). Each of the `FilteredColumnarBatch` contains a [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) which actually contains the data in columnar access format and an optional section vector that allows the connector to specify which rows from the `ColumnarBatch` to write to the table. Partition values are passed as a map of the partition column name to the partition value. For an un-partitioned table, the map should be empty as it has no partition columns. ```plaintext // Transform the logical data to physical data that needs to be written to the Parquet // files CloseableIterator physicalData = Transaction.transformLogicalData(engine, txnState, data, partitionValues); ``` The above code converts the given data for partitions into an iterator of `FilteredColumnarBatch` that needs to be written to the Parquet data files. In order to write the data files, the connector needs to get the [`WriteContext`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/DataWriteContext.html) from Kernel, which tells the connector where to write the data files and what columns to collect statistics from each data file. ```java // Get the write context DataWriteContext writeContext = Transaction.getWriteContext(engine, txnState, partitionValues); ``` Now, the connector has the physical data that needs to be written to Parquet data files, and where those files should be written, it can start writing the data files. ```java CloseableIterator dataFiles = engine.getParquetHandler() .writeParquetFiles( writeContext.getTargetDirectory(), physicalData, writeContext.getStatisticsColumns() ); ``` In the above code, the connector is making use of the `Engine` provided `ParquetHandler` to write the data, but the connector can choose its own Parquet file writer to write the data. Also note that the return of the above call is an iterator of [`DataFileStatus`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/DataFileStatus.html) for each data file written. It basically contains the file path, file metadata, and optional file-level statistics for columns specified by the [`WriteContext.getStatisticsColumns()`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/DataWriteContext.html#getStatisticsColumns--) Convert each `DataFileStatus` into a Delta log action that can be written to the Delta table log. ```java CloseableIterator dataActions = Transaction.generateAppendActions(engine, txnState, dataFiles, writeContext); ``` The next step is constructing [`CloseableIterable`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/CloseableIterable.html) out of the all the Delta log actions generated above. The reason for constructing an `Iterable` is that the transaction committing involves accessing the list of Delta log actions more than one time (in order to resolve conflicts when there are multiple writes to the table). Kernel provides a [utility method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/CloseableIterable.html#inMemoryIterable-io.delta.kernel.utils.CloseableIterator-) to create an in-memory version of `CloseableIterable`. This interface also gives the connector an option to implement a custom implementation that spills the data actions to disk when the contents are too big to fit in memory. ```java // Create a iterable out of the data actions. If the contents are too big to fit in memory, // the connector may choose to write the data actions to a temporary file and return an // iterator that reads from the file. CloseableIterable dataActionsIterable = CloseableIterable.inMemoryIterable(dataActions); ``` The final step is committing the transaction! ```java TransactionCommitStatus commitStatus = txn.commit(engine, dataActionsIterable) ``` The [`TransactionCommitResult`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionCommitResult.html) contains the what version the transaction is committed as and whether the table is ready for a checkpoint. As we are creating a table the version will be `0`. We will be discussing later on what a checkpoint is and what it means for the table to be ready for the checkpoint. A few working examples to create and insert data into partitioned and un-partitioned Delta tables are available [here](https://github.com/delta-io/delta/tree/master/kernel/examples). ## Blind append into an existing Delta table [Section titled “Blind append into an existing Delta table”](#blind-append-into-an-existing-delta-table) In this section, we will walk through how to build a Delta connector that inserts data into an existing Delta table (similar to `INSERT INTO ` construct in SQL) using the default [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) implementation provided by Delta Kernel. You can either write this code yourself in your project, or you can use the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) present in the Delta code repository. The steps are exactly similar to [Create table and insert data into it](#create-a-table-and-insert-data-into-it) except that we won’t be providing any schema or partition columns when building the `TransactionBuilder` ```java // Create a `Table` object with the given destination table path Table table = Table.forPath(engine, tablePath); // Create a transaction builder to build the transaction TransactionBuilder txnBuilder = table.createTransactionBuilder( engine, "Examples", /* engineInfo */ Operation.WRITE ); / Build the transaction - no need to provide the schema as the table already exists. Transaction txn = txnBuilder.build(engine); // Get the transaction state Row txnState = txn.getTransactionState(engine); List dataActions = new ArrayList<>(); // Generate the sample data for three partitions. Process each partition separately. // This is just an example. In a real-world scenario, the data may come from different // partitions. Connectors already have the capability to partition by partition values // before writing to the table // In the test data `city` is a partition column for (String city : Arrays.asList("San Francisco", "Campbell", "San Jose")) { FilteredColumnarBatch batch1 = generatedPartitionedDataBatch( 5 /* offset */, city /* partition value */); FilteredColumnarBatch batch2 = generatedPartitionedDataBatch( 5 /* offset */, city /* partition value */); FilteredColumnarBatch batch3 = generatedPartitionedDataBatch( 10 /* offset */, city /* partition value */); CloseableIterator data = toCloseableIterator(Arrays.asList(batch1, batch2, batch3).iterator()); // Create partition value map Map partitionValues = Collections.singletonMap( "city", // partition column name // partition value. Depending upon the parition column type, the // partition value should be created. In this example, the partition // column is of type StringType, so we are creating a string literal. Literal.ofString(city)); // First transform the logical data to physical data that needs to be written // to the Parquet // files CloseableIterator physicalData = Transaction.transformLogicalData(engine, txnState, data, partitionValues); // Get the write context DataWriteContext writeContext = Transaction.getWriteContext(engine, txnState, partitionValues); // Now write the physical data to Parquet files CloseableIterator dataFiles = engine.getParquetHandler() .writeParquetFiles( writeContext.getTargetDirectory(), physicalData, writeContext.getStatisticsColumns()); // Now convert the data file status to data actions that needs to be written to the Delta // table log CloseableIterator partitionDataActions = Transaction.generateAppendActions( engine, txnState, dataFiles, writeContext); // Now add all the partition data actions to the main data actions list. In a // distributed query engine, the partition data is written to files at tasks on executor // nodes. The data actions are collected at the driver node and then written to the // Delta table log using the `Transaction.commit` while (partitionDataActions.hasNext()) { dataActions.add(partitionDataActions.next()); } } // Create a iterable out of the data actions. If the contents are too big to fit in memory, // the connector may choose to write the data actions to a temporary file and return an // iterator that reads from the file. CloseableIterable dataActionsIterable = CloseableIterable.inMemoryIterable( toCloseableIterator(dataActions.iterator())); // Commit the transaction. TransactionCommitResult commitResult = txn.commit(engine, dataActionsIterable); ``` ## Idempotent Blind Appends to a Delta Table [Section titled “Idempotent Blind Appends to a Delta Table”](#idempotent-blind-appends-to-a-delta-table) Idempotent writes allow the connector to make sure the data belonging to a particular transaction version and application id is inserted into the table at most once. In incremental processing systems (e.g. streaming systems), track progress using their own application-specific versions need to record what progress has been made in order to avoid duplicating data in the face of failures and retries during writes. By setting the transaction identifier, the Delta table can ensure that the data with the same identifier is not written multiple times. For more information refer to the Delta protocol section [Transaction Identifiers](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#transaction-identifiers) To make the data append idempotent, set the transaction identifier on the [`TransactionBuilder`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionBuilder.html#withTransactionId-io.delta.kernel.engine.Engine-java.lang.String-long-) ```java // Set the transaction identifiers for idempotent writes // Delta/Kernel makes sure that there exists only one transaction in the Delta log // with the given application id and txn version txnBuilder = txnBuilder.withTransactionId( engine, "my app id", /* application id */ 100 /* monotonically increasing txn version with each new data insert */ ); ``` That’s all the connector need to do for idempotent blind appends. ## Checkpointing a Delta table [Section titled “Checkpointing a Delta table”](#checkpointing-a-delta-table) [Checkpoints](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoints) are an optimization in Delta Log in order to construct the state of the Delta table faster. It basically contains the state of the table at the version the checkpoint is created. Delta Kernel allows the connector to optionally make the checkpoints. It is created for every few commits (configurable table property) on the table. The result of `Transaction.commit` returns a `TransactionCommitResult` that contains the version the transaction is committed as and whether the table is [read for checkpoint](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/TransactionCommitResult.html#isReadyForCheckpoint--). Creating a checkpoint takes time as it needs to construct the entire state of the table. If the connector doesn’t want to checkpoint by itself but uses other connectors that are faster in creating a checkpoint, it can skip the checkpointing step. If it wants to checkpoint, the `Table` object has an [API](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html#checkpoint-io.delta.kernel.engine.Engine-long-) to checkpoint the table. ```java TransactionCommitResult commitResult = txn.commit(engine, dataActionsIterable); if (commitResult.isReadyForCheckpoint()) { // Checkpoint the table Table.forPath(engine, tablePath).checkpoint(engine, commitResult.getVersion()); } ``` ## Build a Delta connector for a distributed processing engine [Section titled “Build a Delta connector for a distributed processing engine”](#build-a-delta-connector-for-a-distributed-processing-engine) Unlike simple applications that just read the table in a single process, building a connector for complex processing engines like Apache Spark™ and Trino can require quite a bit of additional effort. For example, to build a connector for an SQL engine you have to do the following * Understand the APIs provided by the engine to build connectors and how Delta Kernel can be used to provide the information necessary for the connector + engine to operate on a Delta table. * Decide what libraries to use to do computationally expensive operations like reading Parquet files, parsing JSON, computing expressions, etc. Delta Kernel provides all the extension points to allow you to plug in any library without having to understand all the low-level details of the Delta protocol. * Deal with details specific to distributed engines. For example, * Serialization of Delta table metadata provided by Delta Kernel. * Efficiently transforming data read from Parquet into the engine in-memory processing format. In this section, we are going to outline the steps needed to build a connector. ### Step 0: Validate the prerequisites [Section titled “Step 0: Validate the prerequisites”](#step-0-validate-the-prerequisites) In the previous section showing how to read a simple table, we were briefly introduced to the [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html). This is the main extension point where you can plug in your implementations of computationally-expensive operations like reading Parquet files, parsing JSON, etc. For the simple case, we were using a default implementation of the helper that works in most cases. However, for building a high-performance connector for a complex processing engine, you will very likely need to provide your own implementation using the libraries that work with your engine. So before you start building your connector, it is important to understand these requirements and plan for building your own engine. Here are the libraries/capabilities you need to build a connector that can read the Delta table * Perform file listing and file reads from your storage/file system. * Read Parquet files in columnar data, preferably in an in-memory columnar format. * Parse JSON data * Read JSON files * Evaluate expressions on in-memory columnar batches For each of these capabilities, you can choose to build your own implementation or reuse the default implementation. ### Step 1: Set up Delta Kernel in your connector project [Section titled “Step 1: Set up Delta Kernel in your connector project”](#step-1-set-up-delta-kernel-in-your-connector-project) In the Delta Kernel project, there are multiple dependencies you can choose to depend on. 1. Delta Kernel core APIs - This is a must-have dependency, which contains all the main APIs like Table, Snapshot, and Scan that you will use to access the metadata and data of the Delta table. This has very few dependencies reducing the chance of conflicts with any dependencies in your connector and engine. This also provides the [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) interface which allows you to plug in your implementations of computationally expensive operations, but it does not provide any implementation of this interface. 2. Delta Kernel default- This has a default implementation called [`DefaultEngine`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultEngine.html) and additional dependencies such as `Hadoop`. If you wish to reuse all or parts of this implementation, then you can optionally depend on this. #### Set up Java projects [Section titled “Set up Java projects”](#set-up-java-projects) As discussed above, you can import one or both of the artifacts as follows: ```xml io.delta delta-kernel-api ${delta-kernel.version} io.delta delta-kernel-defaults ${delta-kernel.version} ``` ### Step 2: Build your own Engine [Section titled “Step 2: Build your own Engine”](#step-2-build-your-own-engine) In this section, we are going to explore the [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) interface and walk through how to implement your own implementation so that you can plug in your connector/engine-specific implementations of computationally-intensive operations, threading model, resource management, etc. important During the validation process, if you believe that all the dependencies of the default `Engine` implementation can work with your connector and engine, then you can skip this step and jump to Step 3 of implementing your connector using the default engine. If later you have the need to customize the helper for your connector, you can revisit this step. #### Step 2.1: Implement the `Engine` interface [Section titled “Step 2.1: Implement the Engine interface”](#step-21-implement-the-engine-interface) The [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) interface combines a bunch of sub-interfaces each of which is designed for a specific purpose. Here is a brief overview of the subinterfaces. See the API docs (Java) for a more detailed view. ```java interface Engine { /** * Get the connector provided {@link ExpressionHandler}. * @return An implementation of {@link ExpressionHandler}. */ ExpressionHandler getExpressionHandler(); /** * Get the connector provided {@link JsonHandler}. * @return An implementation of {@link JsonHandler}. */ JsonHandler getJsonHandler(); /** * Get the connector provided {@link FileSystemClient}. * @return An implementation of {@link FileSystemClient}. */ FileSystemClient getFileSystemClient(); /** * Get the connector provided {@link ParquetHandler}. * @return An implementation of {@link ParquetHandler}. */ ParquetHandler getParquetHandler(); } ``` To build your own `Engine` implementation, you can choose to either use the default implementations of each sub-interface or completely build every one from scratch. ```java class MyEngine extends DefaultEngine { FileSystemClient getFileSystemClient() { // Build a new implementation from scratch return new MyFileSystemClient(); } // For all other sub-clients, use the default implementations provided by the `DefaultEngine`. } ``` Next, we will walk through how to implement each interface. #### Step 2.2: Implement `FileSystemClient` interface [Section titled “Step 2.2: Implement FileSystemClient interface”](#step-22-implement-filesystemclient-interface) The [`FileSystemClient`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/FileSystemClient.html) interface contains basic file system operations like listing directories, resolving paths into a fully qualified path and reading bytes from files. Implementation of this interface must take care of the following when interacting with storage systems such as S3, Hadoop, or ADLS: * Credentials and permissions: The connector must populate its `FileSystemClient` with the necessary configurations and credentials for the client to retrieve the necessary data from the storage system. For example, an implementation based on Hadoop’s FileSystem abstractions can be passed S3 credentials via the Hadoop configurations. * Decryption: If file system objects are encrypted, then the implementation must decrypt the data before returning the data. #### Step 2.3: Implement `ParquetHandler` [Section titled “Step 2.3: Implement ParquetHandler”](#step-23-implement-parquethandler) As the name suggests, this interface contains everything related to reading and writing Parquet files. It has been designed such that a connector can plug in a wide variety of implementations, from a simple single-threaded reader to a very advanced multi-threaded reader with pre-fetching and advanced connector-specific expression pushdown. Let’s explore the methods to implement, and the guarantees associated with them. ##### Method `readParquetFiles(CloseableIterator fileIter, StructType physicalSchema, java.util.Optional predicate)` [Section titled “Method readParquetFiles(CloseableIterator\ fileIter, StructType physicalSchema, java.util.Optional\ predicate)”](#method-readparquetfilescloseableiteratorfilestatus-fileiter-structtype-physicalschema-javautiloptionalpredicate-predicate) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ParquetHandler.html#readParquetFiles-io.delta.kernel.utils.CloseableIterator-io.delta.kernel.types.StructType-) takes as input `FileStatus`s which contains metadata such as file path, size etc. of the Parquet file to read. The columns to be read from the Parquet file are defined by the physical schema. To implement this method, you may have to first implement your own [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) and [`ColumnVector`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnVector.html) which is used to represent the in-memory data generated from the Parquet files. When identifying the columns to read, note that there are multiple types of columns in the physical schema (represented as a [`StructType`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/StructType.html)). * Data columns: Columns that are expected to be read from the Parquet file. Based on the `StructField` object defining the column, read the column in the Parquet file that matches the same name or field id. If the column has a field id (stored as `parquet.field.id` in the `StructField` metadata) then the field id should be used to match the column in the Parquet file. Otherwise, the column name should be used for matching. * Metadata columns: These are special columns that must be populated using metadata about the Parquet file ([`StructField#isMetadataColumn`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/StructField.html#isMetadataColumn--) tells whether a column in `StructType` is a metadata column). To understand how to populate such a column, first match the column name against the set of standard metadata column name constants. For example, * `StructFileld#isMetadataColumn()` returns true and the column name is `StructField.METADATA_ROW_INDEX_COLUMN_NAME`, then you have to a generate column vector populated with the actual index of each row in the Parquet file (that is, not indexed by the possible subset of rows returned after Parquet data skipping). ##### Requirements and guarantees [Section titled “Requirements and guarantees”](#requirements-and-guarantees) Any implementation must adhere to the following guarantees. * The schema of the returned `ColumnarBatch`es must match the physical schema. * If a data column is not found and the `StructField.isNullable = true`, then return a `ColumnVector` of nulls. Throw an error if it is not nullable. * The output iterator must maintain ordering as the input iterator. That is, if `file1` is before `file2` in the input iterator, then columnar batches of `file1` must be before those of `file2` in the output iterator. ##### Method `writeParquetFiles(String directoryPath, CloseableIterator dataIter, java.util.List statsColumns)` [Section titled “Method writeParquetFiles(String directoryPath, CloseableIterator\ dataIter, java.util.List\ statsColumns)”](#method-writeparquetfilesstring-directorypath-closeableiteratorfilteredcolumnarbatch-dataiter-javautillistcolumn-statscolumns) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ParquetHandler.html#writeParquetFiles-java.lang.String-io.delta.kernel.utils.CloseableIterator-java.util.List-) takes given data writes it into one or more Parquet files into the given directory. The data is given as an iterator of [FilteredColumnarBatches](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html) which contains a [ColumnarBatch](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) and an optional selection vector containing one entry for each row in `ColumnarBatch` indicating whether a row is selected or not selected. The `ColumnarBatch` also contains the schema of the data. This schema should be converted to Parquet schema, including any field IDs present [`FieldMetadata`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/FieldMetadata.html) for each column `StructField`. There is also the parameter `statsColumns`, which is a hint to the Parquet writer on what set of columns to collect stats for each file. The statistics include `min`, `max` and `null_count` for each column in the `statsColumns` list. Statistics collection is optional, but when present it is used by Kernel to persist the stats as part of the Delta table commit. This will help read queries prune un-needed data files based on the query predicate. For each written data file, the caller is expecting a [`DataFileStatus`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/utils/DataFileStatus.html) object. It contains the data file path, size, modification time, and optional column statistics. #### Method `writeParquetFileAtomically(String filePath, CloseableIterator data)` [Section titled “Method writeParquetFileAtomically(String filePath, CloseableIterator\ data)”](#method-writeparquetfileatomicallystring-filepath-closeableiteratorfilteredcolumnarbatch-data) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ParquetHandler.html#writeParquetFiles-java.lang.String-io.delta.kernel.utils.CloseableIterator-java.util.List-) writes the given `data` into Parquet file at location `filePath`. The write is an atomic write i.e., either a Parquet file is created with all given content or no Parquet file is created at all. This should not create a file with partial content in it. The default implementation makes use of [`LogStore`](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java) implementations from the [`delta-storage`](https://github.com/delta-io/delta/tree/master/storage) module to accomplish the atomicity. A connector that wants to implement their own version of `ParquetHandler` can take a look at the default implementation for details. ##### Performance suggestions [Section titled “Performance suggestions”](#performance-suggestions) * The representation of data as `ColumnVector`s and `ColumnarBatch`es can have a significant impact on the query performance and it’s best to read the Parquet file data directly into vectors and batches of the engine-native format to avoid potentially costly in-memory data format conversion. Create a Kernel `ColumnVector` and `ColumnarBatch` wrappers around the engine-native format equivalent classes. #### Step 2.4: Implement `ExpressionHandler` interface [Section titled “Step 2.4: Implement ExpressionHandler interface”](#step-24-implement-expressionhandler-interface) The [`ExpressionHandler`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html) interface has all the methods needed for handling expressions that may be applied on columnar data. ##### Method `getEvaluator(StructType batchSchema, Expression expresion, DataType outputType)` [Section titled “Method getEvaluator(StructType batchSchema, Expression expresion, DataType outputType)”](#method-getevaluatorstructtype-batchschema-expression-expresion-datatype-outputtype) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html#getEvaluator-io.delta.kernel.types.StructType-io.delta.kernel.expressions.Expression-io.delta.kernel.types.DataType-) generates an object of type [`ExpressionEvaluator`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/expressions/ExpressionEvaluator.html) that can evaluate the `expression` on a batch of row data to produce a result of a single column vector. To generate this function, the `getEvaluator()` method takes as input the expression and the schema of the `ColumnarBatch`es of data on which the expressions will be applied. The same object can be used to evaluate multiple columnar batches of input with the same schema and expression the evaluator is created for. ##### Method `getPredicateEvaluator(StructType inputSchema, Predicate predicate)` [Section titled “Method getPredicateEvaluator(StructType inputSchema, Predicate predicate)”](#method-getpredicateevaluatorstructtype-inputschema-predicate-predicate) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html#createSelectionVector-boolean:A-int-int-) is for creating an expression evaluator for `Predicate` type expressions. The `Predicate` type expressions return a boolean value as output. The returned object is of type [`PredicateEvaluator`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/expressions/PredicateEvaluator.html). This is a special interface for evaluating Predicate on input batch returns a selection vector containing one value for each row in input batch indicating whether the row has passed the predicate or not. Optionally it takes an existing selection vector along with the input batch for evaluation. The result selection vector is combined with the given existing selection vector and a new selection vector is returned. This mechanism allows running an input batch through several predicate evaluations without rewriting the input batch to remove rows that do not pass the predicate after each predicate evaluation. The new selection should be the same or more selective as the existing selection vector. For example, if a row is marked as unselected in the existing selection vector, then it should remain unselected in the returned selection vector even when the given predicate returns true for the row. ##### Method `createSelectionVector(boolean[] values, int from, int to)` [Section titled “Method createSelectionVector(boolean\[\] values, int from, int to)”](#method-createselectionvectorboolean-values-int-from-int-to) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/ExpressionHandler.html#createSelectionVector-boolean:A-int-int-) allows creating `ColumnVector` for boolean type values given as input. This allows the connector to maintain all `ColumnVector`s created in the desired memory format. ##### Requirements and guarantees [Section titled “Requirements and guarantees”](#requirements-and-guarantees-1) Any implementation must adhere to the following guarantees. * Implementation must handle all possible variations of expressions. If the implementation encounters an expression type that it does not know how to handle, then it must throw a specific language-dependent exception. * Java: [NotSupportedException](https://docs.oracle.com/javaee/7/api/javax/resource/NotSupportedException.html) * The `ColumnarBatch`es on which the generated `ExpressionEvaluator` is going to be used are guaranteed to have the schema provided during generation. Hence, it is safe to bind the expression evaluation logic to column ordinals instead of column names, thus making the actual evaluation faster. #### Step 2.5: Implement `JsonHandler` [Section titled “Step 2.5: Implement JsonHandler”](#step-25-implement-jsonhandler) [This](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html) engine interface allows the connector to use plug-in their own JSON handling code and expose it to the Delta Kernel. ##### Method `readJsonFiles(CloseableIterator fileIter, StructType physicalSchema, java.util.Optional predicate)` [Section titled “Method readJsonFiles(CloseableIterator\ fileIter, StructType physicalSchema, java.util.Optional\ predicate)”](#method-readjsonfilescloseableiteratorfilestatus-fileiter-structtype-physicalschema-javautiloptionalpredicate-predicate) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#readJsonFiles-io.delta.kernel.utils.CloseableIterator-io.delta.kernel.types.StructType-) takes as input `FileStatus`s of the JSON files and returns the data in a series of columnar batches. The columns to be read from the JSON file are defined by the physical schema, and the return batches must match that schema. To implement this method, you may have to first implement your own [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html)and [`ColumnVector`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnVector.html) which is used to represent the in-memory data generated from the JSON files. When identifying the columns to read, note that there are multiple types of columns in the physical schema (represented as a [`StructType`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/types/StructType.html)). ##### Method `parseJson(ColumnVector jsonStringVector, StructType outputSchema, java.util.Optional selectionVector)` [Section titled “Method parseJson(ColumnVector jsonStringVector, StructType outputSchema, java.util.Optional\ selectionVector)”](#method-parsejsoncolumnvector-jsonstringvector-structtype-outputschema-javautiloptionalcolumnvector-selectionvector) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#parseJson-io.delta.kernel.data.ColumnVector-io.delta.kernel.types.StructType-) allows parsing a `ColumnVector` of string values which are in JSON format into the output format specified by the `outputSchema`. If a given column in `outputSchema` is not found, then a null value is returned. It optionally takes a selection vector which indicates what entries in the input `ColumnVector` of strings to parse. If an entry is not selected then a `null` value is returned as parsed output for that particular entry in the output. ##### Method `deserializeStructType(String structTypeJson)` [Section titled “Method deserializeStructType(String structTypeJson)”](#method-deserializestructtypestring-structtypejson) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#deserializeStructType-java.lang.String-) allows parsing JSON encoded (according to [Delta schema serialization rules](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#schema-serialization-format)) `StructType` schema into a `StructType`. Most implementations of `JsonHandler` do not need to implement this method and instead use the one in the [default `JsonHandler`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultJsonHandler.html) implementation. #### Method `writeJsonFileAtomically(String filePath, CloseableIterator data, boolean overwrite)` [Section titled “Method writeJsonFileAtomically(String filePath, CloseableIterator\ data, boolean overwrite)”](#method-writejsonfileatomicallystring-filepath-closeableiteratorrow-data-boolean-overwrite) This [method](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#writeJsonFileAtomically-java.lang.String-io.delta.kernel.utils.CloseableIterator-boolean-) writes the given `data` into a JSON file at location `filePath`. The write is an atomic write i.e., either a JSON file is created with all given content or no Parquet file is created at all. This should not create a file with partial content in it. The default implementation makes use of [`LogStore`](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java) implementations from the [`delta-storage`](https://github.com/delta-io/delta/tree/master/storage) module to accomplish the atomicity. A connector that wants to implement their own version of `JsonHandler` can take a look at the default implementation for details. The implementation is expected to handle the serialization rules (converting the `Row` object to JSON string) as described in the [API Javadoc](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/JsonHandler.html#writeJsonFileAtomically-java.lang.String-io.delta.kernel.utils.CloseableIterator-boolean-). #### Step 2.6: Implement `ColumnarBatch` and `ColumnVector` [Section titled “Step 2.6: Implement ColumnarBatch and ColumnVector”](#step-26-implement-columnarbatch-and-columnvector) [`ColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnarBatch.html) and [`ColumnVector`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/ColumnVector.html) are two interfaces to represent the data read into memory from files. This representation can have a significant impact on query performance. Each engine likely has a native representation of in-memory data with which it applies data transformation operations. For example, in Apache Spark™, the row data is internally represented as `UnsafeRow` for efficient processing. So it’s best to read the Parquet file data directly into vectors and batches of the native format to avoid potentially costly in-memory data format conversions. So the recommended approach is to build wrapper classes that extend the two interfaces but internally use engine-native classes to store the data. When the connector has to forward the columnar batches received from the kernel to the engine, it has to be smart enough to skip converting vectors and batches that are already in the engine-native format. ### Step 3: Build read support in your connector [Section titled “Step 3: Build read support in your connector”](#step-3-build-read-support-in-your-connector) In this section, we are going to walk through the likely sequence of Kernel API calls your connector will have to make to read a table. The exact timing of making these calls in your connector in the context of connector-engine interactions depends entirely on the engine-connector APIs and is therefore beyond the scope of this guide. However, we will try to provide broad guidelines that are likely (but not guaranteed) to apply to your connector-engine setup. For this purpose, we are going to assume that the engine goes through the following phases when processing a read/scan query - logical plan analysis, physical plan generation, and physical plan execution. Based on these broad characterizations, a typical control and data flow for reading a Delta table is going to be as follows: .. list-table:: :header-rows: 1 :widths: 30 70 Step Typical query phase when this step occurs Resolve the table snapshot to query Logical plan analysis phase when the plan’s schema and other details need to be resolved and validated Resolve files to scan based on query parameters Physical plan generation, when the final parameters of the scan are available. For example: Schema of data to read after pruning away unused columns Query filters to apply after filter rearrangement Distribute the file information to workers Physical plan execution, only if it is a distributed engine. Read the columnar data using the file information Physical plan execution, when the data is being processed by the engine Let’s understand the details of each step. #### Step 3.1: Resolve the table snapshot to query [Section titled “Step 3.1: Resolve the table snapshot to query”](#step-31-resolve-the-table-snapshot-to-query) The first step is to resolve the consistent snapshot and the schema associated with it. This is often required by the connector/ engine to resolve and validate the logical plan of the scan query (if the concept of logical plan exists in your engine). To achieve this, the connector has to do the following. * Resolve the table path from the query: If the path is directly available, then this is easy. Otherwise, if it is a query based on a catalog table (for example, a Delta table defined in Hive Metastore), then the connector has to resolve the table path from the catalog. * Initialize the `Engine` object: Create a new instance of the `Engine` that you have chosen in \[Step 2]\(#build-your-own Engine). * Initialize the Kernel objects and get the schema: Assuming the query is on the latest available version/snapshot of the table, you can get the table schema as follows: ```java import io.delta.kernel.*; import io.delta.kernel.defaults.engine.*; Engine myEngine = new MyEngine(); Table myTable = Table.forPath(myTablePath); Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); StructType mySchema = mySnapshot.getSchema(myEngine); ``` If you want to query a specific version of the table (that is, not the schema), then you can get the required snapshot as `myTable.getSnapshot(version)`. #### Step 3.2: Resolve files to scan [Section titled “Step 3.2: Resolve files to scan”](#step-32-resolve-files-to-scan) Next, we need to build a Scan object using more information from the query. Here we are going to assume that the connector/engine has been able to extract the following details from the query (say, after optimizing the logical plan): * Read schema: The columns in the table that the query needs to read. This may be the full set of columns or a subset of columns. * Query filters: The filters on partitions or data columns that can be used skip reading table data. To provide this information to Kernel, you have to do the following: * Convert the engine-specific schema and filter expressions to Kernel schema and expressions: For schema, you have to create a `StructType` object. For the filters, you have to create an `Expression` object using all the available subclasses of `Expression`. * Build the scan with the converted information: Build the scan as follows: ```java import io.delta.kernel.expressions.*; import io.delta.kernel.types.*; StructType readSchema = ... ; // convert engine schema Predicate filterExpr = ... ; // convert engine filter expression Scan myScan = mySnapshot.getScanBuilder().withFilter(filterExpr).withReadSchema(readSchema).build(); ``` * Resolve the information required to file reads: The generated Scan object has two sets of information. * Scan files: `myScan.getScanFiles()` returns an iterator of `ColumnarBatch`es. Each batch in the iterator contains rows and each row has information about a single file that has been selected based on the query filter. * Scan state: `myScan.getScanState()` returns a `Row` that contains all the information that is common across all the files that need to be read. ```java Row myScanStateRow = myScan.getScanState(); CloseableIterator myScanFilesAsBatches = myScan.getScanFiles(); while (myScanFilesAsBatches.hasNext()) { FilteredColumnarBatch scanFileBatch = myScanFilesAsBatches.next(); CloseableIterator myScanFilesAsRows = scanFileBatch.getRows(); } ``` As we will soon see, reading the columnar data from a selected file will need to use both, the scan state row, and a scan file row with the file information. ##### Requirements and guarantees [Section titled “Requirements and guarantees”](#requirements-and-guarantees-2) Here are the details you need to ensure when defining this scan. * The provided `readSchema` must be the exact schema of the data that the engine will expect when executing the query. Any mismatch in the schema defined during this query planning and the query execution will result in runtime failures. Hence you must build the scan with the readSchema only after the engine has finalized the logical plan after any optimizations like column pruning. * When applicable (for example, with Java Kernel APIs), you have to make sure to call the close() method as you consume the `ColumnarBatch`es of scan files (that is, either serialize the rows or use them to read the table data). #### Step 3.3: Distribute the file information to the workers [Section titled “Step 3.3: Distribute the file information to the workers”](#step-33-distribute-the-file-information-to-the-workers) If you are building a connector for a distributed engine like Spark/Presto/Trino/Flink, then your connector has to send all the scan metadata from the query planning machine (henceforth called the driver) to task execution machines (henceforth called the workers). You will have to serialize and deserialize the scan state and scan file rows. It is the connector job to implement serialization and deserialization utilities for a [`Row`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/Row.html). If the connector wants to split reading one scan file into multiple tasks, it can add additional connector specific split context to the task. At the task, the connector can use its own Parquet reader to read the specific part of the file indicated by the split info. ##### Custom `Row` Serializer/Deserializer [Section titled “Custom Row Serializer/Deserializer”](#custom-row-serializerdeserializer) Here are steps on how to build your own serializer/deserializer such that it will work with any [`Row`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/Row.html) of any schema. * Serializing * First serialize the row schema, that is, `StructType` object. * Then, use the schema to identify types of each column/ordinal in the `Row` and use that to serialize all the values one by one. * Deserializing * Define your own class that extends the Row interface. It must be able to handle complex types like arrays, nested structs and maps. * First deserialize the schema. * Then, use the schema to deserialize the values and put them in an instance of your custom Row class. ```java import io.delta.kernel.utils.*; // In the driver where query planning is being done Byte[] scanStateRowBytes = RowUtils.serialize(scanStateRow); Byte[] scanFileRowBytes = RowUtils.serialize(scanFileRow); // Optionally the connector adds a split info to the task (scan file, scan state) to // split reading of a Parquet file into multiple tasks. The task gets split info // along with the scan file row and scan state row. Split split = ...; // connector specific class, not related to Kernel // Send these over to the worker // In the worker when data will be read, after rowBytes have been sent over Row scanStateRow = RowUtils.deserialize(scanStateRowBytes); Row scanFileRow = RowUtils.deserialize(scanFileRowBytes); Split split = ... deserialize split info ...; ``` #### Step 3.4: Read the columnar data [Section titled “Step 3.4: Read the columnar data”](#step-34-read-the-columnar-data) Finally, we are ready to read the columnar data. You will have to do the following: * Read the physical data from Parquet file as indicated by the scan file row, scan state, and optionally the split info * Convert the physical data into logical data of the table using the Kernel’s APIs. ```java Row scanStateRow = ... ; Row scanFileRow = ... ; Split split = ...; // Additional option predicate such as dynamic filters the connector wants to // pass to the reader when reading files. Predicate optPredicate = ...; // Get the physical read schema of columns to read from the Parquet data files StructType physicalReadSchema = ScanStateRow.getPhysicalDataReadSchema(engine, scanStateRow); // From the scan file row, extract the file path, size and modification metadata // needed to read the file. FileStatus fileStatus = InternalScanFileUtils.getAddFileStatus(scanFileRow); // Open the scan file which is a Parquet file using connector's own // Parquet reader which supports reading specific parts (split) of the file. // If the connector doesn't have its own Parquet reader, it can use the // default Parquet reader provider which at the moment doesn't support reading // a specific part of the file, but reads the entire file from the beginning. CloseableIterator physicalDataIter = connectParquetReader.readParquetFile( fileStatus physicalReadSchema, split, // what part of the Parquet file to read data from optPredicate /* additional predicate the connector can apply to filter data from the reader */ ); // Now the physical data read from the Parquet data file is converted to logical data // the table represents. // Logical data may include the addition of partition columns and/or // subset of rows deleted CloseableIterator transformedData = Scan.transformPhysicalData( engine, scanState, scanFileRow, physicalDataIter)); ``` * Resolve the data in the batches: Each [`FilteredColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html) has two components: * Columnar batch (returned by `FilteredColumnarBatch.getData()`): This is the data read from the files having the schema matching the readSchema provided when the Scan object was built in the earlier step. * Optional selection vector (returned by `FilteredColumnarBatch.getSelectionVector()`): Optionally, a boolean vector that will define which rows in the batch are valid and should be consumed by the engine. If the selection vector is present, then you will have to apply it to the batch to resolve the final consumable data. * Convert to engine-specific data format: Each connector/engine has its own native row / columnar batch formats and interfaces. To return the read data batches to the engine, you have to convert them to fit those engine-specific formats and/or interfaces. Here are a few tips that you can follow to make this efficient. * Matching the engine-specific format: Some engines may expect the data in an in-memory format that may be different from the data produced by `getData()`. So you will have to do the data conversion for each column vector in the batch as needed. * Matching the engine-specific interfaces: You may have to implement wrapper classes that extend the engine-specific interfaces and appropriately encapsulate the row data. For best performance, you can implement your own Parquet reader and other `Engine` implementations to make sure that every `ColumnVector` generated is already in the engine-native format thus eliminating any need to convert. Now you should be able to read the Delta table correctly. ### Step 4: Build append support in your connector [Section titled “Step 4: Build append support in your connector”](#step-4-build-append-support-in-your-connector) In this section, we are going to walk through the likely sequence of Kernel API calls your connector will have to make to append data to a table. The exact timing of making these calls in your connector in the context of connector-engine interactions depends entirely on the engine-connector APIs and is, therefore, beyond the scope of this guide. However, we will try to provide broad guidelines that are likely (but not guaranteed) to apply to your connector-engine setup. For this purpose, we are going to assume that the engine goes through the following phases when processing a write query - logical plan analysis, physical plan generation, and physical plan execution. Based on these broad characterizations, a typical control and data flow for reading a Delta table is going to be as follows: * Typical steps and query phases | Step | Typical query phase when this step occurs | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Determine the schema of the data that needs to be written to the table. Schema is derived from the existing table or from the parent operation of the `write` operator in the query plan when the table doesn’t exist yet. | Logical plan analysis phase when the plan’s schema (`write` operator schema matches the table schema, etc.) and other details need to be resolved and validated. | | Determine the physical partitioning of the data based on the table schema and partition columns either from the existing table or from the query plan (for new tables) | Physical plan generation, where the number of writer tasks, data schema and partitioning is determined | | Distribute the writer tasks definitions (which include the transaction state) to workers. | Physical plan execution, only if it is a distributed engine. | | Tasks write the data to data files and send the data file info to the driver. | Physical plan execution, when the data is actually written to the table location | | Finalize the query. Here, all the info of the data files written by the tasks is aggregated and committed to the transaction created at the beginning of the physical execution. | Finalize the query. This happens on the driver where the query has started. | Let’s understand the details of each step. #### Step 4.1: Determine the schema of the data that needs to be written to the table [Section titled “Step 4.1: Determine the schema of the data that needs to be written to the table”](#step-41-determine-the-schema-of-the-data-that-needs-to-be-written-to-the-table) The first step is to resolve the output data schema. This is often required by the connector/ engine to resolve and validate the logical plan of the query (if the concept of logical plan exists in your engine). To achieve this, the connector has to do the following. At a high level query plan is a tree of operators where the leaf-level operators generate or read data from storage/tables and feed it upwards towards the parent operator nodes. This data transfer happens until it reaches the root operator node where the query is finalized (either the results are sent to the client or data is written to another table). * Create the `Table` object * From the `Table` object try to get the schema. * If the table is not found * the query includes creating the table (e.g., `CREATE TABLE AS` SQL query); * the schema is derived from the operator above the `write` that feeds the data to the `write` operator. * the query doesn’t include creating new table, an exception is thrown saying the table is not found * If the table already exists * get the schema from the table and check if it matches the schema of the `write` operator. If not throw an exception. * Create a `TransactionBuilder` - this basically begins the steps of transaction construction. ```java import io.delta.kernel.*; import io.delta.kernel.defaults.engine.*; Engine myEngine = new MyEngine(); Table myTable = Table.forPath(myTablePath); StructType writeOperatorSchema = // ... derived from the query operator tree ... StructType dataSchema; boolean isNewTable = false; try { Snapshot mySnapshot = myTable.getLatestSnapshot(myEngine); dataSchema = mySnapshot.getSchema(myEngine); // .. check dataSchema and writeOperatorSchema match ... } catch(TableNotFoundException e) { isNewTable = true; dataSchema = writeOperatorSchema; } TransactionBuilder txnBuilder = myTable.createTransactionBuilder( myEngine, "Examples", /* engineInfo - connector can add its own identifier which is noted in the Delta Log */ Operation /* What is the operation we are trying to perform? This is noted in the Delta Log */ ); if (isNewTable) { // For a new table set the table schema in the transaction builder txnBuilder = txnBuilder.withSchema(engine, dataSchema) } ``` #### Step 4.2: Determine the physical partitioning of the data based on the table schema and partition columns [Section titled “Step 4.2: Determine the physical partitioning of the data based on the table schema and partition columns”](#step-42-determine-the-physical-partitioning-of-the-data-based-on-the-table-schema-and-partition-columns) Partition columns are found either from the query (for new tables, the query defines the partition columns) or from the existing table. ```java TransactionBuilder txnBuilder = ... from the last step ... Transaction txn; List partitionColumns = ... if (newTable) { partitionColumns = ... derive from the query parameters (ex. PARTITION BY clause in SQL) ... txnBuilder = txnBuilder.withPartitionColumns(engine, partitionColumns); txn = txnBuilder.build(engine); } else { txn = txnBuilder.build(engine); partitionColumns = txn.getPartitionColumns(engine); } ``` At the end of this step, we have the `Transaction` and schema of the data to generate and its partitioning. #### Step 4.3: Distribute the writer tasks definitions (which include the transaction state) to workers [Section titled “Step 4.3: Distribute the writer tasks definitions (which include the transaction state) to workers”](#step-43-distribute-the-writer-tasks-definitions-which-include-the-transaction-state-to-workers) If you are building a connector for a distributed engine like Spark/Presto/Trino/Flink, then your connector has to send all the writer metadata from the query planning machine (henceforth called the driver) to task execution machines (henceforth called the workers). You will have to serialize and deserialize the transaction state. It is the connector job to implement serialization and deserialization utilities for a [`Row`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/Row.html). More details on a custom `Row` SerDe are found [here](#custom-row-serializerdeserializer). ```java Row txnState = txn.getState(engine); String jsonTxnState = serializeToJson(txnState); ``` #### Step 4.4: Tasks write the data to data files and send the data file info to the driver. [Section titled “Step 4.4: Tasks write the data to data files and send the data file info to the driver.”](#step-44-tasks-write-the-data-to-data-files-and-send-the-data-file-info-to-the-driver) In this step (which is executed on the worker nodes inside each task): * Deserialize the transaction state * Writer operator within the task gets the data from its parent operator. * The data is converted into a `FilteredColumnarBatch`. Each [`FilteredColumnarBatch`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/data/FilteredColumnarBatch.html) has two components: * Columnar batch (returned by `FilteredColumnarBatch.getData()`): This is the data read from the files having the schema matching the readSchema provided when the Scan object was built in the earlier step. * Optional selection vector (returned by `FilteredColumnarBatch.getSelectionVector()`): Optionally, a boolean vector that will define which rows in the batch are valid and should be consumed by the engine. * The connector can create `FilteredColumnBatch` wrapper around data in its own in-memory format. * Check if the data is partitioned or not. If not partitioned, partition the data by partition values. * For each partition generate the map of the partition column to the partition value * Use Kernel to convert the partitioned data into physical data that should go into the data files * Write the physical data into one or more data files. * Convert data file statues into a Delta log actions * Serialize the Delta log action `Row` objects and send them to the driver node ```plaintext Row txnState = ... deserialize from JSON string sent by the driver ... CloseableIterator data = ... generate data ... // If the table is un-partitioned then this is an empty map Map partitionValues = ... prepare the partition values ... // First transform the logical data to physical data that needs to be written // to the Parquet files CloseableIterator physicalData = Transaction.transformLogicalData(engine, txnState, data, partitionValues); // Get the write context DataWriteContext writeContext = Transaction.getWriteContext(engine, txnState, partitionValues); // Now write the physical data to Parquet files CloseableIterator dataFiles = engine.getParquetHandler() .writeParquetFiles( writeContext.getTargetDirectory(), physicalData, writeContext.getStatisticsColumns()); // Now convert the data file status to data actions that needs to be written to the Delta table log CloseableIterator partitionDataActions = Transaction.generateAppendActions( engine, txnState, dataFiles, writeContext); .... serialize `partitionDataActions` and send them to driver node ``` #### Step 4.5: Finalize the query. [Section titled “Step 4.5: Finalize the query.”](#step-45-finalize-the-query) At the driver node, the delta log actions from all the tasks are received and committed to the transaction. The tasks send the Delta log actions as a serialized JSON and deserialize them back to `Row` objects. ```plaintext // Create a iterable out of the data actions. If the contents are too big to fit in memory, // the connector may choose to write the data actions to a temporary file and return an // iterator that reads from the file. CloseableIterable dataActionsIterable = CloseableIterable.inMemoryIterable( toCloseableIterator(dataActions.iterator())); // Commit the transaction. TransactionCommitResult commitResult = txn.commit(engine, dataActionsIterable); // Optional step if (commitResult.isReadyForCheckpoint()) { // Checkpoint the table Table.forPath(engine, tablePath).checkpoint(engine, commitResult.getVersion()); } ``` Thats it. Now you should be able to append data to Delta tables using the Kernel APIs. ## Migration guide [Section titled “Migration guide”](#migration-guide) Kernel APIs are still evolving and new features are being added. Kernel authors try to make the API changes backward compatible as much as they can with each new release, but sometimes it is hard to maintain the backward compatibility for a project that is evolving rapidly. This section provides guidance on how to migrate your connector to the latest version of Delta Kernel. With each new release the [examples](https://github.com/delta-io/delta/tree/master/kernel/examples) are kept up-to-date with the latest API changes. You can refer to the examples to understand how to use the new APIs. ### Migration from Delta Lake version 3.1.0 to 3.2.0 [Section titled “Migration from Delta Lake version 3.1.0 to 3.2.0”](#migration-from-delta-lake-version-310-to-320) Following are API changes in Delta Kernel 3.2.0 that may require changes in your connector. #### Rename `TableClient` to [`Engine`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/engine/Engine.html) [Section titled “Rename TableClient to Engine”](#rename-tableclient-to-engine) The `TableClient` interface has been renamed to `Engine`. This is the most significant API change in this release. The `TableClient` interface name is not exactly representing the functionality it provides. At a high level it provides capabilities such as reading Parquet files, JSON files, evaluating expressions on data and file system functionality. These are basically the heavy lift operations that Kernel depends on as a separate interface to allow the connectors to substitute their own custom implementation of the same functionality (e.g. custom Parquet reader). Essentially, these functionalities are the core of the `engine` functionalities. By renaming to `Engine`, we are representing the interface functionality with a proper name that is easy to understand. The `DefaultTableClient` has been renamed to [`DefaultEngine`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultEngine.html). #### [`Table.forPath(Engine engine, String tablePath)`](https://delta-io.github.io/delta/snapshot/kernel-api/java/io/delta/kernel/Table.html#forPath-io.delta.kernel.engine.Engine-java.lang.String-) behavior change [Section titled “Table.forPath(Engine engine, String tablePath) behavior change”](#tableforpathengine-engine-string-tablepath-behavior-change) Earlier when a non-existent table path is passed, the API used to throw `TableNotFoundException`. Now it doesn’t throw the exception. Instead, it returns a `Table` object. When trying to get a `Snapshot` from the table object it throws the `TableNotFoundException`. #### [`FileSystemClient.resolvePath`](https://delta-io.github.io/delta/snapshot/kernel-defaults/java/io/delta/kernel/defaults/engine/DefaultFileSystemClient.html#resolvePath-java.lang.String-) behavior change [Section titled “FileSystemClient.resolvePath behavior change”](#filesystemclientresolvepath-behavior-change) Earlier when a non-existent path is passed, the API used to throw `FileNotFoundException`. Now it doesn’t throw the exception. It still resolves the given path into a fully qualified path. # Delta Kernel Rust > Learn how to build connectors to read and write Delta tables using Delta Kernel Rust. Work In Progress # Other connectors ## Apache Druid [Section titled “Apache Druid”](#apache-druid) This [connector](https://druid.apache.org/docs/latest/development/extensions-contrib/delta-lake/) allows [Apache Druid](https://druid.apache.org/) to read from Delta Lake. ## Apache Pulsar [Section titled “Apache Pulsar”](#apache-pulsar) This [connector](https://github.com/streamnative/pulsar-io-lakehouse/blob/master/docs/delta-lake-demo.md) allows [Apache Pulsar](https://pulsar.apache.org/) to read from and write to Delta Lake. ## ClickHouse [Section titled “ClickHouse”](#clickhouse) [ClickHouse](https://clickhouse.com/) is a column-oriented database that allows users to run SQL queries on Delta Lake tables. This [connector](https://clickhouse.com/docs/engines/table-engines/integrations/deltalake) provides a read-only integration with existing Delta Lake tables in Amazon S3. ## Dagster [Section titled “Dagster”](#dagster) Use the [Delta Lake IO Manager](https://delta-io.github.io/delta-rs/integrations/delta-lake-dagster/) to read from and write to Delta Lake tables in your [Dagster](https://dagster.io/) orchestration pipelines. ## FINOS Legend [Section titled “FINOS Legend”](#finos-legend) An [extension](https://github.com/finos/legend-community-delta/blob/main/README.md) to the [FINOS](https://landscape.finos.org/) Legend framework for Apache Spark™ / Delta Lake based environment, combining best of open data standards with open source technologies. This connector allows Trino to read from and write to Delta Lake. ## Hopsworks [Section titled “Hopsworks”](#hopsworks) This [connectors](https://docs.hopsworks.ai/latest/user_guides/fs/feature_group/create/#batch-write-api) allows [Hopsworks Feature Store](https://www.hopsworks.ai/dictionary/feature-store) store, manage, and serve feature data in Delta Lake. ## Apache Hive [Section titled “Apache Hive”](#apache-hive) This integration enables reading Delta tables from Apache Hive. For details on installing the integration, see the [Delta Lake repository](https://github.com/delta-io/delta/tree/master/connectors/hive). ## Kafka Delta Ingest [Section titled “Kafka Delta Ingest”](#kafka-delta-ingest) This [project](https://github.com/delta-io/kafka-delta-ingest) builds a highly efficient daemon for streaming data through Apache Kafka into Delta Lake. ## SQL Delta Import [Section titled “SQL Delta Import”](#sql-delta-import) This [utility](https://github.com/delta-io/delta/blob/master/connectors/sql-delta-import/readme.md) is for importing data from a JDBC source into a Delta Lake table. ## StarRocks [Section titled “StarRocks”](#starrocks) [StarRocks](https://www.starrocks.io/), a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. StarRocks has the [ability to read](https://docs.starrocks.io/docs/introduction/StarRocks_intro/) from Delta Lake. # Presto connector > Learn how to set up an integration to enable you to read Delta tables from Presto. Since Presto [version 0.269](https://prestodb.io/docs/0.269/release/release-0.269.html#delta-lake-connector-changes), Presto natively supports reading Delta Lake tables. For details on using the native Delta Lake connector, see [Delta Lake Connector - Presto](https://prestodb.io/docs/current/connector/deltalake.html). For Presto versions lower than [0.269](https://prestodb.io/docs/0.269/release/release-0.269.html#delta-lake-connector-changes), you can use the manifest-based approach detailed in [Presto, Trino, and Athena to Delta Lake integration using manifests](/presto-integration/). # Delta Lake resources > Learn about resources for understanding Delta Lake. ## Blog posts and talks [Section titled “Blog posts and talks”](#blog-posts-and-talks) [Delta Lake blog posts](https://delta.io/blog) [Delta Lake tutorials](https://delta.io/learn/tutorials/) [Delta Lake videos](https://delta.io/learn/videos/) ## VLDB 2020 paper [Section titled “VLDB 2020 paper”](#vldb-2020-paper) [Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores](https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf) ## Examples [Section titled “Examples”](#examples) The Delta Lake GitHub repository has [Scala and Python examples](https://github.com/delta-io/delta/tree/master/examples/). ## Delta Lake transaction log specification [Section titled “Delta Lake transaction log specification”](#delta-lake-transaction-log-specification) The Delta Lake transaction log has a well-defined open protocol that can be used by any system to read the log. See [Delta Transaction Log Protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md). # Use row tracking for Delta tables > Learn how Delta Lake row tracking allows tracking how rows change across table versions. Row tracking allows Delta Lake to track row-level lineage in a Delta Lake table. When enabled on a Delta Lake table, row tracking adds two new metadata fields to the table: * **Row IDs** provide rows with an identifier that is unique within the table. A row keeps the same ID whenever it is modified using a `MERGE` or `UPDATE` statement. * **Row commit versions** record the last version of the table in which the row was modified. A row is assigned a new version whenever it is modified using a `MERGE` or `UPDATE` statement. Note This feature is available in Delta Lake 3.2.0 and above. Enabling this feature on existing non-empty tables is available in Delta Lake 3.3.0 and above. ## Enable row tracking [Section titled “Enable row tracking”](#enable-row-tracking) Caution Tables created with row tracking enabled have the row tracking Delta Lake table feature enabled at creation and use Delta Lake writer version 7. Table protocol versions cannot be downgraded, and tables with row tracking enabled are not writeable by Delta Lake clients that do not support all enabled Delta Lake writer protocol table features. See [How does Delta Lake manage feature compatibility?](/versioning/). You must explicitly enable row tracking using one of the following methods: * **New table**: Set the table property `delta.enableRowTracking = true` in the `CREATE TABLE` command. - SQL ```sql -- Create an empty table CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIES ('delta.enableRowTracking' = 'true'); -- Using a CTAS statement CREATE TABLE course_new TBLPROPERTIES ('delta.enableRowTracking' = 'true') AS SELECT * FROM course_old; -- Using a LIKE statement to copy configuration CREATE TABLE graduate LIKE student; -- Using a CLONE statement to copy configuration CREATE TABLE graduate CLONE student; ``` * **Existing table**: Available from Delta 3.3 and above, set the table property `'delta.enableRowTracking' = 'true'` in the `ALTER TABLE` command. - SQL ```sql ALTER TABLE grade SET TBLPROPERTIES ('delta.enableRowTracking' = 'true'); ``` * **All new tables**: Set the configuration `spark.databricks.delta.properties.defaults.enableRowTracking = true` for the current session in the `SET` command. - SQL ```sql SET spark.databricks.delta.properties.defaults.enableRowTracking = true; ``` - Python ```python spark.conf.set("spark.databricks.delta.properties.defaults.enableRowTracking", True) ``` - Scala ```scala spark.conf.set("spark.databricks.delta.properties.defaults.enableRowTracking", true) ``` Caution Because cloning a Delta Lake table creates a separate history, the row ids and row commit versions on cloned tables do not match that of the original table. Caution Enabling row tracking on existing table will automatically assign row ids and row commit versions to all existing rows in the table. This process may cause multiple new versions of the table to be created and may take a long time. ### Row tracking storage [Section titled “Row tracking storage”](#row-tracking-storage) Enabling row tracking may increase the size of the table. Delta Lake stores row tracking metadata fields in hidden metadata columns in the data files. Some operations, such as insert-only operations do not use these hidden columns and instead track the row ids and row commit versions using metadata in the Delta Lake log. Data reorganization operations such as `OPTIMIZE` and `REORG` cause the row ids and row commit versions to be tracked using the hidden metadata column, even when they were stored using metadata. ## Read row tracking metadata fields [Section titled “Read row tracking metadata fields”](#read-row-tracking-metadata-fields) Row tracking adds the following metadata fields that can be accessed when reading a table: | Column name | Type | Values | | ------------------------------ | ---- | ---------------------------------------------------------------- | | `_metadata.row_id` | Long | The unique identifier of the row. | | `_metadata.row_commit_version` | Long | The table version at which the row was last inserted or updated. | The row ids and row commit versions metadata fields are not automatically included when reading the table. Instead, these metadata fields must be manually selected from the hidden `_metadata` column which is available for all tables in Apache Spark. * SQL ```sql SELECT _metadata.row_id, _metadata.row_commit_version, * FROM table_name; ``` * Python ```python spark.read.table("table_name") \ .select("_metadata.row_id", "_metadata.row_commit_version", "*") ``` * Scala ```scala spark.read.table("table_name") .select("_metadata.row_id", "_metadata.row_commit_version", "*") ``` ## Disable row tracking [Section titled “Disable row tracking”](#disable-row-tracking) Row tracking can be disabled to reduce the storage overhead of the metadata fields. After disabling row tracking the metadata fields remain available, but all rows always get assigned a new id and commit version whenever they are touched by an operation. * SQL ```sql ALTER TABLE table_name SET TBLPROPERTIES (delta.enableRowTracking = false); ``` * Python ```python spark.sql("ALTER TABLE table_name SET TBLPROPERTIES (delta.enableRowTracking = false)") ``` * Scala ```scala spark.sql("ALTER TABLE table_name SET TBLPROPERTIES (delta.enableRowTracking = false)") ``` Caution Disabling row tracking does not remove the corresponding table feature and does not downgrade the table protocol version. ## Limitations [Section titled “Limitations”](#limitations) The following limitations exist: * The row ids and row commit versions metadata fields cannot be accessed while reading the [Change data feed](/delta-change-data-feed/). * Once the Row Tracking feature is added to the table it cannot be removed without recreating the table. # Read Delta Sharing Tables > Learn how to perform reads on Delta Sharing tables. [Delta Sharing](https://delta.io/sharing/) is an open protocol for secure real-time exchange of large datasets, which enables organizations to share data in real time regardless of which computing platforms they use. It is a simple REST protocol that securely grants access to part of a cloud dataset and leverages modern cloud storage systems, such as S3, ADLS, GCS or R2, to reliably transfer data. In Delta Sharing, data provider is the one who owns the original dataset or table, and shares it with a broad range of recipients. Each table can be configured to be shared with different options (history, filtering, etc.) We will focus on consuming the shared table in this doc. Delta Sharing data source supports most of the options provided by Apache Spark DataFrame for performing reads through [batch](/delta-batch), [streaming](/delta-streaming), or [table changes (CDF)](/delta-change-data-feed) APIs on shared tables. Delta Sharing doesn’t support writing to a shared table. Please refer to the [Delta Sharing Repo](https://github.com/delta-io/delta-sharing/blob/main/README.md) for more details. Please follow the [quick start](https://github.com/delta-io/delta-sharing?tab=readme-ov-file#quick-start) to leverage the Delta Sharing python connector to discover the shared tables. For Delta Sharing reads on shared tables with advanced Delta Lake features such as [Deletion Vectors](/delta-deletion-vectors) and [Column Mapping](/delta-column-mapping), you need to enable integration with Apache Spark DataSourceV2 and Catalog APIs (since delta-sharing-spark 3.1) by setting the same configurations as Delta Lake when you create a new `SparkSession`. See [Configure SparkSession](/delta-batch/#configure-sparksession). ## Read a snapshot [Section titled “Read a snapshot”](#read-a-snapshot) After you save the [Profile File](https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#profile-file-format) locally and launch Spark with the connector library, you can access shared tables. A profile file is provided by the data provider to the data recipient. * SQL ```sql -- A table path is the profile file path followed by `#` and the fully qualified name -- of a table (`..`). CREATE TABLE mytable USING deltaSharing LOCATION '#..'; SELECT * FROM mytable; ``` * Python ```python # A table path is the profile file path followed by `#` and the fully qualified name # of a table (`..`). table_path = "#.." df = spark.read.format("deltaSharing").load(table_path) ``` * Scala ```scala // A table path is the profile file path followed by `#` and the fully qualified name // of a table (`..`). val tablePath = "#.." val df = spark.read.format("deltaSharing").load(tablePath) ``` * Java ```java // A table path is the profile file path followed by `#` and the fully qualified name // of a table (`..`). String tablePath = "#.."; Dataset df = spark.read.format("deltaSharing").load(tablePath); ``` The DataFrame returned automatically reads the most recent snapshot of the table for any query. Delta Sharing supports [predicate pushdown](https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#json-predicates-for-filtering) to efficiently fetch data from the Delta Sharing server when there are applicable predicates in the query. ## Query an older snapshot of a shared table (time travel) [Section titled “Query an older snapshot of a shared table (time travel)”](#query-an-older-snapshot-of-a-shared-table-time-travel) Once the data provider enables history sharing of the shared table, Delta Sharing time travel allows you to query an older snapshot of a shared table. * SQL ```sql SELECT * FROM mytable TIMESTAMP AS OF timestamp_expression SELECT * FROM mytable VERSION AS OF version ``` * Python ```python spark.read.format("deltaSharing").option("timestampAsOf", timestamp_string).load(tablePath) spark.read.format("deltaSharing").option("versionAsOf", version).load(tablePath) ``` * Scala ```scala spark.read.format("deltaSharing").option("timestampAsOf", timestamp_string).load(tablePath) spark.read.format("deltaSharing").option("versionAsOf", version).load(tablePath) ``` The `timestamp_expression` and `version` share the same syntax as [Delta](/delta-batch#timestamp-and-version-syntax). ## Read Table Changes (CDF) [Section titled “Read Table Changes (CDF)”](#read-table-changes-cdf) Once the data provider turns on CDF on the original Delta Lake table and shares it with history through Delta Sharing, the recipient can query CDF of a Delta Sharing table similar to [CDF of a Delta table](/delta-change-data-feed). * SQL ```sql CREATE TABLE mytable USING deltaSharing LOCATION '#..'; -- version as ints or longs e.g. changes from version 0 to 10 SELECT * FROM table_changes('mytable', 0, 10) -- timestamp as string formatted timestamps SELECT * FROM table_changes('mytable', '2021-04-21 05:45:46', '2021-05-21 12:00:00') -- providing only the startingVersion/timestamp SELECT * FROM table_changes('mytable', 0) ``` * Python ```python table_path = "#.." # version as ints or longs spark.read.format("deltaSharing") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .option("endingVersion", 10) \ .load(tablePath) # timestamps as formatted timestamp spark.read.format("deltaSharing") \ .option("readChangeFeed", "true") \ .option("startingTimestamp", '2021-04-21 05:45:46') \ .option("endingTimestamp", '2021-05-21 12:00:00') \ .load(tablePath) # providing only the startingVersion/timestamp spark.read.format("deltaSharing") \ .option("readChangeFeed", "true") \ .option("startingVersion", 0) \ .load(tablePath) ``` * Scala ```scala val tablePath = "#.." // version as ints or longs spark.read.format("deltaSharing") .option("readChangeFeed", "true") .option("startingVersion", 0) .option("endingVersion", 10) .load(tablePath) // timestamps as formatted timestamp spark.read.format("deltaSharing") .option("readChangeFeed", "true") .option("startingTimestamp", "2024-01-18 05:45:46") .option("endingTimestamp", "2024-01-18 12:00:00") .load(tablePath) // providing only the startingVersion/timestamp spark.read.format("deltaSharing") .option("readChangeFeed", "true") .option("startingVersion", 0) .load(tablePath) ``` ## Streaming [Section titled “Streaming”](#streaming) Delta Sharing Streaming is deeply integrated with [Spark Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) through `readStream`, and able to connect with any sink that is able to perform `writeStream`. Once the data provider shares a table with history, the recipient can perform a streaming query on the table. When you load a Delta Sharing table as a stream source and use it in a streaming query, the query processes all of the data present in the shared table as well as any new data that arrives after the stream has started. * Scala ```scala val tablePath = "#.." spark.readStream.format("deltaSharing").load(tablePath) ``` Delta Sharing Streaming supports the following functionalities in the same way as Delta Streaming: [Limit input rate](/delta-streaming/#limit-input-rate), [Ignore updates and deletes](/delta-streaming/#ignore-updates-and-deletes), [Specify initial position](/delta-streaming/#specify-initial-position) In addition, `maxVersionsPerRpc` is provided to decide how many versions of files are requested from the server in every Delta Sharing rpc. This is to help reduce the per rpc workload and make the Delta sharing streaming job more stable, especially when many new versions have accumulated when the streaming resumes from a checkpoint. The default is 100. Note Trigger.AvailableNow is not supported in Delta Sharing Streaming. You can use Trigger.Once as a workaround, and at a proper frequency to catch up with the changes in the server. ## Read Advanced Delta Lake Features in Delta Sharing [Section titled “Read Advanced Delta Lake Features in Delta Sharing”](#read-advanced-delta-lake-features-in-delta-sharing) In order to support advanced Delta Lake features in Delta Sharing, “Delta Format Sharing” was introduced since delta-sharing-client 1.0 and delta-sharing-spark 3.1, in which the actions of a shared table are returned in Delta Lake format, allowing a Delta Lake library to read it. Please remember to set the spark configurations mentioned in [Configure SparkSession](/delta-batch/#configure-sparksession) in order to read shared tables with Deletion Vectors and Column Mapping. | Read Table Feature | Available since version | | -------------------------------------------------------------------------------------------------------------- | ----------------------- | | [Deletion Vectors](/delta-deletion-vectors) | 3.1.0 | | [Column Mapping](/delta-column-mapping) | 3.1.0 | | [Timestamp without Timezone](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) | 3.3.0 | | [Type widening (Preview)](/delta-type-widening) | 3.3.0 | | [Variant Type (Preview)](https://github.com/delta-io/delta/blob/master/protocol_rfcs/accepted/variant-type.md) | 3.3.0 | Batch queries can be performed as is, because it can automatically resolve the `responseFormat` based on the table features of the shared table. An additional option `responseFormat=delta` needs to be set for cdf and streaming queries when reading shared tables with Deletion Vectors or Column Mapping enabled. * Scala ```scala import org.apache.spark.sql.SparkSession val spark = SparkSession .builder() .appName("...") .master("...") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate() val tablePath = "#.." // Batch query spark.read.format("deltaSharing").load(tablePath) // CDF query spark.read.format("deltaSharing") .option("readChangeFeed", "true") .option("responseFormat", "delta") .option("startingVersion", 1) .load(tablePath) // Streaming query spark.readStream.format("deltaSharing").option("responseFormat", "delta").load(tablePath) ``` # Delta Connect (aka Spark Connect Support in Delta) > Learn about Delta Connect - Spark Connect Support in Delta. Note This feature is available in Delta Lake 4.0.0 and above. Please note, Delta Connect is currently in preview and not recommended for production workloads. Delta Connect adds [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html) support to Delta Lake for Apache Spark. Spark Connect is a new initiative that adds a decoupled client-server infrastructure which allows remote connectivity from Spark from everywhere. Delta Connect allows all Delta Lake operations to work in your application running as a client connected to the Spark server. ## Motivation [Section titled “Motivation”](#motivation) Delta Connect is expected to br0ng the same benefits as Spark Connect: 1. Upgrading to more recent versions of Spark and Delta Lake is now easier because the client interface is being completely decoupled from the server. 2. Simpler integration of Spark and Delta Lake with developer tooling. IDEs no longer have to integrate with the full Spark and Delta Lake implementation, and instead can integrate with a thin-client. 3. Support for languages other than Java/Scala and Python. Clients “merely” have to generate Protocol Buffers and therefore become simpler to implement. 4. Spark and Delta Lake will become more stable, as user code is no longer running in the same JVM as Spark’s driver. 5. Remote connectivity. Code can run anywhere now, as there is a gRPC layer between the user interface and the driver. ## How to start the Spark Server with Delta [Section titled “How to start the Spark Server with Delta”](#how-to-start-the-spark-server-with-delta) 1. Download `spark-4.0.0-bin-hadoop3.tgz` from [Spark 4.0.0](https://archive.apache.org/dist/spark/spark-4.0.0). 2. Start the Spark Connect server with the Delta Lake Connect plugins: ```bash sbin/start-connect-server.sh \ --packages io.delta:delta-connect-server_2.13:4.0.0,com.google.protobuf:protobuf-java:3.25.1 \ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \ --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \ --conf "spark.connect.extensions.relation.classes=org.apache.spark.sql.connect.delta.DeltaRelationPlugin" \ --conf "spark.connect.extensions.command.classes=org.apache.spark.sql.connect.delta.DeltaCommandPlugin" ``` ## How to use the Python Spark Connect Client with Delta [Section titled “How to use the Python Spark Connect Client with Delta”](#how-to-use-the-python-spark-connect-client-with-delta) The Delta Lake Connect Python client is included in the same PyPi package as Delta Lake Spark. 1. `pip install pyspark==4.0.0`. 2. `pip install delta-spark==4.0.0`. 3. The usage is the same as Spark Connect (e.g. `./bin/pyspark --remote "sc://localhost"`). We just need to pass in a remote `SparkSession` (instead of a local one) to the `DeltaTable` API. An example: * Python ```python from delta.tables import DeltaTable from pyspark.sql import SparkSession from pyspark.sql.functions import * deltaTable = DeltaTable.forName(spark, "my_table") deltaTable.toDF().show() deltaTable.update( condition = "id % 2 == 0", set = {"id": "id + 100"} ) ``` ## How to use the Scala Spark Connect Client with Delta [Section titled “How to use the Scala Spark Connect Client with Delta”](#how-to-use-the-scala-spark-connect-client-with-delta) Make sure you are using Java 17! ```bash ./bin/spark-shell --remote "sc://localhost" --packages io.delta:delta-connect-client_2.13:4.0.0,com.google.protobuf:protobuf-java:3.25.1 ``` An example: * Scala ```scala import io.delta.tables.DeltaTable val deltaTable = DeltaTable.forName(spark, "my_table") deltaTable.toDF.show() deltaTable.updateExpr( condition = "id % 2 == 0", set = Map("id" -> "id + 100") ) ``` # Delta Standalone (deprecated) > Learn how to read and write Delta tables from JVM applications without Apache Spark. Warning! Delta Standalone is deprecated and will be removed in a future release. We recommend using the [Delta Kernel](/delta-kernel) APIs. The Delta Standalone library is a single-node Java library that can be used to read from and write to Delta tables. Specifically, this library provides APIs to interact with a table’s metadata in the transaction log, implementing the [Delta Transaction Log Protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) to achieve the transactional guarantees of the Delta Lake format. Notably, this project doesn’t depend on Apache Spark and has only a few transitive dependencies. Therefore, it can be used by any processing engine or application to access Delta tables. ## Use cases [Section titled “Use cases”](#use-cases) Delta Standalone is optimized for cases when you want to read and write Delta tables by using a non-Spark engine of your choice. It is a “low-level” library, and we encourage developers to contribute open-source, higher-level connectors for their desired engines that use Delta Standalone for all Delta Lake metadata interaction. You can find a Hive source connector and Flink sink/source connector in the [Delta Lake](https://github.com/delta-io/delta) repository. Additional connectors are in development. ### Caveats [Section titled “Caveats”](#caveats) Delta Standalone minimizes memory usage in the JVM by loading the Delta Lake transaction log incrementally, using an iterator. However, Delta Standalone runs in a single JVM, and is limited to the processing and memory capabilities of that JVM. Users must configure the JVM to avoid out of memory (OOM) issues. Delta Standalone does provide basic APIs for reading Parquet data, but does not include APIs for writing Parquet data. Users must write out new Parquet data files themselves and then use Delta Standalone to commit those changes to the Delta table and make the new data visible to readers. ## APIs [Section titled “APIs”](#apis) Delta Standalone provides classes and entities to read data, query metadata, and commit to the transaction log. A few of them are highlighted here and with their key interfaces. See the [Java API docs](/api/latest/java/standalone/index.html) for the full set of classes and entities. ### DeltaLog [Section titled “DeltaLog”](#deltalog) [DeltaLog](/api/latest/java/io/delta/standalone/DeltaLog.html) is the main interface for programmatically interacting with the metadata in the transaction log of a Delta table. * Instantiate a `DeltaLog` with `DeltaLog.forTable(hadoopConf, path)` and pass in the `path` of the root location of the Delta table. * Access the current snapshot with `DeltaLog::snapshot`. * Get the latest snapshot, including any new data files that were added to the log, with `DeltaLog::update`. * Get the snapshot at some historical state of the log with `DeltaLog::getSnapshotForTimestampAsOf` or `DeltaLog::getSnapshotForVersionAsOf`. * Start a new transaction to commit to the transaction log by using `DeltaLog::startTransaction`. * Get all metadata actions without computing a full Snapshot using `DeltaLog::getChanges`. ### Snapshot [Section titled “Snapshot”](#snapshot) A [Snapshot](/api/latest/java/io/delta/standalone/Snapshot.html) represents the state of the table at a specific version. * Get a list of the metadata files by using `Snapshot::getAllFiles`. * For a memory-optimized iterator over the metadata files, use `Snapshot::scan` to get a `DeltaScan` (as described later), optionally by passing in a `predicate` for partition filtering. * Read actual data with `Snapshot::open`, which returns an iterator over the rows of the Delta table. ### OptimisticTransaction [Section titled “OptimisticTransaction”](#optimistictransaction) The main class for committing a set of updates to the transaction log is [OptimisticTransaction](/api/latest/java/io/delta/standalone/OptimisticTransaction.html). During a transaction, all reads must go through the `OptimisticTransaction` instance rather than the `DeltaLog` in order to detect logical conflicts and concurrent updates. * Read metadata files during a transaction with `OptimisticTransaction::markFilesAsRead`, which returns a `DeltaScan` of files that match the `readPredicate`. * Commit to the transaction log with `OptimisticTransaction::commit`. * Get the latest version committed for a given application ID (for example, for idempotency) with `OptimisticTransaction::txnVersion`. (Note that this API requires users to commit `SetTransaction` actions.) * Update the medadata of the table upon committing with `OptimisticTransaction::updateMetadata`. ### DeltaScan [Section titled “DeltaScan”](#deltascan) [DeltaScan](/api/latest/java/io/delta/standalone/DeltaScan.html) is a wrapper class for the files inside a `Snapshot` that match a given `readPredicate`. * Access the files that match the partition filter portion of the `readPredicate` with `DeltaScan::getFiles`. This returns a memory-optimized iterator over the metadata files in the table. * To further filter the returned files on non-partition columns, get the portion of input predicate not applied with `DeltaScan::getResidualPredicate`. ## API compatibility [Section titled “API compatibility”](#api-compatibility) The only public APIs currently provided by Delta Standalone are in the `io.delta.standalone` package. Classes and methods in the `io.delta.standalone.internal` package are considered internal and are subject to change across minor and patch releases. ## Project setup [Section titled “Project setup”](#project-setup) You can add the Delta Standalone library as a dependency by using your preferred build tool. Delta Standalone depends upon the `hadoop-client` and `parquet-hadoop` packages. Example build files are listed in the following sections. ### Environment requirements [Section titled “Environment requirements”](#environment-requirements) * JDK 8 or above. * Scala 2.11 or 2.12. ### Build files [Section titled “Build files”](#build-files) #### Maven [Section titled “Maven”](#maven) Replace the version of `hadoop-client` with the one you are using. Scala 2.12: ```xml io.delta delta-standalone_2.12 0.5.0 org.apache.hadoop hadoop-client 3.1.0 ``` Scala 2.11: ```xml io.delta delta-standalone_2.11 0.5.0 org.apache.hadoop hadoop-client 3.1.0 ``` #### SBT [Section titled “SBT”](#sbt) Replace the version of `hadoop-client` with the one you are using. ```plaintext libraryDependencies ++= Seq( "io.delta" %% "delta-standalone" % "0.5.0", "org.apache.hadoop" % "hadoop-client" % "3.1.0) ``` #### `ParquetSchemaConverter` caveat [Section titled “ParquetSchemaConverter caveat”](#parquetschemaconverter-caveat) Delta Standalone shades its own Parquet dependencies so that it works out-of-the-box and reduces dependency conflicts in your environment. However, if you would like to use utility class `io.delta.standalone.util.ParquetSchemaConverter`, then you must provide your own version of `org.apache.parquet:parquet-hadoop`. ### Storage configuration [Section titled “Storage configuration”](#storage-configuration) Delta Lake ACID guarantees are based on the atomicity and durability guarantees of the storage system. Not all storage systems provide all the necessary guarantees. Because storage systems do not necessarily provide all of these guarantees out-of-the-box, Delta Lake transactional operations typically go through the [LogStore API](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java) instead of accessing the storage system directly. To provide the ACID guarantees for different storage systems, you may have to use different `LogStore` implementations. This section covers how to configure Delta Standalone for various storage systems. There are two categories of storage systems: * **Storage systems with built-in support**: For some storage systems, you do not need additional configurations. Delta Standalone uses the scheme of the path (that is, `s3a` in `s3a://path`) to dynamically identify the storage system and use the corresponding `LogStore` implementation that provides the transactional guarantees. However, for S3, there are additional caveats on concurrent writes. See the [section on S3](#amazon-s3-configuration) for details. * **Other storage systems**: The `LogStore`, similar to Apache Spark, uses the Hadoop `FileSystem` API to perform reads and writes. Delta Standalone supports concurrent reads on any storage system that provides an implementation of the `FileSystem` API. For concurrent writes with transactional guarantees, there are two cases based on the guarantees provided by the `FileSystem` implementation. If the implementation provides consistent listing and atomic renames-without-overwrite (that is, `rename(... , overwrite = false)` will either generate the target file atomically or fail if it already exists with `java.nio.file.FileAlreadyExistsException`), then the default `LogStore` implementation using renames will allow concurrent writes with guarantees. Otherwise, you must configure a custom implementation of `LogStore` by setting the following Hadoop configuration when you instantiate a `DeltaLog` with `DeltaLog.forTable(hadoopConf, path)`: ```java delta.logStore..impl= ``` Here, `` is the scheme of the paths of your storage system. This configures Delta Standalone to dynamically use the given `LogStore` implementation only for those paths. You can have multiple such configurations for different schemes in your application, thus allowing it to simultaneously read and write from different storage systems. Note Before version 0.5.0, Delta Standalone supported configuring LogStores by setting `io.delta.standalone.LOG_STORE_CLASS_KEY`. This approach is now deprecated. Setting this configuration will use the configured `LogStore` for all paths, thereby disabling the dynamic scheme-based delegation. #### Amazon S3 configuration [Section titled “Amazon S3 configuration”](#amazon-s3-configuration) Delta Standalone supports reads and writes to S3 in two different modes: Single-cluster and Multi-cluster. | | Single-cluster | Multi-cluster | | ------------- | ------------------------------------------------ | ------------------------------------------------ | | Configuration | Comes out-of-the-box | Is experimental and requires extra configuration | | Reads | Supports concurrent reads from multiple clusters | Supports concurrent reads from multiple clusters | | Writes | Supports concurrent writes from a single cluster | Supports multi-cluster writes | | Permissions | S3 credentials | S3 and DynamoDB operating permissions | ##### Single-cluster setup (default) [Section titled “Single-cluster setup (default)”](#single-cluster-setup-default) By default, Delta Standalone supports concurrent reads from multiple clusters. However, concurrent writes to S3 must originate from a single cluster to provide transactional guarantees. This is because S3 currently does not provide mutual exclusion, that is, there is no way to ensure that only one writer is able to create a file. Warning! Concurrent writes to the same Delta table from multiple Spark drivers can lead to data loss. To use Delta Standalone with S3, you must meet the following requirements. If you are using access keys for authentication and authorization, you must configure a Hadoop Configuration specified as follows when you instantiate a `DeltaLog` with `DeltaLog.forTable(hadoopConf, path)`. ###### Requirements (S3 single-cluster) [Section titled “Requirements (S3 single-cluster)”](#requirements-s3-single-cluster) * S3 credentials: [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) (recommended) or access keys. * Hadoop’s [AWS connector (hadoop-aws)](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws) for the version of Hadoop that Delta Standalone is compiled with. ###### Configuration (S3 single-cluster) [Section titled “Configuration (S3 single-cluster)”](#configuration-s3-single-cluster) 1. Include `hadoop-aws` JAR in the classpath. 2. Set up S3 credentials. We recommend that you use [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) for authentication and authorization. But if you want to use keys, configure your `org.apache.hadoop.conf.Configuration` with: ```java conf.set("fs.s3a.access.key", ""); conf.set("fs.s3a.secret.key", ""); ``` ##### Multi-cluster setup [Section titled “Multi-cluster setup”](#multi-cluster-setup) Note This support is new and experimental. This mode supports concurrent writes to S3 from multiple clusters. Enable multi-cluster support by configuring Delta Standalone to use the correct `LogStore` implementation. This implementation uses [DynamoDB](https://aws.amazon.com/dynamodb/) to provide mutual exclusion. Warning! When writing from multiple clusters, all drivers must use this `LogStore` implementation and the same DynamoDB table and region. If some drivers use the default `LogStore` while others use this experimental `LogStore` then data loss can occur. ###### Requirements (S3 multi-cluster) [Section titled “Requirements (S3 multi-cluster)”](#requirements-s3-multi-cluster) * All of the requirements listed in the [Requirements (S3 single-cluster)](#requirements-s3-single-cluster) section * In additon to S3 credentials, you also need DynamoDB operating permissions ###### Configuration (S3 multi-cluster) [Section titled “Configuration (S3 multi-cluster)”](#configuration-s3-multi-cluster) 1. Create the DynamoDB table. See [Create the DynamoDB table](/delta-storage#setup-configuration-s3-multi-cluster) for more details on creating a table yourself (recommended) or having it created for you automatically. 2. Follow the configuration steps listed in [Configuration (S3 single-cluster)](#configuration-s3-single-cluster) section. 3. Include the `delta-storage-s3-dynamodb` JAR in the classpath. 4. Configure the `LogStore` implementation. First, configure this `LogStore` implementation for the scheme `s3`. You can replicate this command for schemes `s3a` and `s3n` as well. ```java conf.set("delta.logStore.s3.impl", "io.delta.storage.S3DynamoDBLogStore"); ``` | Configuration Key | Description | Default | | ------------------------------------------------------------- | ----------------------------------------------- | ---------------------------------- | | io.delta.storage.S3DynamoDBLogStore.ddb.tableName | The name of the DynamoDB table to use | delta\_log | | io.delta.storage.S3DynamoDBLogStore.ddb.region | The region to be used by the client | us-east-1 | | io.delta.storage.S3DynamoDBLogStore.credentials.provider | The AWSCredentialsProvider\* used by the client | DefaultAWSCredentialsProviderChain | | io.delta.storage.S3DynamoDBLogStore.provisionedThroughput.rcu | (Table-creation-only\*\*) Read Capacity Units | 5 | | io.delta.storage.S3DynamoDBLogStore.provisionedThroughput.wcu | (Table-creation-only\*\*) Write Capacity Units | 5 | \*For more details on AWS credential providers, see the [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html). \*\*These configurations are only used when the given DynamoDB table doesn’t already exist and needs to be automatically created. ###### Production Configuration (S3 multi-cluster) [Section titled “Production Configuration (S3 multi-cluster)”](#production-configuration-s3-multi-cluster) By this point, this multi-cluster setup is fully operational. However, there is extra configuration you may do to improve performance and optimize storage when running in production. See the [Delta Lake documentation](/delta-storage#production-configuration-s3-multi-cluster) for more details. #### Microsoft Azure configuration [Section titled “Microsoft Azure configuration”](#microsoft-azure-configuration) Delta Standalone supports concurrent reads and writes from multiple clusters with full transactional guarantees for various Azure storage systems. To use an Azure storage system, you must satisfy the following requirements, and configure a Hadoop Configuration as specified when you instantiate a `DeltaLog` with `DeltaLog.forTable(hadoopConf, path)`. ##### Azure Blob Storage [Section titled “Azure Blob Storage”](#azure-blob-storage) ###### Requirements (Azure Blob storage) [Section titled “Requirements (Azure Blob storage)”](#requirements-azure-blob-storage) * A [shared key](https://docs.microsoft.com/rest/api/storageservices/authorize-with-shared-key) or [shared access signature (SAS)](https://docs.microsoft.com/azure/storage/common/storage-sas-overview). * Hadoop’s [Azure Blob Storage libraries](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure) for a version compatible with the Hadoop version Delta Standalone was compiled with. * 2.9.1+ for Hadoop 2 * 3.0.1+ for Hadoop 3 ###### Configuration (Azure Blob storage) [Section titled “Configuration (Azure Blob storage)”](#configuration-azure-blob-storage) 1. Include `hadoop-azure` JAR in the classpath. 2. Set up credentials. * For an SAS token, configure `org.apache.hadoop.conf.Configuration`: ```java conf.set( "fs.azure.sas...blob.core.windows.net", ""); ``` * To specify an account access key: ```java conf.set( "fs.azure.account.key..blob.core.windows.net", ""); ``` ##### Azure Data Lake Storage Gen1 [Section titled “Azure Data Lake Storage Gen1”](#azure-data-lake-storage-gen1) ###### Requirements (ADLS Gen 1) [Section titled “Requirements (ADLS Gen 1)”](#requirements-adls-gen-1) * A [service principal](https://docs.microsoft.com/azure/active-directory/develop/app-objects-and-service-principals) for OAuth 2.0 access. * Hadoop’s [Azure Data Lake Storage Gen1 libraries](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure-datalake) for a version that is compatible with the Hadoop version that was used to compile Delta Standalone. * 2.9.1+ for Hadoop 2 * 3.0.1+ for Hadoop 3 ###### Configuration (ADLS Gen 1) [Section titled “Configuration (ADLS Gen 1)”](#configuration-adls-gen-1) 1. Include `hadoop-azure-datalake` JAR in the classpath. 2. Set up Azure Data Lake Storage Gen1 credentials. Configure `org.apache.hadoop.conf.Configuration`: ```java conf.set("dfs.adls.oauth2.access.token.provider.type", "ClientCredential"); conf.set("dfs.adls.oauth2.client.id", ""); conf.set("dfs.adls.oauth2.credential", ""); conf.set("dfs.adls.oauth2.refresh.url", "https://login.microsoftonline.com//oauth2/token"); ``` ##### Azure Data Lake Storage Gen2 [Section titled “Azure Data Lake Storage Gen2”](#azure-data-lake-storage-gen2) ###### Requirements (ADLS Gen 2) [Section titled “Requirements (ADLS Gen 2)”](#requirements-adls-gen-2) * Account created in [Azure Data Lake Storage Gen2](https://docs.microsoft.com/azure/storage/blobs/create-data-lake-storage-account). * Service principal [created](https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal) and [assigned the Storage Blob Data Contributor role](https://docs.microsoft.com/azure/storage/blobs/assign-azure-role-data-access) for the storage account. * Make a note of the storage-account-name, directory-id (also known as tenant-id), application-id, and password of the principal. These will be used for configuration. * Hadoop’s [Azure Data Lake Storage Gen2 libraries](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure-datalake) version 3.2+ and Delta Standalone compiled with Hadoop 3.2+. ###### Configuration (ADLS Gen 2) [Section titled “Configuration (ADLS Gen 2)”](#configuration-adls-gen-2) 1. Include `hadoop-azure-datalake` JAR in the classpath. In addition, you may also have to include JARs for Maven artifacts `hadoop-azure` and `wildfly-openssl`. 2. Set up Azure Data Lake Storage Gen2 credentials. Configure your `org.apache.hadoop.conf.Configuration` with: ```java conf.set("fs.azure.account.auth.type..dfs.core.windows.net", "OAuth"); conf.set("fs.azure.account.oauth.provider.type..dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"); conf.set("fs.azure.account.oauth2.client.id..dfs.core.windows.net", ""); conf.set("fs.azure.account.oauth2.client.secret..dfs.core.windows.net",""); conf.set("fs.azure.account.oauth2.client.endpoint..dfs.core.windows.net", "https://login.microsoftonline.com//oauth2/token"); ``` where ``, ``, `` and `` are details of the service principal we set as requirements earlier. #### HDFS [Section titled “HDFS”](#hdfs) Delta Standalone has built-in support for HDFS with full transactional guarantees on concurrent reads and writes from multiple clusters. See [Hadoop documentation](https://hadoop.apache.org/docs/stable/) for configuring credentials. #### Google Cloud Storage [Section titled “Google Cloud Storage”](#google-cloud-storage) ##### Requirements (GCS) [Section titled “Requirements (GCS)”](#requirements-gcs) * JAR of the [GCS Connector (gcs-connector)](https://search.maven.org/search?q=a:gcs-connector) Maven artifact. * Google Cloud Storage account and credentials ##### Configuration (GCS) [Section titled “Configuration (GCS)”](#configuration-gcs) 1. Include the JAR for `gcs-connector` in the classpath. See the [documentation](https://cloud.google.com/dataproc/docs/tutorials/gcs-connector-spark-tutorial) for details on how to configure your project with the GCS connector. ## Usage [Section titled “Usage”](#usage) This example shows how to use Delta Standalone to: * Find parquet files. * Write parquet data. * Commit to the transaction log. * Read from the transaction log. * Read back the Parquet data. Please note that this example uses a fictitious, non-Spark engine `Zappy` to write the actual parquet data, as Delta Standalone does not provide any data-writing APIs. Instead, Delta Standalone Writer lets you commit metadata to the Delta log after you’ve written your data. This is why Delta Standalone works well with so many connectors (e.g. Flink, Presto, Trino, etc.) since they provide the parquet-writing functionality instead. ### 1. SBT configuration [Section titled “1. SBT configuration”](#1-sbt-configuration) The following SBT project configuration is used: ```scala // /build.sbt scalaVersion := "2.12.8" libraryDependencies ++= Seq( "io.delta" %% "delta-standalone" % "0.5.0", "org.apache.hadoop" % "hadoop-client" % "3.1.0") ``` ### 2. Mock situation [Section titled “2. Mock situation”](#2-mock-situation) We have a Delta table `Sales` storing sales data, but have realized all the data written on November 2021 for customer `XYZ` had incorrect `total_cost` values. Thus, we need to update all those records with the correct values. We will use a fictious distributed engine `Zappy` and Delta Standalone to update our Delta table. The sales table schema is given below. ```plaintext Sales |-- year: int // partition column |-- month: int // partition column |-- day: int // partition column |-- customer: string |-- sale_id: string |-- total_cost: float ``` ### 3. Starting a transaction and finding relevant files [Section titled “3. Starting a transaction and finding relevant files”](#3-starting-a-transaction-and-finding-relevant-files) Since we must read existing data in order to perform the desired update operation, we must use `OptimisticTransaction::markFilesAsRead` in order to automatically detect any concurrent modifications made to our read partitions. Since Delta Standalone only supports partition pruning, we must apply the residual predicate to further filter the returned files. ```java import io.delta.standalone.DeltaLog; import io.delta.standalone.DeltaScan; import io.delta.standalone.OptimisticTransaction; import io.delta.standalone.actions.AddFile; import io.delta.standalone.data.CloseableIterator; import io.delta.standalone.expressions.And; import io.delta.standalone.expressions.EqualTo; import io.delta.standalone.expressions.Literal; DeltaLog log = DeltaLog.forTable(new Configuration(), "/data/sales"); OptimisticTransaction txn = log.startTransaction(); DeltaScan scan = txn.markFilesAsRead( new And( new And( new EqualTo(schema.column("year"), Literal.of(2021)), // partition filter new EqualTo(schema.column("month"), Literal.of(11))), // partition filter new EqualTo(schema.column("customer"), Literal.of("XYZ")) // non-partition filter ) ); CloseableIterator iter = scan.getFiles(); Map addFileMap = new HashMap(); // partition filtered files: year=2021, month=11 while (iter.hasNext()) { AddFile addFile = iter.next(); addFileMap.put(addFile.getPath(), addFile); } iter.close(); List filteredFiles = ZappyReader.filterFiles( // fully filtered files: year=2021, month=11, customer=XYZ addFileMap.keySet(), toZappyExpression(scan.getResidualPredicate()) ); ``` ### 4. Writing updated Parquet data [Section titled “4. Writing updated Parquet data”](#4-writing-updated-parquet-data) Since Delta Standalone does not provide any Parquet data write APIs, we use `Zappy` to write the data. ```java ZappyDataFrame correctedSaleIdToTotalCost = ...; ZappyDataFrame invalidSales = ZappyReader.readParquet(filteredFiles); ZappyDataFrame correctedSales = invalidSales.join(correctedSaleIdToTotalCost, "id"); ZappyWriteResult dataWriteResult = ZappyWritter.writeParquet("/data/sales", correctedSales); ``` The written data files from the preceding code will have a hierarchy similar to the following: ```shell $ tree /data/sales . ├── _delta_log │ └── ... │ └── 00000000000000001082.json │ └── 00000000000000001083.json ├── year=2019 │ └── month=1 ... ├── year=2020 │ └── month=1 │ └── day=1 │ └── part-00000-195768ae-bad8-4c53-b0c2-e900e0f3eaee-c000.snappy.parquet // previous │ └── part-00001-53c3c553-f74b-4384-b9b5-7aa45bc2291b-c000.snappy.parquet // new | ... │ └── day=2 │ └── part-00000-b9afbcf5-b90d-4f92-97fd-a2522aa2d4f6-c000.snappy.parquet // previous │ └── part-00001-c0569730-5008-42fa-b6cb-5a152c133fde-c000.snappy.parquet // new | ... ``` ### 5. Committing to our Delta table [Section titled “5. Committing to our Delta table”](#5-committing-to-our-delta-table) Now that we’ve written the correct data, we need to commit to the transaction log to add the new files, and remove the old incorrect files. ```java import io.delta.standalone.Operation; import io.delta.standalone.actions.RemoveFile; import io.delta.standalone.exceptions.DeltaConcurrentModificationException; import io.delta.standalone.types.StructType; List removeOldFiles = filteredFiles.stream() .map(path -> addFileMap.get(path).remove()) .collect(Collectors.toList()); List addNewFiles = dataWriteResult.getNewFiles() .map(file -> new AddFile( file.getPath(), file.getPartitionValues(), file.getSize(), System.currentTimeMillis(), true, // isDataChange null, // stats null // tags ); ).collect(Collectors.toList()); List totalCommitFiles = new ArrayList<>(); totalCommitFiles.addAll(removeOldFiles); totalCommitFiles.addAll(addNewFiles); try { txn.commit(totalCommitFiles, new Operation(Operation.Name.UPDATE), "Zippy/1.0.0"); } catch (DeltaConcurrentModificationException e) { // handle exception here } ``` ### 6. Reading from the Delta table [Section titled “6. Reading from the Delta table”](#6-reading-from-the-delta-table) Delta Standalone provides APIs that read both metadata and data, as follows. #### 6.1. Reading Parquet data (distributed) [Section titled “6.1. Reading Parquet data (distributed)”](#61-reading-parquet-data-distributed) For most use cases, and especially when you deal with large volumes of data, we recommend that you use the Delta Standalone library as your metadata-only reader, and then perform the Parquet data reading yourself, most likely in a distributed manner. Delta Standalone provides two APIs for reading the files in a given table snapshot. `Snapshot::getAllFiles` returns an in-memory list. As of 0.3.0, we also provide `Snapshot::scan(filter)::getFiles`, which supports partition pruning and an optimized internal iterator implementation. We will use the latter here. ```java import io.delta.standalone.Snapshot; DeltaLog log = DeltaLog.forTable(new Configuration(), "/data/sales"); Snapshot latestSnapshot = log.update(); StructType schema = latestSnapshot.getMetadata().getSchema(); DeltaScan scan = latestSnapshot.scan( new And( new And( new EqualTo(schema.column("year"), Literal.of(2021)), new EqualTo(schema.column("month"), Literal.of(11))), new EqualTo(schema.column("customer"), Literal.of("XYZ")) ) ); CloseableIterator iter = scan.getFiles(); try { while (iter.hasNext()) { AddFile addFile = iter.next(); // Zappy engine to handle reading data in `addFile.getPath()` and apply any `scan.getResidualPredicate()` } } finally { iter.close(); } ``` #### 6.2. Reading Parquet data (single-JVM) [Section titled “6.2. Reading Parquet data (single-JVM)”](#62-reading-parquet-data-single-jvm) Delta Standalone allows reading the Parquet data directly, using `Snapshot::open`. ```java import io.delta.standalone.data.RowRecord; CloseableIterator dataIter = log.update().open(); try { while (dataIter.hasNext()) { RowRecord row = dataIter.next(); int year = row.getInt("year"); String customer = row.getString("customer"); float totalCost = row.getFloat("total_cost"); } } finally { dataIter.close(); } ``` ## Reporting issues [Section titled “Reporting issues”](#reporting-issues) We use [GitHub Issues](https://github.com/delta-io/delta/issues) to track community reported issues. You can also [contact](#community) the community for getting answers. ## Contributing [Section titled “Contributing”](#contributing) We welcome contributions to Delta Lake repository. We use [GitHub Pull Requests](https://github.com/delta-io/delta/pulls) for accepting changes. ## Community [Section titled “Community”](#community) There are two ways to communicate with the Delta Lake community: * Public Slack Channel * [Register to join the Slack channel](https://join.slack.com/t/delta-users/shared_invite/enQtNTY1NDg0ODcxOTI1LWJkZGU3ZmQ3MjkzNmY2ZDM0NjNlYjE4MWIzYjg2OWM1OTBmMWIxZTllMjg3ZmJkNjIwZmE1ZTZkMmQ0OTk5ZjA) * [Sign in to the Slack channel](https://delta-users.slack.com/) * Public [mailing list](https://groups.google.com/forum/#!forum/delta-users) ## Local development [Section titled “Local development”](#local-development) Before local debugging of `standalone` tests in IntelliJ, run all tests with `build/sbt standalone/test`. This helps IntelliJ recognize the golden tables as class resources. # Starburst connector > Learn how to set up an integration to enable you to read Delta tables from Starburst. Starburst natively supports reading and writing Delta Lake tables. For details on using the native Delta Lake connector, see [Delta Lake Connector - Starburst](https://docs.starburst.io/latest/connector/delta-lake.html). # Storage configuration > Learn how to configure Delta Lake on different storage systems. Delta Lake ACID guarantees are predicated on the atomicity and durability guarantees of the storage system. Specifically, Delta Lake relies on the following when interacting with storage systems: * **Atomic visibility**: There must a way for a file to visible in its entirety or not visible at all. * **Mutual exclusion**: Only one writer must be able to create (or rename) a file at the final destination. * **Consistent listing**: Once a file has been written in a directory, all future listings for that directory must return that file. Because storage systems do not necessarily provide all of these guarantees out-of-the-box, Delta Lake transactional operations typically go through the [LogStore API](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java) instead of accessing the storage system directly. To provide the ACID guarantees for different storage systems, you may have to use different `LogStore` implementations. This article covers how to configure Delta Lake for various storage systems. There are two categories of storage systems: * **Storage systems with built-in support**: For some storage systems, you do not need additional configurations. Delta Lake uses the scheme of the path (that is, `s3a` in `s3a://path`) to dynamically identify the storage system and use the corresponding `LogStore` implementation that provides the transactional guarantees. However, for S3, there are additional caveats on concurrent writes. See the [section on S3](#amazon-s3) for details. * **Other storage systems**: The `LogStore`, similar to Apache Spark, uses Hadoop `FileSystem` API to perform reads and writes. So Delta Lake supports concurrent reads on any storage system that provides an implementation of `FileSystem` API. For concurrent writes with transactional guarantees, there are two cases based on the guarantees provided by `FileSystem` implementation. If the implementation provides consistent listing and atomic renames-without-overwrite (that is, `rename(..., overwrite = false)` will either generate the target file atomically or fail if it already exists with `java.nio.file.FileAlreadyExistsException`), then the default `LogStore` implementation using renames will allow concurrent writes with guarantees. Otherwise, you must configure a custom implementation of `LogStore` by setting the following Spark configuration: ```ini spark.delta.logStore..impl= ``` where `` is the scheme of the paths of your storage system. This configures Delta Lake to dynamically use the given `LogStore` implementation only for those paths. You can have multiple such configurations for different schemes in your application, thus allowing it to simultaneously read and write from different storage systems. Note * Delta Lake on local file system may not support concurrent transactional writes. This is because the local file system may or may not provide atomic renames. So you should not use the local file system for testing concurrent writes. - Before version 1.0, Delta Lake supported configuring LogStores by setting `spark.delta.logStore.class`. This approach is now deprecated. Setting this configuration will use the configured `LogStore` for all paths, thereby disabling the dynamic scheme-based delegation. ## Troubleshooting Delta Storage dependency error [Section titled “Troubleshooting Delta Storage dependency error”](#troubleshooting-delta-storage-dependency-error) If you see an error like `java.lang.NoClassDefFoundError: io/delta/storage/LogStore` it usually means the **Delta Storage** dependency is missing from the Spark classpath. ##### Error Message with stack trace [Section titled “Error Message with stack trace”](#error-message-with-stack-trace) ```text com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: io/delta/storage/LogStore Please ensure that the delta-storage dependency is included. If using Python, please ensure you call `configure_spark_with_delta_pip` or use `--packages io.delta:delta-spark_:`. See https://docs.delta.io/latest/quick-start.html#python. More information about this dependency and how to include it can be found here: https://docs.delta.io/latest/porting.html#delta-lake-1-1-or-below-to-delta-lake-1-2-or-above. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2084) at com.google.common.cache.LocalCache.get(LocalCache.java:4017) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4898) at org.apache.spark.sql.delta.DeltaLog$.getDeltaLogFromCache$1(DeltaLog.scala:995) at org.apache.spark.sql.delta.DeltaLog$.initializeDeltaLog$1(DeltaLog.scala:1006) at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:1017) at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:801) at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:197) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:55) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:79) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:77) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:88) at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:155) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$8(SQLExecution.scala:162) at org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:268) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$7(SQLExecution.scala:124) at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94) at org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:112) at org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:106) at org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:111) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:124) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:291) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:123) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:77) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:233) at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:155) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:654) at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$eagerlyExecute$1(QueryExecution.scala:154) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:169) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:164) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:470) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:86) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:470) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:37) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:360) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:356) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:446) at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:164) at org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyCommandExecuted$1(QueryExecution.scala:126) at scala.util.Try$.apply(Try.scala:217) at org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1378) at org.apache.spark.util.Utils$.getTryWithCallerStacktrace(Utils.scala:1439) at org.apache.spark.util.LazyTry.get(LazyTry.scala:58) at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:131) at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:192) at org.apache.spark.sql.classic.DataFrameWriter.runCommand(DataFrameWriter.scala:622) at org.apache.spark.sql.classic.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:273) at org.apache.spark.sql.classic.DataFrameWriter.saveInternal(DataFrameWriter.scala:235) at org.apache.spark.sql.classic.DataFrameWriter.save(DataFrameWriter.scala:118) ... 42 elided ``` ### Why this happens [Section titled “Why this happens”](#why-this-happens) When you provide the Delta Spark JAR using the `--jars` option (for example, when testing a locally-built JAR), Spark **does not automatically fetch transitive dependencies**. In this case, `delta-spark` may be present, but `delta-storage` is not. As a result, operations that need to initialize the Delta log (for example, writing a Delta table) can fail when Delta attempts to load the LogStore API: ```scala df.write.format("delta").save("/tmp/delta") ``` ### How to fix [Section titled “How to fix”](#how-to-fix) When using the `--jars` option, you can do either of the following: * Include both `delta-spark` and `delta-storage` JARs: ```bash spark-shell \ --jars delta-spark_-.jar,delta-storage-.jar \ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \ --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ``` Note * When specifying the JARs with `--jars`, do not include spaces after the comma. - Use the assembly JAR (build it using `build/sbt "spark/assembly"`): ```bash spark-shell \ --jars delta-spark-assembly-.jar \ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \ --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ``` ## Amazon S3 [Section titled “Amazon S3”](#amazon-s3) Delta Lake supports reads and writes to S3 in two different modes: Single-cluster and Multi-cluster. | | Single-cluster | Multi-cluster | | ------------- | ------------------------------------------------------- | ------------------------------------------------ | | Configuration | Comes with Delta Lake out-of-the-box | Is experimental and requires extra configuration | | Reads | Supports concurrent reads from multiple clusters | Supports concurrent reads from multiple clusters | | Writes | Supports concurrent writes from a *single* Spark driver | Supports multi-cluster writes | | Permissions | S3 credentials | S3 and DynamoDB operating permissions | ### Single-cluster setup (default) [Section titled “Single-cluster setup (default)”](#single-cluster-setup-default) In this default mode, Delta Lake supports concurrent reads from multiple clusters, but concurrent writes to S3 must originate from a *single* Spark driver in order for Delta Lake to provide transactional guarantees. This is because S3 currently does not provide mutual exclusion, that is, there is no way to ensure that only one writer is able to create a file. Caution Concurrent writes to the same Delta table on S3 storage from multiple Spark drivers can lead to data loss. For a multi-cluster solution, please see the [Multi-cluster setup](#multi-cluster-setup) section below. #### Requirements (S3 single-cluster) [Section titled “Requirements (S3 single-cluster)”](#requirements-s3-single-cluster) * S3 credentials: [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) (recommended) or access keys * Apache Spark associated with the corresponding Delta Lake version. * Hadoop’s [AWS connector (hadoop-aws)](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/) for the version of Hadoop that Apache Spark is compiled for. #### Quickstart (S3 single-cluster) [Section titled “Quickstart (S3 single-cluster)”](#quickstart-s3-single-cluster) This section explains how to quickly start reading and writing Delta tables on S3 using single-cluster mode. For a detailed explanation of the configuration, see [Setup Configuration (S3 multi-cluster)](#setup-configuration-s3-multi-cluster). 1. Use the following command to launch a Spark shell with Delta Lake and S3 support (assuming you use Spark 4.0.0 which is pre-built for Hadoop 3.4.0): * Bash ```bash bin/spark-shell \ --packages io.delta:delta-spark_2.13:4.0.0,org.apache.hadoop:hadoop-aws:3.4.0 \ --conf spark.hadoop.fs.s3a.access.key= \ --conf spark.hadoop.fs.s3a.secret.key= ``` 2. Try out some basic Delta table operations on S3 (in Scala): * Scala ```scala // Create a Delta table on S3: spark.range(5).write.format("delta").save("s3a:///") // Read a Delta table on S3: spark.read.format("delta").load("s3a:///").show() ``` For other languages and more examples of Delta table operations, see the [Quickstart](/quick-start/) page. For efficient listing of Delta Lake metadata files on S3, set the configuration `delta.enableFastS3AListFrom=true`. This performance optimization is in experimental support mode. It will only work on `S3A` filesystems and will not work on [Amazon’s EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html) default filesystem `S3`. * Scala ```scala bin/spark-shell \ --packages io.delta:delta-spark_2.13:4.0.0,org.apache.hadoop:hadoop-aws:3.4.0 \ --conf spark.hadoop.fs.s3a.access.key= \ --conf spark.hadoop.fs.s3a.secret.key= \ --conf "spark.hadoop.delta.enableFastS3AListFrom=true ``` ### Multi-cluster setup [Section titled “Multi-cluster setup”](#multi-cluster-setup) Note This support is new and experimental. This mode supports concurrent writes to S3 from multiple clusters and has to be explicitly enabled by configuring Delta Lake to use the right `LogStore` implementation. This implementation uses [DynamoDB](https://aws.amazon.com/dynamodb/) to provide the mutual exclusion that S3 is lacking. Caution This multi-cluster writing solution is only safe when all writers use this `LogStore` implementation as well as the same DynamoDB table and region. If some drivers use out-of-the-box Delta Lake while others use this experimental `LogStore`, then data loss can occur. #### Requirements (S3 multi-cluster) [Section titled “Requirements (S3 multi-cluster)”](#requirements-s3-multi-cluster) * All of the requirements listed in [Requirements (S3 single-cluster)](#requirements-s3-single-cluster) section * In additon to S3 credentials, you also need DynamoDB operating permissions #### Quickstart (S3 multi-cluster) [Section titled “Quickstart (S3 multi-cluster)”](#quickstart-s3-multi-cluster) This section explains how to quickly start reading and writing Delta tables on S3 using multi-cluster mode. 1. Use the following command to launch a Spark shell with Delta Lake and S3 support (assuming you use Spark 4.0.0 which is pre-built for Hadoop 3.4.0): * Bash ```bash bin/spark-shell \ --packages io.delta:delta-spark_2.13:3,org.apache.hadoop:hadoop-aws:3.4.0,io.delta:delta-storage-s3-dynamodb:4.0.0 \ --conf spark.hadoop.fs.s3a.access.key= \ --conf spark.hadoop.fs.s3a.secret.key= \ --conf spark.delta.logStore.s3a.impl=io.delta.storage.S3DynamoDBLogStore \ --conf spark.io.delta.storage.S3DynamoDBLogStore.ddb.region=us-west-2 ``` 2. Try out some basic Delta table operations on S3 (in Scala): * Scala ```scala // Create a Delta table on S3: spark.range(5).write.format("delta").save("s3a:///") // Read a Delta table on S3: spark.read.format("delta").load("s3a:///").show() ``` For other languages and more examples of Delta table operations, see the [Quickstart](/quick-start/) page. #### Setup Configuration (S3 multi-cluster) [Section titled “Setup Configuration (S3 multi-cluster)”](#setup-configuration-s3-multi-cluster) 1. Create the DynamoDB table. You have the choice of creating the DynamoDB table yourself (recommended) or having it created for you automatically. * Creating the DynamoDB table yourself This DynamoDB table will maintain commit metadata for multiple Delta tables, and it is important that it is configured with the [Read/Write Capacity Mode](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html) (for example, on-demand or provisioned) that is right for your use cases. As such, we strongly recommend that you create your DynamoDB table yourself. The following example uses the AWS CLI. To learn more, see the [create-table](https://docs.aws.amazon.com/cli/latest/reference/dynamodb/create-table.html) command reference. * Bash ```bash aws dynamodb create-table \ --region us-east-1 \ --table-name delta_log \ --attribute-definitions AttributeName=tablePath,AttributeType=S \ AttributeName=fileName,AttributeType=S \ --key-schema AttributeName=tablePath,KeyType=HASH \ AttributeName=fileName,KeyType=RANGE \ --billing-mode PAY_PER_REQUEST ``` 2. Follow the configuration steps listed in [Configuration (S3 single-cluster)](#configuration-s3-single-cluster) section. 3. Include the `delta-storage-s3-dynamodb` JAR in the classpath. 4. Configure the `LogStore` implementation in your Spark session. First, configure this `LogStore` implementation for the scheme `s3`. You can replicate this command for schemes `s3a` and `s3n` as well. * ini ```ini spark.delta.logStore.s3.impl=io.delta.storage.S3DynamoDBLogStore ``` Next, specify additional information necessary to instantiate the DynamoDB client. You must instantiate the DynamoDB client with the same `tableName` and `region` each Spark session for this multi-cluster mode to work correctly. A list of per-session configurations and their defaults is given below: * ini ```ini spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName=delta_log spark.io.delta.storage.S3DynamoDBLogStore.ddb.region=us-east-1 spark.io.delta.storage.S3DynamoDBLogStore.credentials.provider= spark.io.delta.storage.S3DynamoDBLogStore.provisionedThroughput.rcu=5 spark.io.delta.storage.S3DynamoDBLogStore.provisionedThroughput.wcu=5 ``` #### Production Configuration (S3 multi-cluster) [Section titled “Production Configuration (S3 multi-cluster)”](#production-configuration-s3-multi-cluster) By this point, this multi-cluster setup is fully operational. However, there is extra configuration you may do to improve performance and optimize storage when running in production. 1. Adjust your Read and Write Capacity Mode. If you are using the default DynamoDB table created for you by this `LogStore` implementation, its default RCU and WCU might not be enough for your workloads. You can [adjust the provisioned throughput](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ProvisionedThroughput.html#ProvisionedThroughput.CapacityUnits.Modifying) or [update to On-Demand Mode](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.Basics.html#WorkingWithTables.Basics.UpdateTable). 2. Cleanup old DynamoDB entries using Time to Live (TTL). Once a DynamoDB metadata entry is marked as complete, and after sufficient time such that we can now rely on S3 alone to prevent accidental overwrites on its corresponding Delta file, it is safe to delete that entry from DynamoDB. The cheapest way to do this is using [DynamoDB’s TTL](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html) feature which is a free, automated means to delete items from your DynamoDB table. Run the following command on your given DynamoDB table to enable TTL: * Bash ```bash aws dynamodb update-time-to-live \ --region us-east-1 \ --table-name delta_log \ --time-to-live-specification "Enabled=true, AttributeName=expireTime" ``` The default `expireTime` will be one day after the DynamoDB entry was marked as completed. 3. Cleanup old AWS S3 temp files using S3 Lifecycle Expiration. In this `LogStore` implementation, a temp file is created containing a copy of the metadata to be committed into the Delta log. Once that commit to the Delta log is complete, and after the corresponding DynamoDB entry has been removed, it is safe to delete this temp file. In practice, only the latest temp file will ever be used during recovery of a failed commit. Here are two simple options for deleting these temp files: 1. Delete manually using S3 CLI. This is the safest option. The following command will delete all but the latest temp file in your given `` and ``: * Bash ```bash aws s3 ls s3:////_delta_log/.tmp/ --recursive | awk 'NF>1{print $4}' | grep . | sort | head -n -1 | while read -r line ; do echo "Removing ${line}" aws s3 rm s3:////_delta_log/.tmp/${line} done ``` 2. Delete using an S3 Lifecycle Expiration Rule A more automated option is to use an [S3 Lifecycle Expiration rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html), with filter prefix pointing to the `/_delta_log/.tmp/` folder located in your table path, and an expiration value of 30 days. Note It is important that you choose a sufficiently large expiration value. As stated above, the latest temp file will be used during recovery of a failed commit. If this temp file is deleted, then your DynamoDB table and S3 `/_delta_log/.tmp/` folder will be out of sync. There are a variety of ways to configuring a bucket lifecycle configuration, described in AWS docs [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html). One way to do this is using S3’s `put-bucket-lifecycle-configuration` command. See [S3 Lifecycle Configuration](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html) for details. An example rule and command invocation is given below: In a file referenced as `file://lifecycle.json`: * JSON ```json { "Rules":[ { "ID":"expire_tmp_files", "Filter":{ "Prefix":"path/to/table/_delta_log/.tmp/" }, "Status":"Enabled", "Expiration":{ "Days":30 } } ] } ``` - Bash ```bash aws s3api put-bucket-lifecycle-configuration \ --bucket my-bucket \ --lifecycle-configuration file://lifecycle.json ``` Note AWS S3 may have a limit on the number of rules per bucket. See [PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html) for details. ## Microsoft Azure storage [Section titled “Microsoft Azure storage”](#microsoft-azure-storage) Delta Lake has built-in support for the various Azure storage systems with full transactional guarantees for concurrent reads and writes from multiple clusters. Delta Lake relies on Hadoop `FileSystem` APIs to access Azure storage services. Specifically, Delta Lake requires the implementation of `FileSystem.rename()` to be atomic, which is only supported in newer Hadoop versions ([Hadoop-15156](https://issues.apache.org/jira/browse/HADOOP-15156) and [Hadoop-15086](https://issues.apache.org/jira/browse/HADOOP-15086)). For this reason, you may need to build Spark with newer Hadoop versions and use them for deploying your application. See [Specifying the Hadoop Version and Enabling YARN](https://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn) for building Spark with a specific Hadoop version and [Quickstart](/quick-start/) for setting up Spark with Delta Lake. Here is a list of requirements specific to each type of Azure storage system: ### Azure Blob storage [Section titled “Azure Blob storage”](#azure-blob-storage) #### Requirements (Azure Blob storage) [Section titled “Requirements (Azure Blob storage)”](#requirements-azure-blob-storage) * A [shared key](https://docs.microsoft.com/rest/api/storageservices/authorize-with-shared-key) or [shared access signature (SAS)](https://docs.microsoft.com/azure/storage/common/storage-dotnet-shared-access-signature-part-1) * Delta Lake 0.2.0 or above * Hadoop’s Azure Blob Storage libraries for deployment with the following versions: * 2.9.1+ for Hadoop 2 * 3.0.1+ for Hadoop 3 * Apache Spark associated with the corresponding Delta Lake version and [compiled with Hadoop version](https://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn) that is compatible with the chosen Hadoop libraries. For example, a possible combination that will work is Delta 0.7.0 or above, along with Apache Spark 3.0 compiled and deployed with Hadoop 3.2. #### Configuration (Azure Blob storage) [Section titled “Configuration (Azure Blob storage)”](#configuration-azure-blob-storage) Here are the steps to configure Delta Lake on Azure Blob storage. 1. Include `hadoop-azure` JAR in the classpath. See the requirements above for version details. 2. Set up credentials. You can set up your credentials in the [Spark configuration property](https://spark.apache.org/docs/latest/configuration.html). We recommend that you use a SAS token. In Scala, you can use the following: * Scala ```scala spark.conf.set( "fs.azure.sas...blob.core.windows.net", "") ``` Or you can specify an account access key: * Scala ```scala spark.conf.set( "fs.azure.account.key..blob.core.windows.net", "") ``` #### Usage (Azure Blob storage) [Section titled “Usage (Azure Blob storage)”](#usage-azure-blob-storage) * Scala ```scala spark.range(5).write.format("delta").save("wasbs://@.blob.core.windows.net/") spark.read.format("delta").load("wasbs://@.blob.core.windows.net/").show() ``` ### Azure Data Lake Storage Gen1 [Section titled “Azure Data Lake Storage Gen1”](#azure-data-lake-storage-gen1) #### Requirements (ADLS Gen1) [Section titled “Requirements (ADLS Gen1)”](#requirements-adls-gen1) * A [service principal](https://docs.microsoft.com/azure/active-directory/develop/app-objects-and-service-principals) for OAuth 2.0 access * Delta Lake 0.2.0 or above * Hadoop’s Azure Data Lake Storage Gen1 libraries for deployment with the following versions: * 2.9.1+ for Hadoop 2 * 3.0.1+ for Hadoop 3 * Apache Spark associated with the corresponding Delta Lake version and [compiled with Hadoop version](https://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn) that is compatible with the chosen Hadoop libraries. #### Configuration (ADLS Gen1) [Section titled “Configuration (ADLS Gen1)”](#configuration-adls-gen1) Here are the steps to configure Delta Lake on Azure Data Lake Storage Gen1. 1. Include `hadoop-azure-datalake` JAR in the classpath. See the requirements above for version details. 2. Set up Azure Data Lake Storage Gen1 credentials. You can set the following [Hadoop configurations](https://spark.apache.org/docs/latest/configuration.html#custom-hadoophive-configuration) with your credentials (in Scala): * Scala ```scala spark.conf.set("dfs.adls.oauth2.access.token.provider.type", "ClientCredential") spark.conf.set("dfs.adls.oauth2.client.id", "") spark.conf.set("dfs.adls.oauth2.credential", "") spark.conf.set("dfs.adls.oauth2.refresh.url", "https://login.microsoftonline.com//oauth2/token") ``` #### Usage (ADLS Gen1) [Section titled “Usage (ADLS Gen1)”](#usage-adls-gen1) * Scala ```scala spark.range(5).write.format("delta").save("adl://.azuredatalakestore.net/") spark.read.format("delta").load("adl://.azuredatalakestore.net/").show() ``` ### Azure Data Lake Storage Gen2 [Section titled “Azure Data Lake Storage Gen2”](#azure-data-lake-storage-gen2) #### Requirements (ADLS Gen2) [Section titled “Requirements (ADLS Gen2)”](#requirements-adls-gen2) * A [service principal](https://docs.microsoft.com/azure/active-directory/develop/app-objects-and-service-principals) for OAuth 2.0 access or a [shared key](https://docs.microsoft.com/rest/api/storageservices/authorize-with-shared-key) * Delta Lake 0.2.0 or above * Hadoop’s Azure Data Lake Storage Gen2 libraries for deployment with the following versions: * 3.2.0+ for Hadoop 3 * Apache Spark associated with the corresponding Delta Lake version and [compiled with Hadoop version](https://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn) that is compatible with the chosen Hadoop libraries. #### Configuration (ADLS Gen2) [Section titled “Configuration (ADLS Gen2)”](#configuration-adls-gen2) Here are the steps to configure Delta Lake on Azure Data Lake Storage Gen2. 1. Include `hadoop-azure` and `azure-storage` JARs in the classpath. See the requirements above for version details. 2. Set up credentials. You can use either OAuth 2.0 with service principal or shared key authentication: For OAuth 2.0 with service principal (recommended): * Scala ```scala spark.conf.set("fs.azure.account.auth.type..dfs.core.windows.net", "OAuth") spark.conf.set("fs.azure.account.oauth.provider.type..dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set("fs.azure.account.oauth2.client.id..dfs.core.windows.net", "") spark.conf.set("fs.azure.account.oauth2.client.secret..dfs.core.windows.net", "") spark.conf.set("fs.azure.account.oauth2.client.endpoint..dfs.core.windows.net", "https://login.microsoftonline.com//oauth2/token") ``` For shared key authentication: * Scala ```scala spark.conf.set("fs.azure.account.key..dfs.core.windows.net", "") ``` #### Usage (ADLS Gen2) [Section titled “Usage (ADLS Gen2)”](#usage-adls-gen2) * Scala ```scala spark.range(5).write.format("delta").save("abfss://@.dfs.core.windows.net/") spark.read.format("delta").load("abfss://@.dfs.core.windows.net/").show() ``` ## HDFS [Section titled “HDFS”](#hdfs) Delta Lake has built-in support for HDFS with full transactional guarantees for concurrent reads and writes from multiple clusters. No additional configuration is required. #### Usage (HDFS) [Section titled “Usage (HDFS)”](#usage-hdfs) * Scala ```scala spark.range(5).write.format("delta").save("hdfs://:/") spark.read.format("delta").load("hdfs://:/").show() ``` ## Google Cloud Storage [Section titled “Google Cloud Storage”](#google-cloud-storage) Delta Lake has built-in support for Google Cloud Storage (GCS) with full transactional guarantees for concurrent reads and writes from multiple clusters. ### Requirements (GCS) [Section titled “Requirements (GCS)”](#requirements-gcs) * Google Cloud Storage credentials * Delta Lake 0.2.0 or above * Hadoop’s [GCS connector](https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage) for the version of Hadoop that Apache Spark is compiled for ### Configuration (GCS) [Section titled “Configuration (GCS)”](#configuration-gcs) 1. Include the GCS connector JAR in the classpath. 2. Set up credentials using one of the following methods: * Use [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) * Configure service account credentials in Spark configuration * Scala ```scala spark.conf.set("google.cloud.auth.service.account.json.keyfile", "") ``` ### Usage (GCS) [Section titled “Usage (GCS)”](#usage-gcs) * Scala ```scala spark.range(5).write.format("delta").save("gs:///") spark.read.format("delta").load("gs:///").show() ``` ## Oracle Cloud Infrastructure [Section titled “Oracle Cloud Infrastructure”](#oracle-cloud-infrastructure) Delta Lake supports Oracle Cloud Infrastructure (OCI) Object Storage with full transactional guarantees for concurrent reads and writes from multiple clusters. ### Requirements (OCI) [Section titled “Requirements (OCI)”](#requirements-oci) * OCI credentials * Delta Lake 0.2.0 or above * Hadoop’s [OCI connector](https://docs.oracle.com/en-us/iaas/Content/api/SDKDocs/hdfsconnector.htm) for the version of Hadoop that Apache Spark is compiled for ### Configuration (OCI) [Section titled “Configuration (OCI)”](#configuration-oci) 1. Include the OCI connector JAR in the classpath. 2. Set up credentials in Spark configuration: * Scala ```scala spark.conf.set("fs.oci.client.auth.tenantId", "") spark.conf.set("fs.oci.client.auth.userId", "") spark.conf.set("fs.oci.client.auth.fingerprint", "") spark.conf.set("fs.oci.client.auth.pemfilepath", "") ``` ### Usage (OCI) [Section titled “Usage (OCI)”](#usage-oci) * Scala ```scala spark.range(5).write.format("delta").save("oci://@/") spark.read.format("delta").load("oci://@/").show() ``` ## IBM Cloud Object Storage [Section titled “IBM Cloud Object Storage”](#ibm-cloud-object-storage) Delta Lake supports IBM Cloud Object Storage with full transactional guarantees for concurrent reads and writes from multiple clusters. ### Requirements (IBM COS) [Section titled “Requirements (IBM COS)”](#requirements-ibm-cos) * IBM Cloud Object Storage credentials * Delta Lake 0.2.0 or above * Hadoop’s [Stocator connector](https://github.com/CODAIT/stocator) for the version of Hadoop that Apache Spark is compiled for ### Configuration (IBM COS) [Section titled “Configuration (IBM COS)”](#configuration-ibm-cos) 1. Include the Stocator connector JAR in the classpath. 2. Set up credentials in Spark configuration: * Scala ```scala spark.conf.set("fs.cos.service.endpoint", "") spark.conf.set("fs.cos.service.access.key", "") spark.conf.set("fs.cos.service.secret.key", "") ``` ### Usage (IBM COS) [Section titled “Usage (IBM COS)”](#usage-ibm-cos) * Scala ```scala spark.range(5).write.format("delta").save("cos://./") spark.read.format("delta").load("cos://./").show() ``` # Table streaming reads and writes > Learn how to use Delta tables as streaming sources and sinks. Delta Lake is deeply integrated with [Spark Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) through `readStream` and `writeStream`. Delta Lake overcomes many of the limitations typically associated with streaming systems and files, including: * Maintaining “exactly-once” processing with more than one stream (or concurrent batch jobs) * Efficiently discovering which files are new when using files as the source for a stream For many Delta Lake operations on tables, you enable integration with Apache Spark DataSourceV2 and Catalog APIs (since 3.0) by setting configurations when you create a new `SparkSession`. See [Configure SparkSession](/delta-batch/#configure-sparksession). ## Delta table as a source [Section titled “Delta table as a source”](#delta-table-as-a-source) When you load a Delta table as a stream source and use it in a streaming query, the query processes all of the data present in the table as well as any new data that arrives after the stream is started. * Scala ```scala spark.readStream.format("delta") .load("/tmp/delta/events") import io.delta.implicits._ spark.readStream.delta("/tmp/delta/events") ``` ### Limit input rate [Section titled “Limit input rate”](#limit-input-rate) The following options are available to control micro-batches: * `maxFilesPerTrigger`: How many new files to be considered in every micro-batch. The default is 1000. * `maxBytesPerTrigger`: How much data gets processed in each micro-batch. This option sets a “soft max”, meaning that a batch processes approximately this amount of data and may process more than the limit in order to make the streaming query move forward in cases when the smallest input unit is larger than this limit. If you use `Trigger.Once` for your streaming, this option is ignored. This is not set by default. If you use `maxBytesPerTrigger` in conjunction with `maxFilesPerTrigger`, the micro-batch processes data until either the `maxFilesPerTrigger` or `maxBytesPerTrigger` limit is reached. Note In cases when the source table transactions are cleaned up due to the `logRetentionDuration` [configuration](/delta-batch/#data-retention) and the stream lags in processing, Delta Lake processes the data corresponding to the latest available transaction history of the source table but does not fail the stream. This can result in data being dropped. ### Ignore updates and deletes [Section titled “Ignore updates and deletes”](#ignore-updates-and-deletes) Structured Streaming does not handle input that is not an append and throws an exception if any modifications occur on the table being used as a source. There are two main strategies for dealing with changes that cannot be automatically propagated downstream: * You can delete the output and checkpoint and restart the stream from the beginning. * You can set one of the following options: * `skipChangeCommits` (recommended): skip commits that contain data-changing operations such as `UPDATE`, `MERGE INTO`, `DELETE`, or `OVERWRITE`. Skipped commits are not processed, so downstream does not see rows from those commits. * `ignoreDeletes` (legacy): ignore transactions that delete data at partition boundaries. * `ignoreChanges` (deprecated in favor of `skipChangeCommits`): re-process updates if files had to be rewritten in the source table due to a data changing operation such as `UPDATE`, `MERGE INTO`, `DELETE` (within partitions), or `OVERWRITE`. Unchanged rows may still be emitted, therefore your downstream consumers should be able to handle duplicates. Deletes are not propagated downstream. `ignoreChanges` subsumes `ignoreDeletes`. Therefore if you use `ignoreChanges`, your stream will not be disrupted by either deletions or updates to the source table. ### Specify initial position [Section titled “Specify initial position”](#specify-initial-position) You can use the following options to specify the starting point of the Delta Lake streaming source without processing the entire table. * `startingVersion`: The Delta Lake version to start from. All table changes starting from this version (inclusive) will be read by the streaming source. You can obtain the commit versions from the `version` column of the [DESCRIBE HISTORY](/delta-utility/#retrieve-delta-table-history) command output. * To return only the latest changes, specify `latest`. * `startingTimestamp`: The timestamp to start from. All table changes committed at or after the timestamp (inclusive) will be read by the streaming source. One of: * A timestamp string. For example, `"2019-01-01T00:00:00.000Z"`. * A date string. For example, `"2019-01-01"`. You cannot set both options at the same time; you can use only one of them. They take effect only when starting a new streaming query. If a streaming query has started and the progress has been recorded in its checkpoint, these options are ignored. Caution Although you can start the streaming source from a specified version or timestamp, the schema of the streaming source is always the latest schema of the Delta table. You must ensure there is no incompatible schema change to the Delta table after the specified version or timestamp. Otherwise, the streaming source may return incorrect results when reading the data with an incorrect schema. #### Example [Section titled “Example”](#example) For example, suppose you have a table `user_events`. If you want to read changes since version 5, use: * Scala ```scala spark.readStream.format("delta") .option("startingVersion", "5") .load("/tmp/delta/user_events") ``` If you want to read changes since 2018-10-18, use: * Scala ```scala spark.readStream.format("delta") .option("startingTimestamp", "2018-10-18") .load("/tmp/delta/user_events") ``` ### Process initial snapshot without data being dropped [Section titled “Process initial snapshot without data being dropped”](#process-initial-snapshot-without-data-being-dropped) When using a Delta table as a stream source, the query first processes all of the data present in the table. The Delta table at this version is called the initial snapshot. By default, the Delta table’s data files are processed based on which file was last modified. However, the last modification time does not necessarily represent the record event time order. In a stateful streaming query with a defined watermark, processing files by modification time can result in records being processed in the wrong order. This could lead to records dropping as late events by the watermark. You can avoid the data drop issue by enabling the following option: * withEventTimeOrder: Whether the initial snapshot should be processed with event time order. With event time order enabled, the event time range of initial snapshot data is divided into time buckets. Each micro batch processes a bucket by filtering data within the time range. The maxFilesPerTrigger and maxBytesPerTrigger configuration options are still applicable to control the microbatch size but only in an approximate way due to the nature of the processing. The graphic below shows this process:  Notable information about this feature: * The data drop issue only happens when the initial Delta snapshot of a stateful streaming query is processed in the default order. * You cannot change `withEventTimeOrder` once the stream query is started while the initial snapshot is still being processed. To restart with `withEventTimeOrder` changed, you need to delete the checkpoint. * If you are running a stream query with withEventTimeOrder enabled, you cannot downgrade it to a Delta version which doesn’t support this feature until the initial snapshot processing is completed. If you need to downgrade, you can wait for the initial snapshot to finish, or delete the checkpoint and restart the query. * This feature is not supported in the following uncommon scenarios: * The event time column is a generated column and there are non-projection transformations between the Delta source and watermark. * There is a watermark that has more than one Delta source in the stream query. * With event time order enabled, the performance of the Delta initial snapshot processing might be slower. * Each micro batch scans the initial snapshot to filter data within the corresponding event time range. For faster filter action, it is advised to use a Delta source column as the event time so that data skipping can be applied (check \_ for when it’s applicable). Additionally, table partitioning along the event time column can further speed the processing. You can check Spark UI to see how many delta files are scanned for a specific micro batch. #### Example [Section titled “Example”](#example-1) Suppose you have a table `user_events` with an `event_time` column. Your streaming query is an aggregation query. If you want to ensure no data drop during the initial snapshot processing, you can use: * Scala ```scala spark.readStream.format("delta") .option("withEventTimeOrder", "true") .load("/tmp/delta/user_events") .withWatermark("event_time", "10 seconds") ``` Note You can also enable this with Spark config on the cluster which will apply to all streaming queries: ```plaintext spark.databricks.delta.withEventTimeOrder.enabled true ``` ### Tracking non-additive schema changes [Section titled “Tracking non-additive schema changes”](#tracking-non-additive-schema-changes) You can provide a schema tracking location to enable streaming from Delta tables with column mapping enabled. This overcomes an issue in which non-additive schema changes could result in broken streams by allowing streams to read past table data in their exact schema as if the table is time-travelled. Each streaming read against a data source must have its own `schemaTrackingLocation` specified. The specified `schemaTrackingLocation` must be contained within the directory specified for the `checkpointLocation` of the target table for streaming write. Note For streaming workloads that combine data from multiple source Delta tables, you need to specify unique directories within the `checkpointLocation` for each source table. #### Example [Section titled “Example”](#example-2) The option `schemaTrackingLocation` is used to specify the path for schema tracking, as shown in the following code example: * Python ```python checkpoint_path = "/path/to/checkpointLocation" (spark.readStream .option("schemaTrackingLocation", checkpoint_path) .table("delta_source_table") .writeStream .option("checkpointLocation", checkpoint_path) .toTable("output_table") ) ``` ## Delta table as a sink [Section titled “Delta table as a sink”](#delta-table-as-a-sink) You can also write data into a Delta table using Structured Streaming. The transaction log enables Delta Lake to guarantee exactly-once processing, even when there are other streams or batch queries running concurrently against the table. Note The Delta Lake `VACUUM` function removes all files not managed by Delta Lake but skips any directories that begin with `_`. You can safely store checkpoints alongside other data and metadata for a Delta table using a directory structure such as `/_checkpoints`. ### Append mode [Section titled “Append mode”](#append-mode) By default, streams run in append mode, which adds new records to the table. You can use the path method: * Python ```python events.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/tmp/delta/_checkpoints/") .start("/delta/events") ``` * Scala ```scala events.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/tmp/delta/events/_checkpoints/") .start("/tmp/delta/events") import io.delta.implicits._ events.writeStream .outputMode("append") .option("checkpointLocation", "/tmp/delta/events/_checkpoints/") .delta("/tmp/delta/events") ``` or the `toTable` method (in Spark 3.1 and higher) as follows: * Python ```python events.writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "/tmp/delta/events/_checkpoints/") .toTable("events") ``` * Scala ```scala events.writeStream .outputMode("append") .option("checkpointLocation", "/tmp/delta/events/_checkpoints/") .toTable("events") ``` ### Complete mode [Section titled “Complete mode”](#complete-mode) You can also use Structured Streaming to replace the entire table with every batch. One example use case is to compute a summary using aggregation: * Python ```python (spark.readStream .format("delta") .load("/tmp/delta/events") .groupBy("customerId") .count() .writeStream .format("delta") .outputMode("complete") .option("checkpointLocation", "/tmp/delta/eventsByCustomer/_checkpoints/") .start("/tmp/delta/eventsByCustomer") ) ``` * Scala ```scala spark.readStream .format("delta") .load("/tmp/delta/events") .groupBy("customerId") .count() .writeStream .format("delta") .outputMode("complete") .option("checkpointLocation", "/tmp/delta/eventsByCustomer/_checkpoints/") .start("/tmp/delta/eventsByCustomer") ``` The preceding example continuously updates a table that contains the aggregate number of events by customer. For applications with more lenient latency requirements, you can save computing resources with one-time triggers. Use these to update summary aggregation tables on a given schedule, processing only new data that has arrived since the last update. ## Idempotent table writes in `foreachBatch` [Section titled “Idempotent table writes in foreachBatch”](#idempotent-table-writes-in-foreachbatch) Note Available in Delta Lake 2.0.0 and above. The command foreachBatch allows you to specify a function that is executed on the output of every micro-batch after arbitrary transformations in the streaming query. This allows implementating a `foreachBatch` function that can write the micro-batch output to one or more target Delta table destinations. However, `foreachBatch` does not make those writes idempotent as those write attempts lack the information of whether the batch is being re-executed or not. For example, rerunning a failed batch could result in duplicate data writes. To address this, Delta tables support the following `DataFrameWriter` options to make the writes idempotent: * `txnAppId`: A unique string that you can pass on each `DataFrame` write. For example, you can use the StreamingQuery ID as `txnAppId`. * `txnVersion`: A monotonically increasing number that acts as transaction version. Delta table uses the combination of `txnAppId` and `txnVersion` to identify duplicate writes and ignore them. If a batch write is interrupted with a failure, rerunning the batch uses the same application and batch ID, which would help the runtime correctly identify duplicate writes and ignore them. Application ID (`txnAppId`) can be any user-generated unique string and does not have to be related to the stream ID. Caution If you delete the streaming checkpoint and restart the query with a new checkpoint, you must provide a different `appId`; otherwise, writes from the restarted query will be ignored because it will contain the same `txnAppId` and the batch ID would start from 0. The same `DataFrameWriter` options can be used to achieve the idempotent writes in non-Streaming job. For details [Idempotent writes](/delta-batch/#idempotent-writes). ### Example [Section titled “Example”](#example-3) * Python ```python app_id = ... # A unique string that is used as an application ID. def writeToDeltaLakeTableIdempotent(batch_df, batch_id): batch_df.write.format(...).option("txnVersion", batch_id).option("txnAppId", app_id).save(...) # location 1 batch_df.write.format(...).option("txnVersion", batch_id).option("txnAppId", app_id).save(...) # location 2 ``` * Scala ```scala val appId = ... // A unique string that is used as an application ID. streamingDF.writeStream.foreachBatch { (batchDF: DataFrame, batchId: Long) => batchDF.write.format(...).option("txnVersion", batchId).option("txnAppId", appId).save(...) // location 1 batchDF.write.format(...).option("txnVersion", batchId).option("txnAppId", appId).save(...) // location 2 } ``` # Trino connector > Learn how to set up an integration to enable you to read Delta tables from Trino. Since Trino [version 373](https://trino.io/docs/current/release/release-373.html), Trino natively supports reading and writing the Delta Lake tables. For details on using the native Delta Lake connector, see [Delta Lake Connector - Trino](https://trino.io/docs/current/connector/delta-lake.html). For Trino versions lower than [version 373](https://trino.io/docs/current/release/release-373.html), you can use the manifest-based approach detailed in [Presto, Trino, and Athena to Delta Lake integration using manifests](/presto-integration/). # Delta type widening > Learn about type widening in Delta. Note This feature is available in preview in Delta Lake 3.2 and above, and fully supported in Delta Lake 4.0 and above. The type widening feature allows changing the type of columns in a Delta table to a wider type. This enables manual type changes using the `ALTER TABLE ALTER COLUMN` command and automatic type migration with schema evolution during write operations. ## Supported type changes [Section titled “Supported type changes”](#supported-type-changes) The feature introduces a limited set of supported type changes in Delta Lake 3.2 and expands it in Delta Lake 4.0 and above. | Source type | Supported wider types - Delta 3.2 | Supported wider types - Delta 4.0 | | ----------- | --------------------------------- | ------------------------------------------- | | `byte` | `short`, `int` | `short`, `int`, `long`, `decimal`, `double` | | `short` | `int` | `int`, `long`, `decimal`, `double` | | `int` | | `long`, `decimal`, `double` | | `long` | | `decimal` | | `float` | | `double` | | `decimal` | | `decimal` with greater precision and scale | | `date` | | `timestampNTZ` | To avoid accidentally promoting integer values to decimals, you must **manually commit** type changes from `byte`, `short`, `int`, or `long` to `decimal` or `double`. When promoting an integer type to `decimal` or `double`, if any downstream ingestion writes this value back to an integer column, Spark will truncate the fractional part of the values by default. Note When changing an integer or decimal type to decimal, the total precision must be equal to or greater than the starting precision. If you also increase the scale, the total precision must increase by a corresponding amount. That is, `decimal(p, s)` can be changed to `decimal(p + k1, s + k2)` iff `k1 >= k2 >= 0`. For example, if you want to add two decimal places to a field with `decimal(10,1)`, the minimum target is `decimal(12,3)`. The minimum target for `byte`, `short`, and `int` types is `decimal(10,0)`. The minimum target for `long` is `decimal(20,0)`. Type changes are supported for top-level columns as well as fields nested inside structs, maps and arrays. ## How to enable Delta Lake type widening [Section titled “How to enable Delta Lake type widening”](#how-to-enable-delta-lake-type-widening) Important Enabling type widening using Delta Lake 3.3 and above sets the Delta table feature `typeWidening`, a reader/writer protocol feature. Only clients that support this table feature can read and write to the table once the table feature is set. You must use Delta Lake 3.3 or above to read and write to such Delta tables. Enabling type widening using Delta Lake 3.2 sets the Delta table feature `typeWidening-preview` on the table instead. You must use Delta Lake 3.2 or above to read and write to such Delta tables. You can enable type widening on an existing table by setting the `delta.enableTypeWidening` table property to `true`: * SQL ```sql ALTER TABLE SET TBLPROPERTIES ('delta.enableTypeWidening' = 'true') ``` Alternatively, you can enable type widening during table creation: * SQL ```sql CREATE TABLE USING DELTA TBLPROPERTIES('delta.enableTypeWidening' = 'true') ``` To disable type widening: * SQL ```sql ALTER TABLE SET TBLPROPERTIES ('delta.enableTypeWidening' = 'false') ``` Disabling type widening prevents future type changes from being applied to the table. It doesn’t affect type changes previously applied and in particular, it doesn’t remove the type widening table feature and doesn’t allow clients that don’t support the type widening table feature to read and write to the table. To remove the type widening table feature from the table and allow other clients that don’t support this feature to read and write to the table, see [Removing the type widening table feature](#removing-the-type-widening-table-feature). ## Manually applying a type change [Section titled “Manually applying a type change”](#manually-applying-a-type-change) When type widening is enabled on a Delta table, you can change the type of a column using the `ALTER COLUMN` command: * SQL ```sql ALTER TABLE ALTER COLUMN TYPE ``` The table schema is updated without rewriting the underlying Parquet files. ## Type changes with automatic schema evolution [Section titled “Type changes with automatic schema evolution”](#type-changes-with-automatic-schema-evolution) Schema evolution works with type widening to update data types in target tables to match the type of incoming data. Note Without type widening enabled, schema evolution always attempts to downcast data to match column types in the target table. If you don’t want to automatically widen data types in your target tables, disable type widening before you run workloads with schema evolution enabled. To use schema evolution to widen the data type of a column during ingestion, you must meet the following conditions: * The write command runs with automatic schema evolution enabled. * The target table has type widening enabled. * The source column type is wider than the target column type. * Type widening supports the type change. * The type change is not one of `byte`, `short`, `int`, or `long` to `decimal` or `double`. These type changes can only be applied manually using ALTER TABLE to avoid accidental promotion of integers to decimals. Type mismatches that don’t meet all of these conditions follow normal schema enforcement rules. ## Removing the type widening table feature [Section titled “Removing the type widening table feature”](#removing-the-type-widening-table-feature) The type widening feature can be removed from a Delta table using the `DROP FEATURE` command: * SQL ```sql ALTER TABLE DROP FEATURE 'typeWidening' [TRUNCATE HISTORY] ``` Note Tables that enabled type widening using Delta Lake 3.2 require dropping feature `typeWidening-preview` instead. See [Drop Delta table features](/delta-drop-feature/) for more information on dropping Delta table features. When dropping the type widening feature, the underlying Parquet files are rewritten when necessary to ensure that the column types in the files match the column types in the Delta table schema. After the type widening feature is removed from the table, Delta clients that don’t support the feature can read and write to the table. ## Limitations [Section titled “Limitations”](#limitations) ### Iceberg Compatibility [Section titled “Iceberg Compatibility”](#iceberg-compatibility) Iceberg doesn’t support all type changes covered by type widening, see [Iceberg Schema Evolution](https://iceberg.apache.org/spec/#schema-evolution). In particular, Iceberg V2 does not support the following type changes: * `byte`, `short`, `int`, `long` to `decimal` or `double` * decimal scale increase * `date` to `timestampNTZ` When [UniForm with Iceberg compatibility](/delta-uniform) is enabled on a Delta table, applying one of these type changes results in an error. If you apply one of these unsupported type changes to a Delta table, enabling [Uniform with Iceberg compatibility](/delta-uniform) on the table results in an error. To resolve the error, you must [drop the type widening table feature](#removing-the-type-widening-table-feature). # Universal Format (UniForm) > Configure Delta tables to be read as Iceberg/Hudi tables using UniForm. Delta Universal Format (UniForm) allows you to read Delta tables with Iceberg and Hudi clients. UniForm takes advantage of the fact that Delta Lake, Iceberg, and Hudi all consist of Parquet data files and a metadata layer. UniForm automatically generates Iceberg metadata asynchronously, allowing Iceberg clients to read Delta tables as if they were Iceberg or Hudi tables. You can expect negligible Delta write overhead when UniForm is enabled, as the metadata conversion and transaction occurs asynchronously after the Delta commit. A single copy of the data files provides access to clients of all formats. ## Requirements [Section titled “Requirements”](#requirements) To enable UniForm, you must fulfill the following requirements: ### Uniform Iceberg [Section titled “Uniform Iceberg”](#uniform-iceberg) * The table must have column mapping enabled. See [Delta column mapping](/delta-column-mapping). * The Delta table must have a `minReaderVersion` >= 2 and `minWriterVersion` >= 7. * Writes to the table must use Delta Lake 3.1 or above. * Hive Metastore (HMS) must be configured as the catalog. See [the HMS documentation](https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html) for how to configure Apache Spark to use Hive Metastore. ### Uniform Hudi (preview) [Section titled “Uniform Hudi (preview)”](#uniform-hudi-preview) * Writes to the table must use Delta Lake 3.2 or above. ## Enable Delta Lake UniForm [Section titled “Enable Delta Lake UniForm”](#enable-delta-lake-uniform) Important Enabling Delta UniForm Iceberg requires the Delta table feature `IcebergCompatV2`, a write protocol feature. Only clients that support this table feature can write to enabled tables. You must use Delta Lake 3.1 or above to write to Delta tables with this feature enabled. Enabling Delta UniForm Iceberg requires “delta-iceberg” to be provided to Spark shell: ```plaintext --packages io.delta:io.delta:delta-iceberg_2.12: ``` Enabling Delta UniForm Hudi requires “delta-hudi” to be provided to Spark shell: ```plaintext --packages io.delta:io.delta:delta-hudi_2.12: ``` The following table properties enable UniForm support for Iceberg. ```plaintext 'delta.enableIcebergCompatV2' = 'true' 'delta.universalFormat.enabledFormats' = 'iceberg' ``` The following table properties enable UniForm support for Hudi. ```plaintext 'delta.universalFormat.enabledFormats' = 'hudi' ``` The following table properties enable UniForm support for both. ```plaintext 'delta.enableIcebergCompatV2' = 'true' 'delta.universalFormat.enabledFormats' = 'iceberg,hudi' ``` You must also enable column mapping to use UniForm. It is set automatically during table creation, as in the following example: * SQL ```sql CREATE TABLE T(c1 INT) USING DELTA TBLPROPERTIES( 'delta.enableIcebergCompatV2' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg'); ``` In Delta 3.3 and above, you can enable or upgrade UniForm Iceberg on an existing table using the following syntax: * SQL ```sql ALTER TABLE table_name SET TBLPROPERTIES( 'delta.enableIcebergCompatV2' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg'); ``` You can also use REORG to enable UniForm Iceberg and rewrite underlying data files, as in the following example: * SQL ```sql REORG TABLE table_name APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2)); ``` Use REORG if any of following are true: * Your table has deletion vectors enabled. * You previously enabled the IcebergCompatV1 version of UniForm Iceberg. * You need to read from Iceberg engines that don’t support Hive-style Parquet files, such as Athena or Redshift. You can enable UniForm Hudi on an existing table using the following syntax: * SQL ```sql ALTER TABLE table_name SET TBLPROPERTIES ('delta.universalFormat.enabledFormats' = 'hudi'); ``` Note This syntax requires [Delta column mapping](/delta-column-mapping) to be enabled on the table prior to running on Delta 3.1. This syntax also works to upgrade from the IcbergCompatV1. It may rewrite existing files to make those Iceberg compatible, and it automatically disables and purges Deletion Vectors from the table. Important When you first enable UniForm, asynchronous metadata generation begins. This task must complete before external clients can query the table using Iceberg or Hudi. See [Check Iceberg/Hudi metadata generation status](#check-iceberghudi-metadata-generation-status). Caution You can turn off UniForm by unsetting the `delta.universalFormat.enabledFormats` table property. You cannot turn off column mapping once enabled, and upgrades to Delta Lake reader and writer protocol versions cannot be undone. See [Limitations](#limitations). ## When does UniForm generate metadata? [Section titled “When does UniForm generate metadata?”](#when-does-uniform-generate-metadata) Delta Lake triggers Iceberg/Hudi metadata generation asynchronously after a Delta Lake write transaction completes using the same compute that completed the Delta transaction. Iceberg/Hudi can have significantly higher write latencies than Delta Lake. Delta tables with frequent commits might bundle multiple Delta commits into a single Iceberg/Hudi commit. Delta Lake ensures that only one metadata generation process per format is in progress at any time in a single cluster. Commits that would trigger a second concurrent metadata generation process successfully commit to Delta, but do not trigger asynchronous metadata generation. This prevents cascading latency for metadata generation for workloads with frequent commits (seconds to minutes between commits). ## Check Iceberg/Hudi metadata generation status [Section titled “Check Iceberg/Hudi metadata generation status”](#check-iceberghudi-metadata-generation-status) UniForm adds the following properties to Iceberg/Hudi table metadata to track metadata generation status: | Table property | Description | | --------------------------- | --------------------------------------------------------------------------------------- | | `converted_delta_version` | The latest version of the Delta table for which metadata was successfully generated. | | `converted_delta_timestamp` | The timestamp of the latest Delta commit for which metadata was successfully generated. | See documentation for your Iceberg/Hudi reader client for how to review table properties outside Delta Lake. For Apache Spark, you can see these properties using the following syntax: * SQL ```sql SHOW TBLPROPERTIES ; ``` ## Read UniForm tables as Iceberg tables in Apache Spark [Section titled “Read UniForm tables as Iceberg tables in Apache Spark”](#read-uniform-tables-as-iceberg-tables-in-apache-spark) You are able to read UniForm tables as Iceberg tables in Apache Spark with the following steps: * Start Apache Spark with Iceberg, and connect to the Hive Metastore used by UniForm. Please refer to the [Iceberg documentation](https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs) for how to run Iceberg with Apache Spark and connect to a Hive Metastore. * Use the `SHOW TABLES` command to see a list of available Iceberg tables in the catalog. * Read an Iceberg table using standard SQL such as `SELECT`. ## Read UniForm tables as Iceberg tables using a metadata JSON path [Section titled “Read UniForm tables as Iceberg tables using a metadata JSON path”](#read-uniform-tables-as-iceberg-tables-using-a-metadata-json-path) Some Iceberg clients allow you to register external Iceberg tables by providing a path to versioned metadata files. Each time UniForm converts a new version of the Delta table to Iceberg, it creates a new metadata JSON file. Clients that use metadata JSON paths for configuring Iceberg include BigQuery. Refer to documentation for the Iceberg reader client for configuration details. Delta Lake stores Iceberg metadata under the table directory, using the following pattern: ```ini /metadata/v-uuid.metadata.json ``` ## Read UniForm tables as Hudi tables in Apache Spark [Section titled “Read UniForm tables as Hudi tables in Apache Spark”](#read-uniform-tables-as-hudi-tables-in-apache-spark) You are able to read UniForm tables as Hudi tables in Apache Spark with the following steps: * See [Hudi documentation](https://hudi.apache.org/docs/quick-start-guide#spark-shellsql) for how to run Hudi on Apache Spark - Scala ```scala spark.read.format("hudi") .option("hoodie.metadata.enable", "true") .load("PATH_TO_UNIFORM_TABLE_DIRECTORY") ``` ## Delta and Iceberg/Hudi table versions [Section titled “Delta and Iceberg/Hudi table versions”](#delta-and-iceberghudi-table-versions) All Delta Lake, Iceberg and Hudi allow time travel queries using table versions or timestamps stored in table metadata. Delta and Iceberg table versions do not align by either the commit timestamp or the version ID. However, Delta and Hudi commit timestamp align, but version ID does not. If you wish to verify which version of a Delta table a given version of an Iceberg/Hudi table corresponds to, you can use the corresponding table properties set on the Iceberg/Hudi table. See [Check Iceberg/Hudi metadata generation status](#check-iceberghudi-metadata-generation-status). ## Limitations [Section titled “Limitations”](#limitations) Caution UniForm is read-only from an Iceberg and Hudi perspective. This, however, cannot be enforced as for Iceberg, UniForm uses HMS as an Iceberg catalog and for Hudi, metadata is stored on the file system. If any external writer (not Delta Lake) writes to this Iceberg/Hudi table, this may destroy your Delta table and cause data loss, as the Iceberg/Hudi writer may perform data cleanup or garbage collection that Delta is unaware of. The following limitations exist: * UniForm does not work on tables with deletion vectors enabled. See [What are deletion vectors?](/delta-deletion-vectors). * Delta tables with UniForm enabled do not support `VOID` type. * Iceberg/Hudi clients can only read from UniForm. Writes are not supported. * Iceberg/Hudi reader clients might have individual limitations, regardless of UniForm. See documentation for your target client. The following Delta Lake features work for Delta clients when UniForm is enabled, but do not have support in Iceberg: * Change Data Feed * Delta Sharing # Table deletes, updates, and merges > Learn how to delete data from and update data in Delta tables. Delta Lake supports several statements to facilitate deleting data from and updating data in Delta tables. ## Delete from a table [Section titled “Delete from a table”](#delete-from-a-table) You can remove data that matches a predicate from a Delta table. For instance, in a table named `people10m` or a path at `/tmp/delta/people-10m`, to delete all rows corresponding to people with a value in the `birthDate` column from before `1955`, you can run the following: * SQL ```sql DELETE FROM people10m WHERE birthDate < '1955-01-01' DELETE FROM delta.`/tmp/delta/people-10m` WHERE birthDate < '1955-01-01' ``` See [Configure SparkSession](/delta-batch#configure-sparksession) for the steps to enable support for SQL commands. * Python ```python from delta.tables import * from pyspark.sql.functions import * deltaTable = DeltaTable.forPath(spark, '/tmp/delta/people-10m') # Declare the predicate by using a SQL-formatted string. deltaTable.delete("birthDate < '1955-01-01'") # Declare the predicate by using Spark SQL functions. deltaTable.delete(col('birthDate') < '1960-01-01') ``` * Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, "/tmp/delta/people-10m") // Declare the predicate by using a SQL-formatted string. deltaTable.delete("birthDate < '1955-01-01'") import org.apache.spark.sql.functions._ import spark.implicits._ // Declare the predicate by using Spark SQL functions and implicits. deltaTable.delete(col("birthDate") < "1955-01-01") ``` * Java ```java import io.delta.tables.*; import org.apache.spark.sql.functions; DeltaTable deltaTable = DeltaTable.forPath(spark, "/tmp/delta/people-10m"); // Declare the predicate by using a SQL-formatted string. deltaTable.delete("birthDate < '1955-01-01'"); // Declare the predicate by using Spark SQL functions. deltaTable.delete(functions.col("birthDate").lt(functions.lit("1955-01-01"))); ``` See the [Delta Lake APIs](/delta-apidoc/) for details. Important `delete` removes the data from the latest version of the Delta table but does not remove it from the physical storage until the old versions are explicitly vacuumed. See [vacuum](/delta-utility/#remove-files-no-longer-referenced-by-a-delta-table) for details. Tip When possible, provide predicates on the partition columns for a partitioned Delta table as such predicates can significantly speed up the operation. ## Update a table [Section titled “Update a table”](#update-a-table) You can update data that matches a predicate in a Delta table. For example, in a table named `people10m` or a path at `/tmp/delta/people-10m`, to change an abbreviation in the `gender` column from `M` or `F` to `Male` or `Female`, you can run the following: * SQL ```sql UPDATE people10m SET gender = 'Female' WHERE gender = 'F'; UPDATE people10m SET gender = 'Male' WHERE gender = 'M'; UPDATE delta.`/tmp/delta/people-10m` SET gender = 'Female' WHERE gender = 'F'; UPDATE delta.`/tmp/delta/people-10m` SET gender = 'Male' WHERE gender = 'M'; ``` See [Configure SparkSession](/delta-batch#configure-sparksession) for the steps to enable support for SQL commands. * Python ```python from delta.tables import * from pyspark.sql.functions import * deltaTable = DeltaTable.forPath(spark, '/tmp/delta/people-10m') # Declare the predicate by using a SQL-formatted string. deltaTable.update( condition = "gender = 'F'", set = { "gender": "'Female'" } ) # Declare the predicate by using Spark SQL functions. deltaTable.update( condition = col('gender') == 'M', set = { 'gender': lit('Male') } ) ``` * Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, "/tmp/delta/people-10m") // Declare the predicate by using a SQL-formatted string. deltaTable.updateExpr( "gender = 'F'", Map("gender" -> "'Female'") import org.apache.spark.sql.functions._ import spark.implicits._ // Declare the predicate by using Spark SQL functions and implicits. deltaTable.update( col("gender") === "M", Map("gender" -> lit("Male"))); ``` * Java ```java import io.delta.tables.*; import org.apache.spark.sql.functions; import java.util.HashMap; DeltaTable deltaTable = DeltaTable.forPath(spark, "/data/events/"); // Declare the predicate by using a SQL-formatted string. deltaTable.updateExpr( "gender = 'F'", new HashMap() {{ put("gender", "'Female'"); }} ); // Declare the predicate by using Spark SQL functions. deltaTable.update( functions.col(gender).eq("M"), new HashMap() {{ put("gender", functions.lit("Male")); }} ); ``` See the [Delta Lake APIs](/delta-apidoc/) for details. Tip Similar to delete, update operations can get a significant speedup with predicates on partitions. ## Upsert into a table using merge [Section titled “Upsert into a table using merge”](#upsert-into-a-table-using-merge) You can upsert data from a source table, view, or DataFrame into a target Delta table by using the `MERGE` SQL operation. Delta Lake supports inserts, updates and deletes in `MERGE`, and it supports extended syntax beyond the SQL standards to facilitate advanced use cases. Suppose you have a source table named `people10mupdates` or a source path at `/tmp/delta/people-10m-updates` that contains new data for a target table named `people10m` or a target path at `/tmp/delta/people-10m`. Some of these new records may already be present in the target data. To merge the new data, you want to update rows where the person’s `id` is already present and insert the new rows where no matching `id` is present. You can run the following: * SQL ```sql MERGE INTO people10m USING people10mupdates ON people10m.id = people10mupdates.id WHEN MATCHED THEN UPDATE SET id = people10mupdates.id, firstName = people10mupdates.firstName, middleName = people10mupdates.middleName, lastName = people10mupdates.lastName, gender = people10mupdates.gender, birthDate = people10mupdates.birthDate, ssn = people10mupdates.ssn, salary = people10mupdates.salary WHEN NOT MATCHED THEN INSERT ( id, firstName, middleName, lastName, gender, birthDate, ssn, salary ) VALUES ( people10mupdates.id, people10mupdates.firstName, people10mupdates.middleName, people10mupdates.lastName, people10mupdates.gender, people10mupdates.birthDate, people10mupdates.ssn, people10mupdates.salary ) ``` See [Configure SparkSession](/delta-batch/#configure-sparksession) for the steps to enable support for SQL commands. * Python ```python from delta.tables import * deltaTablePeople = DeltaTable.forPath(spark, '/tmp/delta/people-10m') deltaTablePeopleUpdates = DeltaTable.forPath(spark, '/tmp/delta/people-10m-updates') dfUpdates = deltaTablePeopleUpdates.toDF() deltaTablePeople.alias('people') \ .merge( dfUpdates.alias('updates'), 'people.id = updates.id' ) \ .whenMatchedUpdate(set = { "id": "updates.id", "firstName": "updates.firstName", "middleName": "updates.middleName", "lastName": "updates.lastName", "gender": "updates.gender", "birthDate": "updates.birthDate", "ssn": "updates.ssn", "salary": "updates.salary" } ) \ .whenNotMatchedInsert(values = { "id": "updates.id", "firstName": "updates.firstName", "middleName": "updates.middleName", "lastName": "updates.lastName", "gender": "updates.gender", "birthDate": "updates.birthDate", "ssn": "updates.ssn", "salary": "updates.salary" } ) \ .execute() ``` * Scala ```scala import io.delta.tables._ import org.apache.spark.sql.functions._ val deltaTablePeople = DeltaTable.forPath(spark, "/tmp/delta/people-10m") val deltaTablePeopleUpdates = DeltaTable.forPath(spark, "tmp/delta/people-10m-updates") val dfUpdates = deltaTablePeopleUpdates.toDF() deltaTablePeople .as("people") .merge( dfUpdates.as("updates"), "people.id = updates.id") .whenMatched .updateExpr( Map( "id" -> "updates.id", "firstName" -> "updates.firstName", "middleName" -> "updates.middleName", "lastName" -> "updates.lastName", "gender" -> "updates.gender", "birthDate" -> "updates.birthDate", "ssn" -> "updates.ssn", "salary" -> "updates.salary" )) .whenNotMatched .insertExpr( Map( "id" -> "updates.id", "firstName" -> "updates.firstName", "middleName" -> "updates.middleName", "lastName" -> "updates.lastName", "gender" -> "updates.gender", "birthDate" -> "updates.birthDate", "ssn" -> "updates.ssn", "salary" -> "updates.salary" )) .execute() ``` * Java ```java import io.delta.tables.*; import org.apache.spark.sql.functions; import java.util.HashMap; DeltaTable deltaTable = DeltaTable.forPath(spark, "/tmp/delta/people-10m") Dataset dfUpdates = spark.read("delta").load("/tmp/delta/people-10m-updates") deltaTable .as("people") .merge( dfUpdates.as("updates"), "people.id = updates.id") .whenMatched() .updateExpr( new HashMap() {{ put("id", "updates.id"); put("firstName", "updates.firstName"); put("middleName", "updates.middleName"); put("lastName", "updates.lastName"); put("gender", "updates.gender"); put("birthDate", "updates.birthDate"); put("ssn", "updates.ssn"); put("salary", "updates.salary"); }}) .whenNotMatched() .insertExpr( new HashMap() {{ put("id", "updates.id"); put("firstName", "updates.firstName"); put("middleName", "updates.middleName"); put("lastName", "updates.lastName"); put("gender", "updates.gender"); put("birthDate", "updates.birthDate"); put("ssn", "updates.ssn"); put("salary", "updates.salary"); }}) .execute(); ``` See the [Delta Lake APIs](/delta-apidoc/) for Scala, Java, and Python syntax details. Important Delta Lake merge operations typically require two passes over the source data. If your source data contains nondeterministic expressions, multiple passes on the source data can produce different rows causing incorrect results. Some common examples of nondeterministic expressions include the `current_date` and `current_timestamp` functions. In Delta Lake 2.2 and above this issue is solved by automatically materializing the source data as part of the merge command, so that the source data is deterministic in multiple passes. In Delta Lake 2.1 and below if you cannot avoid using non-deterministic functions, consider saving the source data to storage, for example as a temporary Delta table. Caching the source data may not address this issue, as cache invalidation can cause the source data to be recomputed partially or completely (for example when a cluster loses some of it executors when scaling down). ### Modify all unmatched rows using merge [Section titled “Modify all unmatched rows using merge”](#modify-all-unmatched-rows-using-merge) Note `WHEN NOT MATCHED BY SOURCE` clauses are supported by the Scala, Python and Java [Delta Lake APIs](/delta-apidoc/) in Delta 2.3 and above. SQL is supported in Delta 2.4 and above. You can use the `WHEN NOT MATCHED BY SOURCE` clause to `UPDATE` or `DELETE` records in the target table that do not have corresponding records in the source table. We recommend adding an optional conditional clause to avoid fully rewriting the target table. The following code example shows the basic syntax of using this for deletes, overwriting the target table with the contents of the source table and deleting unmatched records in the target table. * SQL ```sql MERGE INTO target USING source ON source.key = target.key WHEN MATCHED UPDATE SET * WHEN NOT MATCHED INSERT * WHEN NOT MATCHED BY SOURCE DELETE ``` * Python ```python (targetDF .merge(sourceDF, "source.key = target.key") .whenMatchedUpdateAll() .whenNotMatchedInsertAll() .whenNotMatchedBySourceDelete() .execute() ) ``` * Scala ```scala targetDF .merge(sourceDF, "source.key = target.key") .whenMatched() .updateAll() .whenNotMatched() .insertAll() .whenNotMatchedBySource() .delete() .execute() ``` The following example adds conditions to the `WHEN NOT MATCHED BY SOURCE` clause and specifies values to update in unmatched target rows. * SQL ```sql MERGE INTO target USING source ON source.key = target.key WHEN MATCHED THEN UPDATE SET target.lastSeen = source.timestamp WHEN NOT MATCHED THEN INSERT (key, lastSeen, status) VALUES (source.key, source.timestamp, 'active') WHEN NOT MATCHED BY SOURCE AND target.lastSeen >= (current_date() - INTERVAL '5' DAY) THEN UPDATE SET target.status = 'inactive' ``` * Python ```python (targetDF .merge(sourceDF, "source.key = target.key") .whenMatchedUpdate( set = {"target.lastSeen": "source.timestamp"} ) .whenNotMatchedInsert( values = { "target.key": "source.key", "target.lastSeen": "source.timestamp", "target.status": "'active'" } ) .whenNotMatchedBySourceUpdate( condition="target.lastSeen >= (current_date() - INTERVAL '5' DAY)", set = {"target.status": "'inactive'"} ) .execute() ) ``` * Scala ```scala targetDF .merge(sourceDF, "source.key = target.key") .whenMatched() .updateExpr(Map("target.lastSeen" -> "source.timestamp")) .whenNotMatched() .insertExpr(Map( "target.key" -> "source.key", "target.lastSeen" -> "source.timestamp", "target.status" -> "'active'", ) ) .whenNotMatchedBySource("target.lastSeen >= (current_date() - INTERVAL '5' DAY)") .updateExpr(Map("target.status" -> "'inactive'")) .execute() ``` ### Operation semantics [Section titled “Operation semantics”](#operation-semantics) Here is a detailed description of the `merge` programmatic operation. * There can be any number of `whenMatched` and `whenNotMatched` clauses. * `whenMatched` clauses are executed when a source row matches a target table row based on the match condition. These clauses have the following semantics. * `whenMatched` clauses can have at most one `update` and one `delete` action. The `update` action in `merge` only updates the specified columns (similar to the `update` [operation](/delta-update/#update-a-table)) of the matched target row. The `delete` action deletes the matched row. * Each `whenMatched` clause can have an optional condition. If this clause condition exists, the `update` or `delete` action is executed for any matching source-target row pair only when the clause condition is true. * If there are multiple `whenMatched` clauses, then they are evaluated in the order they are specified. All `whenMatched` clauses, except the last one, must have conditions. * If none of the `whenMatched` conditions evaluate to true for a source and target row pair that matches the merge condition, then the target row is left unchanged. * To update all the columns of the target Delta table with the corresponding columns of the source dataset, use `whenMatched(...).updateAll()`. This is equivalent to: * Scala ```scala whenMatched(...).updateExpr(Map("col1" -> "source.col1", "col2" -> "source.col2", ...)) ``` for all the columns of the target Delta table. Therefore, this action assumes that the source table has the same columns as those in the target table, otherwise the query throws an analysis error. Note This behavior changes when automatic schema migration is enabled. See [Automatic schema evolution](/delta-update/#automatic-schema-evolution) for details. * `whenNotMatched` clauses are executed when a source row does not match any target row based on the match condition. These clauses have the following semantics. * `whenNotMatched` clauses can have only the `insert` action. The new row is generated based on the specified column and corresponding expressions. You do not need to specify all the columns in the target table. For unspecified target columns, `NULL` is inserted. * Each `whenNotMatched` clause can have an optional condition. If the clause condition is present, a source row is inserted only if that condition is true for that row. Otherwise, the source column is ignored. * If there are multiple `whenNotMatched` clauses, then they are evaluated in the order they are specified. All `whenNotMatched` clauses, except the last one, must have conditions. * To insert all the columns of the target Delta table with the corresponding columns of the source dataset, use `whenNotMatched(...).insertAll()`. This is equivalent to: * Scala ```scala whenNotMatched(...).insertExpr(Map("col1" -> "source.col1", "col2" -> "source.col2", ...)) ``` for all the columns of the target Delta table. Therefore, this action assumes that the source table has the same columns as those in the target table, otherwise the query throws an analysis error. Note This behavior changes when automatic schema migration is enabled. See [Automatic schema evolution](/delta-update/#automatic-schema-evolution) for details. * `whenNotMatchedBySource` clauses are executed when a target row does not match any source row based on the merge condition. These clauses have the following semantics. * `whenNotMatchedBySource` clauses can specify `delete` and `update` actions. * Each `whenNotMatchedBySource` clause can have an optional condition. If the clause condition is present, a target row is modified only if that condition is true for that row. Otherwise, the target row is left unchanged. * If there are multiple `whenNotMatchedBySource` clauses, then they are evaluated in the order they are specified. All `whenNotMatchedBySource` clauses, except the last one, must have conditions. * By definition, `whenNotMatchedBySource` clauses do not have a source row to pull column values from, and so source columns can’t be referenced. For each column to be modified, you can either specify a literal or perform an action on the target column, such as `SET target.deleted_count = target.deleted_count + 1`. Important! * A `merge` operation can fail if multiple rows of the source dataset match and the merge attempts to update the same rows of the target Delta table. According to the SQL semantics of merge, such an update operation is ambiguous as it is unclear which source row should be used to update the matched target row. You can preprocess the source table to eliminate the possibility of multiple matches. See the [change data capture example](/delta-update/#write-change-data-into-a-delta-table)—it shows how to preprocess the change dataset (that is, the source dataset) to retain only the latest change for each key before applying that change into the target Delta table. * You can apply a SQL `MERGE` operation on a SQL VIEW only if the view has been defined as `CREATE VIEW viewName AS SELECT * FROM deltaTable`. ### Schema validation [Section titled “Schema validation”](#schema-validation) `merge` automatically validates that the schema of the data generated by insert and update expressions are compatible with the schema of the table. It uses the following rules to determine whether the `merge` operation is compatible: * For `update` and `insert` actions, the specified target columns must exist in the target Delta table. * For `updateAll` and `insertAll` actions, the source dataset must have all the columns of the target Delta table. The source dataset can have extra columns and they are ignored. If you do not want the extra columns to be ignored and instead want to update the target table schema to include new columns, see [Automatic schema evolution](/delta-update/#automatic-schema-evolution). * For all actions, if the data type generated by the expressions producing the target columns are different from the corresponding columns in the target Delta table, `merge` tries to cast them to the types in the table. ### Automatic schema evolution [Section titled “Automatic schema evolution”](#automatic-schema-evolution) Schema evolution allows users to resolve schema mismatches between the target and source table in merge. It handles the following two cases: 1. A column in the source table is not present in the target table. The new column is added to the target schema, and its values are inserted or updated using the source values. 2. A column in the target table is not present in the source table. The target schema is left unchanged; the values in the additional target column are either left unchanged (for `UPDATE`) or set to `NULL` (for `INSERT`). Caution To use schema evolution, you must set the Spark session configuration`spark.databricks.delta.schema.autoMerge.enabled` to `true` before you run the `merge` command. Note In Delta 2.3 and above, columns present in the source table can be specified by name in insert or update actions. In Delta 2.2 and below, only `INSERT *` or `UPDATE SET *` actions can be used for schema evolution with merge. Here are a few examples of the effects of `merge` operation with and without schema evolution. | Columns | Query (in SQL) | Behavior without schema evolution (default) | Behavior with schema evolution | | :--------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Target: `key, value` Source: `key, value, new_value` | `sql MERGE INTO target_table t USING source_table s ON t.key = s.key WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *` | The table schema remains unchanged; only columns `key`, `value` are updated/inserted. | The table schema is changed to `(key, value, new_value)`. Existing records with matches are updated with the `value` and `new_value` in the source. New rows are inserted with the schema `(key, value, new_value)`. | | Target: `key, old_value` Source: `key, new_value` | `sql MERGE INTO target_table t USING source_table s ON t.key = s.key WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *` | `UPDATE` and `INSERT` actions throw an error because the target column `old_value` is not in the source. | The table schema is changed to `(key, old_value, new_value)`. Existing records with matches are updated with the `new_value` in the source leaving `old_value` unchanged. New records are inserted with the specified `key`, `new_value`, and `NULL` for the `old_value`. | | Target: `key, old_value` Source: `key, new_value` | `sql MERGE INTO target_table t USING source_table s ON t.key = s.key WHEN MATCHED THEN UPDATE SET new_value = s.new_value` | `UPDATE` throws an error because column `new_value` does not exist in the target table. | The table schema is changed to `(key, old_value, new_value)`. Existing records with matches are updated with the `new_value` in the source leaving `old_value` unchanged, and unmatched records have `NULL` entered for `new_value`. | | Target: `key, old_value` Source: `key, new_value` | `sql MERGE INTO target_table t USING source_table s ON t.key = s.key WHEN NOT MATCHED THEN INSERT (key, new_value) VALUES (s.key, s.new_value)` | `INSERT`throws an error because column`new_value`does not exist in the target table. | The table schema is changed to`(key, old_value, new_value)`. New records are inserted with the specified `key`, `new_value`, and `NULL`for the`old_value`. Existing records have `NULL`entered for`new_value`leaving`old_value` unchanged. See note (1). | ## Special considerations for schemas that contain arrays of structs [Section titled “Special considerations for schemas that contain arrays of structs”](#special-considerations-for-schemas-that-contain-arrays-of-structs) Delta `MERGE INTO` supports resolving struct fields by name and evolving schemas for arrays of structs. With schema evolution enabled, target table schemas will evolve for arrays of structs, which also works with any nested structs inside of arrays. Note In Delta 2.3 and above, struct fields present in the source table can be specified by name in insert or update commands. In Delta 2.2 and below, only `INSERT *` or `UPDATE SET *` commands can be used for schema evolution with merge. Here are a few examples of the effects of merge operations with and without schema evolution for arrays of structs. | Source schema | Target schema | Behavior without schema evolution (default) | Behavior with schema evolution | | :----------------------------------------------------------- | :------------------------------------------------ | :-------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | array\> | array\> | The table schema remains unchanged. Columns will be resolved by name and updated or inserted. | The table schema remains unchanged. Columns will be resolved by name and updated or inserted. | | array\> | array\> | `update` and `insert` throw errors because `c` and `d` do not exist in the target table. | The table schema is changed to array\>. `c` and `d` are inserted as `NULL` for existing entries in the target table. `update` and `insert` fill entries in the source table with `a` casted to string and `b` as `NULL`. | | array\>> | array\>> | `update` and `insert` throw errors because `d` does not exist in the target table. | The target table schema is changed to array\>>. `d` is inserted as `NULL` for existing entries in the target table. | ### Performance tuning [Section titled “Performance tuning”](#performance-tuning) You can reduce the time taken by merge using the following approaches: * **Reduce the search space for matches**: By default, the `merge` operation searches the entire Delta table to find matches in the source table. One way to speed up `merge` is to reduce the search space by adding known constraints in the match condition. For example, suppose you have a table that is partitioned by `country` and `date` and you want to use `merge` to update information for the last day and a specific country. Adding the condition * SQL ```sql events.date = current_date() AND events.country = 'USA' ``` will make the query faster as it looks for matches only in the relevant partitions. Furthermore, it will also reduce the chances of conflicts with other concurrent operations. See [Concurrency control](/concurrency-control/) for more details. * **Compact files**: If the data is stored in many small files, reading the data to search for matches can become slow. You can compact small files into larger files to improve read throughput. See [Compact files](/best-practices/#compact-files) for details. * **Control the shuffle partitions for writes**: The `merge` operation shuffles data multiple times to compute and write the updated data. The number of tasks used to shuffle is controlled by the Spark session configuration `spark.sql.shuffle.partitions`. Setting this parameter not only controls the parallelism but also determines the number of output files. Increasing the value increases parallelism but also generates a larger number of smaller data files. * **Repartition output data before write**: For partitioned tables, `merge` can produce a much larger number of small files than the number of shuffle partitions. This is because every shuffle task can write multiple files in multiple partitions, and can become a performance bottleneck. In many cases, it helps to repartition the output data by the table’s partition columns before writing it. You enable this by setting the Spark session configuration `spark.databricks.delta.merge.repartitionBeforeWrite.enabled` to `true`. ## Merge examples [Section titled “Merge examples”](#merge-examples) Here are a few examples on how to use `merge` in different scenarios. ### Data deduplication when writing into Delta tables [Section titled “Data deduplication when writing into Delta tables”](#data-deduplication-when-writing-into-delta-tables) A common ETL use case is to collect logs into Delta table by appending them to a table. However, often the sources can generate duplicate log records and downstream deduplication steps are needed to take care of them. With `merge`, you can avoid inserting the duplicate records. * SQL ```sql MERGE INTO logs USING newDedupedLogs ON logs.uniqueId = newDedupedLogs.uniqueId AND logs.date > current_date() - INTERVAL 7 DAYS WHEN NOT MATCHED AND newDedupedLogs.date > current_date() - INTERVAL 7 DAYS THEN INSERT * ``` * Python ```python deltaTable.alias("logs").merge( newDedupedLogs.alias("newDedupedLogs"), "logs.uniqueId = newDedupedLogs.uniqueId") \ .whenNotMatchedInsertAll() \ .execute() ``` * Scala ```scala deltaTable .as("logs") .merge( newDedupedLogs.as("newDedupedLogs"), "logs.uniqueId = newDedupedLogs.uniqueId") .whenNotMatched() .insertAll() .execute() ``` * Java ```java deltaTable .as("logs") .merge( newDedupedLogs.as("newDedupedLogs"), "logs.uniqueId = newDedupedLogs.uniqueId") .whenNotMatched() .insertAll() .execute(); ``` Note The dataset containing the new logs needs to be deduplicated within itself. If you know that you may get duplicate records only for a few days, you can optimized your query further by partitioning the table by date, and then specifying the date range of the target table to match on. * SQL ```sql MERGE INTO logs USING newDedupedLogs ON logs.uniqueId = newDedupedLogs.uniqueId AND logs.date > current_date() - INTERVAL 7 DAYS WHEN NOT MATCHED AND newDedupedLogs.date > current_date() - INTERVAL 7 DAYS THEN INSERT * ``` * Python ```python deltaTable.alias("logs").merge( newDedupedLogs.alias("newDedupedLogs"), "logs.uniqueId = newDedupedLogs.uniqueId AND logs.date > current_date() - INTERVAL 7 DAYS") \ .whenNotMatchedInsertAll("newDedupedLogs.date > current_date() - INTERVAL 7 DAYS") \ .execute() ``` * Scala ```scala deltaTable.as("logs").merge( newDedupedLogs.as("newDedupedLogs"), "logs.uniqueId = newDedupedLogs.uniqueId AND logs.date > current_date() - INTERVAL 7 DAYS") .whenNotMatched("newDedupedLogs.date > current_date() - INTERVAL 7 DAYS") .insertAll() .execute() ``` * Java ```java deltaTable.as("logs").merge( newDedupedLogs.as("newDedupedLogs"), "logs.uniqueId = newDedupedLogs.uniqueId AND logs.date > current_date() - INTERVAL 7 DAYS") .whenNotMatched("newDedupedLogs.date > current_date() - INTERVAL 7 DAYS") .insertAll() .execute(); ``` This is more efficient than the previous command as it looks for duplicates only in the last 7 days of logs, not the entire table. Furthermore, you can use this insert-only merge with Structured Streaming to perform continuous deduplication of the logs. * In a streaming query, you can use merge operation in `foreachBatch` to continuously write any streaming data to a Delta table with deduplication. See the following [streaming example](#upsert-from-streaming-queries-using-foreachbatch) for more information on `foreachBatch`. * In another streaming query, you can continuously read deduplicated data from this Delta table. This is possible because an insert-only merge only appends new data to the Delta table. ### Slowly changing data (SCD) Type 2 operation into Delta tables [Section titled “Slowly changing data (SCD) Type 2 operation into Delta tables”](#slowly-changing-data-scd-type-2-operation-into-delta-tables) Another common operation is SCD Type 2, which maintains history of all changes made to each key in a dimensional table. Such operations require updating existing rows to mark previous values of keys as old, and the inserting the new rows as the latest values. Given a source table with updates and the target table with the dimensional data, SCD Type 2 can be expressed with `merge`. Here is a concrete example of maintaining the history of addresses for a customer along with the active date range of each address. When a customer’s address needs to be updated, you have to mark the previous address as not the current one, update its active date range, and add the new address as the current one. * Python ```python customersTable = ... # DeltaTable with schema (customerId, address, current, effectiveDate, endDate) updatesDF = ... # DataFrame with schema (customerId, address, effectiveDate) # Rows to INSERT new addresses of existing customers newAddressesToInsert = updatesDF \ .alias("updates") \ .join(customersTable.toDF().alias("customers"), "customerid") \ .where("customers.current = true AND updates.address <> customers.address") # Stage the update by unioning two sets of rows # 1. Rows that will be inserted in the whenNotMatched clause # 2. Rows that will either update the current addresses of existing customers or insert the new addresses of new customers stagedUpdates = ( newAddressesToInsert .selectExpr("NULL as mergeKey", "updates.*") # Rows for 1 .union(updatesDF.selectExpr("updates.customerId as mergeKey", "*")) # Rows for 2. ) # Apply SCD Type 2 operation using merge customersTable.alias("customers").merge( stagedUpdates.alias("staged_updates"), "customers.customerId = mergeKey") \ .whenMatchedUpdate( condition = "customers.current = true AND customers.address <> staged_updates.address", set = { # Set current to false and endDate to source's effective date. "current": "false", "endDate": "staged_updates.effectiveDate" } ).whenNotMatchedInsert( values = { "customerid": "staged_updates.customerId", "address": "staged_updates.address", "current": "true", "effectiveDate": "staged_updates.effectiveDate", # Set current to true along with the new address and its effective date. "endDate": "null" } ).execute() ``` * Scala ```scala val customersTable: DeltaTable = ... // table with schema (customerId, address, current, effectiveDate, endDate) val updatesDF: DataFrame = ... // DataFrame with schema (customerId, address, effectiveDate) // Rows to INSERT new addresses of existing customers val newAddressesToInsert = updatesDF .as("updates") .join(customersTable.toDF.as("customers"), "customerid") .where("customers.current = true AND updates.address <> customers.address") // Stage the update by unioning two sets of rows // 1. Rows that will be inserted in the whenNotMatched clause // 2. Rows that will either update the current addresses of existing customers or insert the new addresses of new customers val stagedUpdates = newAddressesToInsert .selectExpr("NULL as mergeKey", "updates.*") // Rows for 1. .union( updatesDF.selectExpr("updates.customerId as mergeKey", "*") // Rows for 2. ) // Apply SCD Type 2 operation using merge customersTable .as("customers") .merge( stagedUpdates.as("staged_updates"), "customers.customerId = mergeKey") .whenMatched("customers.current = true AND customers.address <> staged_updates.address") .updateExpr(Map( // Set current to false and endDate to source's effective date. "current" -> "false", "endDate" -> "staged_updates.effectiveDate")) .whenNotMatched() .insertExpr(Map( "customerid" -> "staged_updates.customerId", "address" -> "staged_updates.address", "current" -> "true", "effectiveDate" -> "staged_updates.effectiveDate", // Set current to true along with the new address and its effective date. "endDate" -> "null")) .execute() ``` ### Write change data into a Delta table [Section titled “Write change data into a Delta table”](#write-change-data-into-a-delta-table) Similar to SCD, another common use case, often called change data capture (CDC), is to apply all data changes generated from an external database into a Delta table. In other words, a set of updates, deletes, and inserts applied to an external table needs to be applied to a Delta table. You can do this using `merge` as follows. * Python ```python deltaTable = ... # DeltaTable with schema (key, value) # DataFrame with changes having following columns # - key: key of the change # - time: time of change for ordering between changes (can replaced by other ordering id) # - newValue: updated or inserted value if key was not deleted # - deleted: true if the key was deleted, false if the key was inserted or updated changesDF = spark.table("changes") # Find the latest change for each key based on the timestamp # Note: For nested structs, max on struct is computed as # max on first struct field, if equal fall back to second fields, and so on. latestChangeForEachKey = changesDF \ .selectExpr("key", "struct(time, newValue, deleted) as otherCols") \ .groupBy("key") \ .agg(max("otherCols").alias("latest")) \ .select("key", "latest.*") \ deltaTable.alias("t").merge( latestChangeForEachKey.alias("s"), "s.key = t.key") \ .whenMatchedDelete(condition = "s.deleted = true") \ .whenMatchedUpdate(set = { "key": "s.key", "value": "s.newValue" }) \ .whenNotMatchedInsert( condition = "s.deleted = false", values = { "key": "s.key", "value": "s.newValue" } ).execute() ``` * Scala ```scala val deltaTable: DeltaTable = ... // DeltaTable with schema (key, value) // DataFrame with changes having following columns // - key: key of the change // - time: time of change for ordering between changes (can replaced by other ordering id) // - newValue: updated or inserted value if key was not deleted // - deleted: true if the key was deleted, false if the key was inserted or updated val changesDF: DataFrame = ... // Find the latest change for each key based on the timestamp // Note: For nested structs, max on struct is computed as // max on first struct field, if equal fall back to second fields, and so on. val latestChangeForEachKey = changesDF .selectExpr("key", "struct(time, newValue, deleted) as otherCols" ) .groupBy("key") .agg(max("otherCols").as("latest")) .selectExpr("key", "latest.*") deltaTable.as("t") .merge( latestChangeForEachKey.as("s"), "s.key = t.key") .whenMatched("s.deleted = true") .delete() .whenMatched() .updateExpr(Map("key" -> "s.key", "value" -> "s.newValue")) .whenNotMatched("s.deleted = false") .insertExpr(Map("key" -> "s.key", "value" -> "s.newValue")) .execute() ``` ### Upsert from streaming queries using `foreachBatch` [Section titled “Upsert from streaming queries using foreachBatch”](#upsert-from-streaming-queries-using-foreachbatch) You can use a combination of `merge` and `foreachBatch` (see [foreachbatch](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#foreachbatch) for more information) to write complex upserts from a streaming query into a Delta table. For example: * **Write streaming aggregates in Update Mode**: This is much more efficient than Complete Mode. * Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, "/data/aggregates") # Function to upsert microBatchOutputDF into Delta table using merge def upsertToDelta(microBatchOutputDF, batchId): deltaTable.alias("t").merge( microBatchOutputDF.alias("s"), "s.key = t.key") \ .whenMatchedUpdateAll() \ .whenNotMatchedInsertAll() \ .execute() } # Write the output of a streaming aggregation query into Delta table streamingAggregatesDF.writeStream \ .format("delta") \ .foreachBatch(upsertToDelta) \ .outputMode("update") \ .start() ``` * Scala ```scala import io.delta.tables.* val deltaTable = DeltaTable.forPath(spark, "/data/aggregates") // Function to upsert microBatchOutputDF into Delta table using merge def upsertToDelta(microBatchOutputDF: DataFrame, batchId: Long) { deltaTable.as("t") .merge( microBatchOutputDF.as("s"), "s.key = t.key") .whenMatched().updateAll() .whenNotMatched().insertAll() .execute() } // Write the output of a streaming aggregation query into Delta table streamingAggregatesDF.writeStream .format("delta") .foreachBatch(upsertToDelta _) .outputMode("update") .start() ``` * **Write a stream of database changes into a Delta table**: The [merge query for writing change data](#write-change-data-into-a-delta-table) can be used in `foreachBatch` to continuously apply a stream of changes to a Delta table. * **Write a stream data into Delta table with deduplication**: The [insert-only merge query for deduplication](#data-deduplication-when-writing-into-delta-tables) can be used in `foreachBatch` to continuously write data (with duplicates) to a Delta table with automatic deduplication. Note * Make sure that your `merge` statement inside `foreachBatch` is idempotent as restarts of the streaming query can apply the operation on the same batch of data multiple times. - When `merge` is used in `foreachBatch`, the input data rate of the streaming query (reported through `StreamingQueryProgress` and visible in the notebook rate graph) may be reported as a multiple of the actual rate at which data is generated at the source. This is because `merge` reads the input data multiple times causing the input metrics to be multiplied. If this is a bottleneck, you can cache the batch DataFrame before `merge` and then uncache it after `merge`. # Table utility commands > Learn about Delta Lake utility commands. Delta tables support a number of utility commands. For many Delta Lake operations, you enable integration with Apache Spark DataSourceV2 and Catalog APIs (since 3.0) by setting configurations when you create a new `SparkSession`. See [Configure SparkSession](/delta-batch/#configure-sparksession). ## Remove files no longer referenced by a Delta table [Section titled “Remove files no longer referenced by a Delta table”](#remove-files-no-longer-referenced-by-a-delta-table) You can remove files no longer referenced by a Delta table and are older than the retention threshold by running the `vacuum` command on the table. `vacuum` is not triggered automatically. The default retention threshold for the files is 7 days. To change this behavior, see [Data retention](/delta-batch/#data-retention). Important * `vacuum` removes all files from directories not managed by Delta Lake, ignoring directories beginning with `_`. If you are storing additional metadata like Structured Streaming checkpoints within a Delta table directory, use a directory name such as `_checkpoints`. * `vacuum` deletes only data files, not log files. Log files are deleted automatically and asynchronously after checkpoint operations. The default retention period of log files is 30 days, configurable through the `delta.logRetentionDuration` property which you set with the `ALTER TABLE SET TBLPROPERTIES` SQL method. See [Table properties](/delta-batch/#table-properties). * The ability to [time travel](/delta-batch/#query-an-older-snapshot-of-a-table-time-travel) back to a version older than the retention period is lost after running `vacuum`. - SQL ```sql VACUUM eventsTable -- This runs VACUUM in ‘FULL’ mode and deletes data files outside of the retention duration and all files in the table directory not referenced by the table. VACUUM eventsTable LITE -- This VACUUM in ‘LITE’ mode runs faster. -- Instead of finding all files in the table directory, `VACUUM LITE` uses the Delta transaction log to identify and remove files no longer referenced by any table versions within the retention duration. -- If `VACUUM LITE` cannot be completed because the Delta log has been pruned a `DELTA_CANNOT_VACUUM_LITE` exception is raised. -- This mode is available only in Delta 3.3 and above. VACUUM '/data/events' -- vacuum files in path-based table VACUUM delta.`/data/events/` VACUUM delta.`/data/events/` RETAIN 100 HOURS -- vacuum files not required by versions more than 100 hours old VACUUM eventsTable DRY RUN -- do dry run to get the list of files to be deleted VACUUM eventsTable USING INVENTORY inventoryTable —- vacuum files based on a provided reservoir of files as a delta table VACUUM eventsTable USING INVENTORY (select * from inventoryTable) —- vacuum files based on a provided reservoir of files as spark SQL query ``` - Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, pathToTable) # path-based tables, or deltaTable = DeltaTable.forName(spark, tableName) # Hive metastore-based tables deltaTable.vacuum() # vacuum files not required by versions older than the default retention period deltaTable.vacuum(100) # vacuum files not required by versions more than 100 hours old ``` - Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, pathToTable) deltaTable.vacuum() // vacuum files not required by versions older than the default retention period deltaTable.vacuum(100) // vacuum files not required by versions more than 100 hours old ``` - Java ```java import io.delta.tables.*; import org.apache.spark.sql.functions; DeltaTable deltaTable = DeltaTable.forPath(spark, pathToTable); deltaTable.vacuum(); // vacuum files not required by versions older than the default retention period deltaTable.vacuum(100); // vacuum files not required by versions more than 100 hours old ``` Note When using `VACUUM`, to configure Spark to delete files in parallel (based on the number of shuffle partitions) set the session configuration `"spark.databricks.delta.vacuum.parallelDelete.enabled"` to `"true"`. See the [Delta Lake APIs](/delta-apidoc/) for Scala, Java, and Python syntax details. Caution It is recommended that you set a retention interval to be at least 7 days, because old snapshots and uncommitted files can still be in use by concurrent readers or writers to the table. If `VACUUM` cleans up active files, concurrent readers can fail or, worse, tables can be corrupted when `VACUUM` deletes files that have not yet been committed. You must choose an interval that is longer than the longest running concurrent transaction and the longest period that any stream can lag behind the most recent update to the table. Delta Lake has a safety check to prevent you from running a dangerous `VACUUM` command. If you are certain that there are no operations being performed on this table that take longer than the retention interval you plan to specify, you can turn off this safety check by setting the Spark configuration property `spark.databricks.delta.retentionDurationCheck.enabled` to `false`. ### Inventory Table [Section titled “Inventory Table”](#inventory-table) An inventory table contains a list of file paths together with their size, type (directory or not), and the last modification time. When an INVENTORY option is provided, VACUUM will consider the files listed there instead of doing the full listing of the table directory, which can be time consuming for very large tables. The inventory table can be specified as a delta table or a spark SQL query that gives the expected table schema. The schema should be as follows: | Column Name | Type | Description | | :--------------- | :------ | :-------------------------------------- | | path | string | fully qualified uri | | length | integer | size in bytes | | isDir | boolean | boolean indicating if it is a directory | | modificationTime | integer | file update time in milliseconds | ## Retrieve Delta table history [Section titled “Retrieve Delta table history”](#retrieve-delta-table-history) You can retrieve information on the operations, user, timestamp, and so on for each write to a Delta table by running the `history` command. The operations are returned in reverse chronological order. By default table history is retained for 30 days. See [Configure SparkSession](/delta-batch/#configure-sparksession) for the steps to enable support for SQL commands in Apache Spark. * SQL ```sql DESCRIBE HISTORY '/data/events/' -- get the full history of the table DESCRIBE HISTORY delta.`/data/events/` DESCRIBE HISTORY '/data/events/' LIMIT 1 -- get the last operation only DESCRIBE HISTORY eventsTable ``` * Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, pathToTable) fullHistoryDF = deltaTable.history() # get the full history of the table lastOperationDF = deltaTable.history(1) # get the last operation ``` * Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, pathToTable) val fullHistoryDF = deltaTable.history() // get the full history of the table val lastOperationDF = deltaTable.history(1) // get the last operation ``` * Java ```java import io.delta.tables.*; DeltaTable deltaTable = DeltaTable.forPath(spark, pathToTable); DataFrame fullHistoryDF = deltaTable.history(); // get the full history of the table DataFrame lastOperationDF = deltaTable.history(1); // fetch the last operation on the DeltaTable ``` See the [Delta Lake APIs](/delta-apidoc/) for Scala/Java/Python syntax details. The output of the `history` operation has the following columns. | Column | Type | Description | | :------------------ | :-------- | :------------------------------------------------------------------------- | | version | long | Table version generated by the operation. | | timestamp | timestamp | When this version was committed. | | userId | string | ID of the user that ran the operation. | | userName | string | Name of the user that ran the operation. | | operation | string | Name of the operation. | | operationParameters | map | Parameters of the operation (for example, predicates.) | | job | struct | Details of the job that ran the operation. | | notebook | struct | Details of notebook from which the operation was run. | | clusterId | string | ID of the cluster on which the operation ran. | | readVersion | long | Version of the table that was read to perform the write operation. | | isolationLevel | string | Isolation level used for this operation. | | isBlindAppend | boolean | Whether this operation appended data. | | operationMetrics | map | Metrics of the operation (for example, number of rows and files modified.) | | userMetadata | string | User-defined commit metadata if it was specified | ```plaintext +-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+ |version| timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend| operationMetrics| +-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+ | 5|2019-07-29 14:07:47| null| null| DELETE|[predicate -> ["(...|null| null| null| 4| Serializable| false|[numTotalRows -> ...| | 4|2019-07-29 14:07:41| null| null| UPDATE|[predicate -> (id...|null| null| null| 3| Serializable| false|[numTotalRows -> ...| | 3|2019-07-29 14:07:29| null| null| DELETE|[predicate -> ["(...|null| null| null| 2| Serializable| false|[numTotalRows -> ...| | 2|2019-07-29 14:06:56| null| null| UPDATE|[predicate -> (id...|null| null| null| 1| Serializable| false|[numTotalRows -> ...| | 1|2019-07-29 14:04:31| null| null| DELETE|[predicate -> ["(...|null| null| null| 0| Serializable| false|[numTotalRows -> ...| | 0|2019-07-29 14:01:40| null| null| WRITE|[mode -> ErrorIfE...|null| null| null| null| Serializable| true|[numFiles -> 2, n...| +-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+ ``` Note Some of the columns may be nulls because the corresponding information may not be available in your environment. - Columns added in the future will always be added after the last column. The `history` operation returns a collection of operations metrics in the `operationMetrics` column map. The following table lists the map key definitions by operation. | Operation | Metric name | Description | | :---------------------------------------------------------------- | :--------------------- | :----------------------------------------------------------------------------- | | WRITE, CREATE TABLE AS SELECT, REPLACE TABLE AS SELECT, COPY INTO | | | | | numFiles | Number of files written. | | | numOutputBytes | Size in bytes of the written contents. | | | numOutputRows | Number of rows written. | | STREAMING UPDATE | | | | | numAddedFiles | Number of files added. | | | numRemovedFiles | Number of files removed. | | | numOutputRows | Number of rows written. | | | numOutputBytes | Size of write in bytes. | | DELETE | | | | | numAddedFiles | Number of files added. Not provided when partitions of the table are deleted. | | | numRemovedFiles | Number of files removed. | | | numDeletedRows | Number of rows removed. Not provided when partitions of the table are deleted. | | | numCopiedRows | Number of rows copied in the process of deleting files. | | | executionTimeMs | Time taken to execute the entire operation. | | | scanTimeMs | Time taken to scan the files for matches. | | | rewriteTimeMs | Time taken to rewrite the matched files. | | TRUNCATE | | | | | numRemovedFiles | Number of files removed. | | | executionTimeMs | Time taken to execute the entire operation. | | MERGE | | | | | numSourceRows | Number of rows in the source DataFrame. | | | numTargetRowsInserted | Number of rows inserted into the target table. | | | numTargetRowsUpdated | Number of rows updated in the target table. | | | numTargetRowsDeleted | Number of rows deleted in the target table. | | | numTargetRowsCopied | Number of target rows copied. | | | numOutputRows | Total number of rows written out. | | | numTargetFilesAdded | Number of files added to the sink(target). | | | numTargetFilesRemoved | Number of files removed from the sink(target). | | | executionTimeMs | Time taken to execute the entire operation. | | | scanTimeMs | Time taken to scan the files for matches. | | | rewriteTimeMs | Time taken to rewrite the matched files. | | UPDATE | | | | | numAddedFiles | Number of files added. | | | numRemovedFiles | Number of files removed. | | | numUpdatedRows | Number of rows updated. | | | numCopiedRows | Number of rows just copied over in the process of updating files. | | | executionTimeMs | Time taken to execute the entire operation. | | | scanTimeMs | Time taken to scan the files for matches. | | | rewriteTimeMs | Time taken to rewrite the matched files. | | FSCK | numRemovedFiles | Number of files removed. | | CONVERT | numConvertedFiles | Number of Parquet files that have been converted. | | OPTIMIZE | | | | | numAddedFiles | Number of files added. | | | numRemovedFiles | Number of files optimized. | | | numAddedBytes | Number of bytes added after the table was optimized. | | | numRemovedBytes | Number of bytes removed. | | | minFileSize | Size of the smallest file after the table was optimized. | | | p25FileSize | Size of the 25th percentile file after the table was optimized. | | | p50FileSize | Median file size after the table was optimized. | | | p75FileSize | Size of the 75th percentile file after the table was optimized. | | | maxFileSize | Size of the largest file after the table was optimized. | | VACUUM | | | | | numDeletedFiles | Number of deleted files. | | | numVacuumedDirectories | Number of vacuumed directories. | | | numFilesToDelete | Number of files to delete. | | RESTORE | | | | | tableSizeAfterRestore | Table size in bytes after restore. | | | numOfFilesAfterRestore | Number of files in the table after restore. | | | numRemovedFiles | Number of files removed by the restore operation. | | | numRestoredFiles | Number of files that were added as a result of the restore. | | | removedFilesSize | Size in bytes of files removed by the restore. | | | restoredFilesSize | Size in bytes of files added by the restore. | ## Retrieve Delta table details [Section titled “Retrieve Delta table details”](#retrieve-delta-table-details) You can retrieve detailed information about a Delta table (for example, number of files, data size) using `DESCRIBE DETAIL`. See [Configure SparkSession](/delta-batch/#configure-sparksession) for the steps to enable support for SQL commands in Apache Spark. * SQL ```sql DESCRIBE DETAIL '/data/events/' DESCRIBE DETAIL eventsTable ``` * Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, pathToTable) detailDF = deltaTable.detail() ``` * Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, pathToTable) val detailDF = deltaTable.detail() ``` * Java ```java import io.delta.tables.*; DeltaTable deltaTable = DeltaTable.forPath(spark, pathToTable); DataFrame detailDF = deltaTable.detail(); ``` See the [Delta Lake APIs](/delta-apidoc/) for Scala/Java/Python syntax details. The output of this operation has only one row with the following schema. | Column | Type | Description | | :--------------- | :---------------- | :-------------------------------------------------------------------------------------- | | format | string | Format of the table, that is, `delta`. | | id | string | Unique ID of the table. | | name | string | Name of the table as defined in the metastore. | | description | string | Description of the table. | | location | string | Location of the table. | | createdAt | timestamp | When the table was created. | | lastModified | timestamp | When the table was last modified. | | partitionColumns | array of strings | Names of the partition columns if the table is partitioned. | | numFiles | long | Number of the files in the latest version of the table. | | sizeInBytes | int | The size of the latest snapshot of the table in bytes. | | properties | string-string map | All the properties set for this table. | | minReaderVersion | int | Minimum version of readers (according to the log protocol) that can read the table. | | minWriterVersion | int | Minimum version of writers (according to the log protocol) that can write to the table. | ```plaintext +------+--------------------+------------------+-----------+--------------------+--------------------+-------------------+----------------+--------+-----------+----------+----------------+----------------+ |format| id| name|description| location| createdAt| lastModified|partitionColumns|numFiles|sizeInBytes|properties|minReaderVersion|minWriterVersion| +------+--------------------+------------------+-----------+--------------------+--------------------+-------------------+----------------+--------+-----------+----------+----------------+----------------+ | delta|d31f82d2-a69f-42e...|default.deltatable| null|file:/Users/tuor/...|2020-06-05 12:20:...|2020-06-05 12:20:20| []| 10| 12345| []| 1| 2| +------+--------------------+------------------+-----------+--------------------+--------------------+-------------------+----------------+--------+-----------+----------+----------------+----------------+ ``` ## Generate a manifest file [Section titled “Generate a manifest file”](#generate-a-manifest-file) You can a generate manifest file for a Delta table that can be used by other processing engines (that is, other than Apache Spark) to read the Delta table. For example, to generate a manifest file that can be used by Presto and Athena to read a Delta table, you run the following: * SQL ```sql GENERATE symlink_format_manifest FOR TABLE delta.`` ``` * Python ```python deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` * Scala ```scala val deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` * Java ```java DeltaTable deltaTable = DeltaTable.forPath(); deltaTable.generate("symlink_format_manifest"); ``` See [Configure SparkSession](/delta-batch/#configure-sparksession) for the steps to enable support for SQL commands in Apache Spark. ## Convert a Parquet table to a Delta table [Section titled “Convert a Parquet table to a Delta table”](#convert-a-parquet-table-to-a-delta-table) Convert a Parquet table to a Delta table in-place. This command lists all the files in the directory, creates a Delta Lake transaction log that tracks these files, and automatically infers the data schema by reading the footers of all Parquet files. If your data is partitioned, you must specify the schema of the partition columns as a DDL-formatted string (that is, ` , , ...`). By default, this command will collect per-file statistics (e.g. minimum and maximum values for each column). These statistics will be used at query time to provide faster queries. You can disable this statistics collection in the SQL API using `NO STATISTICS`. Note If a Parquet table was created by Structured Streaming, the listing of files can be avoided by using the `_spark_metadata` sub-directory as the source of truth for files contained in the table setting the SQL configuration `spark.databricks.delta.convert.useMetadataLog` to `true`. * SQL ```sql -- Convert unpartitioned Parquet table at path '' CONVERT TO DELTA parquet.`` -- Convert unpartitioned Parquet table and disable statistics collection CONVERT TO DELTA parquet.`` NO STATISTICS -- Convert partitioned Parquet table at path '' and partitioned by integer columns named 'part' and 'part2' CONVERT TO DELTA parquet.`` PARTITIONED BY (part int, part2 int) -- Convert partitioned Parquet table and disable statistics collection CONVERT TO DELTA parquet.`` NO STATISTICS PARTITIONED BY (part int, part2 int) ``` * Python ```python from delta.tables import * # Convert unpartitioned Parquet table at path '' deltaTable = DeltaTable.convertToDelta(spark, "parquet.``") # Convert partitioned parquet table at path '' and partitioned by integer column named 'part' partitionedDeltaTable = DeltaTable.convertToDelta(spark, "parquet.``", "part int") ``` * Scala ```scala import io.delta.tables._ // Convert unpartitioned Parquet table at path '' val deltaTable = DeltaTable.convertToDelta(spark, "parquet.``") // Convert partitioned Parquet table at path '' and partitioned by integer columns named 'part' and 'part2' val partitionedDeltaTable = DeltaTable.convertToDelta(spark, "parquet.``", "part int, part2 int") ``` * Java ```java import io.delta.tables.*; // Convert unpartitioned Parquet table at path '' DeltaTable deltaTable = DeltaTable.convertToDelta(spark, "parquet.``"); // Convert partitioned Parquet table at path '' and partitioned by integer columns named 'part' and 'part2' DeltaTable deltaTable = DeltaTable.convertToDelta(spark, "parquet.``", "part int, part2 int"); ``` Note Any file not tracked by Delta Lake is invisible and can be deleted when you run `vacuum`. You should avoid updating or appending data files during the conversion process. After the table is converted, make sure all writes go through Delta Lake. ## Convert an Iceberg table to a Delta table [Section titled “Convert an Iceberg table to a Delta table”](#convert-an-iceberg-table-to-a-delta-table) Note It is available from Delta Lake 2.3 and above. You can convert an Iceberg table to a Delta table in place if the underlying file format of the Iceberg table is Parquet. Similar to a conversion from a Parquet table, the conversion is in-place and there won’t be any data copy or data rewrite. The original Iceberg table and the converted Delta table have separate history, so modifying the Delta table should not affect the Iceberg table as long as the source data Parquet files are not touched or deleted. The following command creates a Delta Lake transaction log based on the Iceberg table’s native file manifest, schema and partitioning information. The converter also collects column stats during the conversion, unless `NO STATISTICS` is specified. * SQL ```sql -- Convert the Iceberg table in the path . CONVERT TO DELTA iceberg.\`\` -- Convert the Iceberg table in the path without collecting statistics. CONVERT TO DELTA iceberg.\`\` NO STATISTICS ``` Important An additional jar `delta-iceberg` is needed to use the converter. For example, `bin/spark-sql --packages io.delta:delta-spark_2.12:3.0.0,io.delta:delta-iceberg_2.12:3.0.0:...`. `delta-iceberg` is currently not available for the Delta Lake 2.4.0 release since `iceberg-spark-runtime` does not support Spark 3.4 yet. It is available for Delta Lake 2.3.0. Note Converting Iceberg metastore tables is not supported. - Converting Iceberg tables that have experienced [partition evolution](https://iceberg.apache.org/docs/latest/evolution/#partition-evolution) is not supported. - Converting Iceberg merge-on-read tables that have experienced updates, deletions, or merges is not supported. ## Convert a Delta table to a Parquet table [Section titled “Convert a Delta table to a Parquet table”](#convert-a-delta-table-to-a-parquet-table) You can easily convert a Delta table back to a Parquet table using the following steps: 1. If you have performed Delta Lake operations that can change the data files (for example, `delete` or `merge`), run [vacuum](#remove-files-no-longer-referenced-by-a-delta-table) ) with retention of 0 hours to delete all data files that do not belong to the latest version of the table. 2. Delete the `_delta_log` directory in the table directory. ## Restore a Delta table to an earlier state [Section titled “Restore a Delta table to an earlier state”](#restore-a-delta-table-to-an-earlier-state) You can restore a Delta table to its earlier state by using the `RESTORE` command. A Delta table internally maintains historic versions of the table that enable it to be restored to an earlier state. A version corresponding to the earlier state or a timestamp of when the earlier state was created are supported as options by the `RESTORE` command. Important * You can restore an already restored table. * Restoring a table to an older version where the data files were deleted manually or by `vacuum` will fail. Restoring to this version partially is still possible if `spark.sql.files.ignoreMissingFiles` is set to `true`. * The timestamp format for restoring to an earlier state is `yyyy-MM-dd HH:mm:ss`. Providing only a date(`yyyy-MM-dd`) string is also supported. - SQL ```sql RESTORE TABLE db.target_table TO VERSION AS OF RESTORE TABLE delta.`/data/target/` TO TIMESTAMP AS OF ``` - Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, ) # path-based tables, or deltaTable = DeltaTable.forName(spark, ) # Hive metastore-based tables deltaTable.restoreToVersion(0) # restore table to oldest version deltaTable.restoreToTimestamp('2019-02-14') # restore to a specific timestamp ``` - Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, ) val deltaTable = DeltaTable.forName(spark, ) deltaTable.restoreToVersion(0) // restore table to oldest version deltaTable.restoreToTimestamp("2019-02-14") // restore to a specific timestamp ``` - Java ```java import io.delta.tables.*; DeltaTable deltaTable = DeltaTable.forPath(spark, ); DeltaTable deltaTable = DeltaTable.forName(spark, ); deltaTable.restoreToVersion(0) // restore table to oldest version deltaTable.restoreToTimestamp("2019-02-14") // restore to a specific timestamp ``` Important Restore is considered a data-changing operation. Delta Lake log entries added by the `RESTORE` command contain [dataChange](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#add-file-and-remove-file) set to true. If there is a downstream application, such as a [Structured streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) job that processes the updates to a Delta Lake table, the data change log entries added by the restore operation are considered as new data updates, and processing them may result in duplicate data. For example: | Table version | Operation | Delta log updates | Records in data change log updates | | :------------ | :----------------- | :-------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------ | | 0 | INSERT | AddFile(/path/to/file-1, dataChange = true) | (name = Viktor, age = 29), (name = George, age = 55) | | 1 | INSERT | AddFile(/path/to/file-2, dataChange = true) | (name = George, age = 39) | | 2 | OPTIMIZE | AddFile(/path/to/file-3, dataChange = false), RemoveFile(/path/to/file-1), RemoveFile(/path/to/file-2) | (No records as Optimize compaction does not change the data in the table) | | 3 | RESTORE(version=1) | RemoveFile(/path/to/file-3), AddFile(/path/to/file-1, dataChange = true), AddFile(/path/to/file-2, dataChange = true) | (name = Viktor, age = 29), (name = George, age = 55), (name = George, age = 39) | In the preceding example, the `RESTORE` command results in updates that were already seen when reading the Delta table version 0 and 1. If a streaming query was reading this table, then these files will be considered as newly added data and will be processed again. `RESTORE` reports the following metrics as a single row DataFrame once the operation is complete: * `table_size_after_restore`: The size of the table after restoring. * `num_of_files_after_restore`: The number of files in the table after restoring. * `num_removed_files`: Number of files removed (logically deleted) from the table. * `num_restored_files`: Number of files restored due to rolling back. * `removed_files_size`: Total size in bytes of the files that are removed from the table. * `restored_files_size`: Total size in bytes of the files that are restored.  ## Shallow clone a Delta table [Section titled “Shallow clone a Delta table”](#shallow-clone-a-delta-table) Note It is available from Delta Lake 2.3 and above. You can create a shallow copy of an existing Delta table at a specific version using the `shallow clone` command. Any changes made to shallow clones affect only the clones themselves and not the source table, as long as they don’t touch the source data Parquet files. The metadata that is cloned includes: schema, partitioning information, invariants, nullability. For shallow clones, stream metadata is not cloned. Metadata not cloned are the table description and [user-defined commit metadata](/delta-batch/#set-user-defined-commit-metadata). Important * Shallow clones reference data files in the source directory. If you run `vacuum` on the source table, clients will no longer be able to read the referenced data files and a `FileNotFoundException` will be thrown. In this case, running clone with `replace` over the shallow clone will repair the clone. * If a target already has a non-Delta table at that path, cloning with `replace` to that target will create a Delta log. Then, you can clean up any existing data by running `vacuum`. * If a Delta table exists in the target path, a new commit is created that includes the new metadata and new data from the source table. In the case of `replace`, the target table needs to be emptied first to avoid data duplication. * Cloning a table is not the same as `Create Table As Select` or `CTAS`. A shallow clone takes the metadata of the source table. Cloning also has simpler syntax: you don’t need to specify partitioning, format, invariants, nullability and so on as they are taken from the source table. * A cloned table has an independent history from its source table. Time travel queries on a cloned table will not work with the same inputs as they work on its source table. For example, if the source table was at version 100 and we are creating a new table by cloning it, the new table will have version 0, and therefore we could not run time travel queries on the new table such as `SELECT * FROM tbl AS OF VERSION 99`. - SQL ```sql CREATE TABLE delta.`/data/target/` SHALLOW CLONE delta.`/data/source/` -- Create a shallow clone of /data/source at /data/target CREATE OR REPLACE TABLE db.target_table SHALLOW CLONE db.source_table -- Replace the target. target needs to be emptied CREATE TABLE IF NOT EXISTS delta.`/data/target/` SHALLOW CLONE db.source_table -- No-op if the target table exists CREATE TABLE db.target_table SHALLOW CLONE delta.`/data/source` CREATE TABLE db.target_table SHALLOW CLONE delta.`/data/source` VERSION AS OF version CREATE TABLE db.target_table SHALLOW CLONE delta.`/data/source` TIMESTAMP AS OF timestamp_expression -- timestamp can be like “2019-01-01” or like date_sub(current_date(), 1) ``` `CLONE` reports the following metrics as a single row DataFrame once the operation is complete: * `source_table_size`: Size of the source table that’s being cloned in bytes. * `source_num_of_files`: The number of files in the source table. ### Cloud provider permissions [Section titled “Cloud provider permissions”](#cloud-provider-permissions) If you have created a shallow clone, any user that reads the shallow clone needs permission to read the files in the original table, since the data files remain in the source table’s directory where we cloned from. To make changes to the clone, users will need write access to the clone’s directory. #### Clone use cases [Section titled “Clone use cases”](#clone-use-cases) ### Machine learning flow reproduction [Section titled “Machine learning flow reproduction”](#machine-learning-flow-reproduction) When doing machine learning, you may want to archive a certain version of a table on which you trained an ML model. Future models can be tested using this archived data set. * SQL ```sql -- Trained model on version 15 of Delta table CREATE TABLE delta.`/model/dataset` SHALLOW CLONE entire_dataset VERSION AS OF 15 ``` ### Short-term experiments on a production table [Section titled “Short-term experiments on a production table”](#short-term-experiments-on-a-production-table) To test a workflow on a production table without corrupting the table, you can easily create a shallow clone. This allows you to run arbitrary workflows on the cloned table that contains all the production data but does not affect any production workloads. * SQL ```sql -- Perform shallow clone CREATE OR REPLACE TABLE my_test SHALLOW CLONE my_prod_table; UPDATE my_test WHERE user_id is null SET invalid=true; -- Run a bunch of validations. Once happy: -- This should leverage the update information in the clone to prune to only -- changed files in the clone if possible MERGE INTO my_prod_table USING my_test ON my_test.user_id <=> my_prod_table.user_id WHEN MATCHED AND my_test.user_id is null THEN UPDATE *; DROP TABLE my_test; ``` ### Table property overrides [Section titled “Table property overrides”](#table-property-overrides) Table property overrides are particularly useful for: * Annotating tables with owner or user information when sharing data with different business units. * Archiving Delta tables and time travel is required. You can specify the log retention period independently for the archive table. For example: - SQL ```sql CREATE OR REPLACE TABLE archive.my_table SHALLOW CLONE prod.my_table TBLPROPERTIES ( delta.logRetentionDuration = '3650 days', delta.deletedFileRetentionDuration = '3650 days' ) LOCATION 'xx://archive/my_table' ``` ## Clone Parquet or Iceberg table to Delta [Section titled “Clone Parquet or Iceberg table to Delta”](#clone-parquet-or-iceberg-table-to-delta) Note It is available from Delta Lake 2.3 and above. Shallow clone for Parquet and Iceberg combines functionality used to clone Delta tables and convert tables to Delta Lake, you can use clone functionality to convert data from Parquet or Iceberg data sources to managed or external Delta tables with the same basic syntax. `replace` has the same limitation as Delta shallow clone, the target table must be emptied before applying replace. * SQL ```sql CREATE OR REPLACE TABLE SHALLOW CLONE parquet.`/path/to/data`; CREATE OR REPLACE TABLE SHALLOW CLONE iceberg.`/path/to/data`; ``` # Apache Flink connector > Learn how to set up an integration to enable you to write Delta tables from Apache Flink. This integration enables reading from and writing to Delta tables from Apache Flink. For details on using the Flink/Delta Connector, see the [Delta Lake repository](https://github.com/delta-io/delta/tree/master/flink). # Apache Hive > Learn how to set up an integration to enable you to read Delta tables from . Page moved to [Apache Hive](/delta-more-connectors#apache-hive) # Integrations > Learn how to access Delta tables from external data processing engines. This page is moved to [Welcome to the Delta Lake documentation](/). # Optimizations > Learn about the optimizations available with Delta Lake. ## Optimize performance with file management [Section titled “Optimize performance with file management”](#optimize-performance-with-file-management) To improve query speed, Delta Lake supports the ability to optimize the layout of data in storage. There are various ways to optimize the layout. ### Compaction (bin-packing) [Section titled “Compaction (bin-packing)”](#compaction-bin-packing) Note This feature is available in Delta Lake 1.2.0 and above. Delta Lake can improve the speed of read queries from a table by coalescing small files into larger ones. * SQL ```sql OPTIMIZE '/path/to/delta/table' -- Optimizes the path-based Delta Lake table OPTIMIZE delta_table_name; OPTIMIZE delta.`/path/to/delta/table`; -- If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate using `WHERE`: OPTIMIZE delta_table_name WHERE date >= '2017-01-01' ``` * Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, pathToTable) # For path-based tables # For Hive metastore-based tables: deltaTable = DeltaTable.forName(spark, tableName) deltaTable.optimize().executeCompaction() # If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate using `where` deltaTable.optimize().where("date='2021-11-18'").executeCompaction() ``` * Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, pathToTable) // For path-based tables // For Hive metastore-based tables: val deltaTable = DeltaTable.forName(spark, tableName) deltaTable.optimize().executeCompaction() // If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate using `where` deltaTable.optimize().where("date='2021-11-18'").executeCompaction() ``` For Scala, Java, and Python API syntax details, see the [Delta Lake APIs](/delta-apidoc/). Note * Bin-packing optimization is *idempotent*, meaning that if it is run twice on the same dataset, the second run has no effect. - Bin-packing aims to produce evenly-balanced data files with respect to their size on disk, but not necessarily number of tuples per file. However, the two measures are most often correlated. - Python and Scala APIs for executing `OPTIMIZE` operation are available from Delta Lake 2.0 and above. - Set Spark session configuration `spark.databricks.delta.optimize.repartition.enabled=true` to use `repartition(1)` instead of `coalesce(1)` for better performance when compacting many small files. Readers of Delta tables use snapshot isolation, which means that they are not interrupted when `OPTIMIZE` removes unnecessary files from the transaction log. `OPTIMIZE` makes no data related changes to the table, so a read before and after an `OPTIMIZE` has the same results. Performing `OPTIMIZE` on a table that is a streaming source does not affect any current or future streams that treat this table as a source. `OPTIMIZE` returns the file statistics (min, max, total, and so on) for the files removed and the files added by the operation. Optimize stats also contains the number of batches, and partitions optimized. ## Auto compaction [Section titled “Auto compaction”](#auto-compaction) Note This feature is available in Delta Lake 3.1.0 and above. Auto compaction combines small files within Delta table partitions to automatically reduce small file problems. Auto compaction occurs after a write to a table has succeeded and runs synchronously on the cluster that has performed the write. Auto compaction only compacts files that haven’t been compacted previously. You can control the output file size by setting the configuration `spark.databricks.delta.autoCompact.maxFileSize`. Auto compaction is only triggered for partitions or tables that have at least a certain number of small files. You can optionally change the minimum number of files required to trigger auto compaction by setting `spark.databricks.delta.autoCompact.minNumFiles`. Auto compaction can be enabled at the table or session level using the following settings: * Table property: `delta.autoOptimize.autoCompact` * SparkSession setting: `spark.databricks.delta.autoCompact.enabled` These settings accept the following options: | Options | Behavior | | ------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `true` | Enable auto compaction. By default will use 128 MB as the target file size. | | `false` | Turns off auto compaction. Can be set at the session level to override auto compaction for all Delta tables modified in the workload. | ## Data skipping [Section titled “Data skipping”](#data-skipping) Note This feature is available in Delta Lake 1.2.0 and above. Data skipping information is collected automatically when you write data into a Delta Lake table. Delta Lake takes advantage of this information (minimum and maximum values for each column) at query time to provide faster queries. You do not need to configure data skipping; the feature is activated whenever applicable. However, its effectiveness depends on the layout of your data. For best results, apply [Z-Ordering](#z-ordering-multi-dimensional-clustering). Collecting statistics on a column containing long values such as `string` or `binary` is an expensive operation. To avoid collecting statistics on such columns you can configure the [table property](/delta-batch/#table-properties) `delta.dataSkippingNumIndexedCols`. This property indicates the position index of a column in the table’s schema. All columns with a position index less than the `delta.dataSkippingNumIndexedCols` property will have statistics collected. For the purposes of collecting statistics, each field within a nested column is considered as an individual column. To avoid collecting statistics on columns containing long values, either set the `delta.dataSkippingNumIndexedCols` property so that the long value columns are after this index in the table’s schema, or move columns containing long strings to an index position greater than the `delta.dataSkippingNumIndexedCols` property by using [ALTER TABLE ALTER COLUMN](https://spark.apache.org/docs/latest/sql-ref-syntax-ddl-alter-table.html#alter-or-change-column). ## Z-Ordering (multi-dimensional clustering) [Section titled “Z-Ordering (multi-dimensional clustering)”](#z-ordering-multi-dimensional-clustering) Note This feature is available in Delta Lake 2.0.0 and above. Z-Ordering is a [technique](https://en.wikipedia.org/wiki/Z-order_curve) to colocate related information in the same set of files. This co-locality is automatically used by Delta Lake in data-skipping algorithms. This behavior dramatically reduces the amount of data that Delta Lake on Apache Spark needs to read. To Z-Order data, you specify the columns to order on in the `ZORDER BY` clause: * SQL ```sql OPTIMIZE events ZORDER BY (eventType) -- If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate by using "where". OPTIMIZE events WHERE date = '2021-11-18' ZORDER BY (eventType) ``` * Python ```python from delta.tables import * deltaTable = DeltaTable.forPath(spark, pathToTable) # path-based table # For Hive metastore-based tables: deltaTable = DeltaTable.forName(spark, tableName) deltaTable.optimize().executeZOrderBy(eventType) # If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate using `where` deltaTable.optimize().where("date='2021-11-18'").executeZOrderBy(eventType) ``` * Scala ```scala import io.delta.tables._ val deltaTable = DeltaTable.forPath(spark, pathToTable) // path-based table // For Hive metastore-based tables: val deltaTable = DeltaTable.forName(spark, tableName) deltaTable.optimize().executeZOrderBy(eventType) // If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate by using "where". deltaTable.optimize().where("date='2021-11-18'").executeZOrderBy(eventType) ``` For Scala, Java, and Python API syntax details, see the [Delta Lake APIs](/delta-apidoc/) If you expect a column to be commonly used in query predicates and if that column has high cardinality (that is, a large number of distinct values), then use `ZORDER BY`. You can specify multiple columns for `ZORDER BY` as a comma-separated list. However, the effectiveness of the locality drops with each extra column. Z-Ordering on columns that do not have statistics collected on them would be ineffective and a waste of resources. This is because data skipping requires column-local stats such as min, max, and count. You can configure statistics collection on certain columns by reordering columns in the schema, or you can increase the number of columns to collect statistics on. See [Data skipping](#data-skipping). Note * Z-Ordering is *not idempotent*. Everytime the Z-Ordering is executed it will try to create a new clustering of data in all files (new and existing files that were part of previous Z-Ordering) in a partition. * Z-Ordering aims to produce evenly-balanced data files with respect to the number of tuples, but not necessarily data size on disk. The two measures are most often correlated, but there can be situations when that is not the case, leading to skew in optimize task times. For example, if you `ZORDER BY` *date* and your most recent records are all much wider (for example longer arrays or string values) than the ones in the past, it is expected that the `OPTIMIZE` job’s task durations will be skewed, as well as the resulting file sizes. This is, however, only a problem for the `OPTIMIZE` command itself; it should not have any negative impact on subsequent queries. ## Multi-part checkpointing [Section titled “Multi-part checkpointing”](#multi-part-checkpointing) Note This feature is available in Delta Lake 2.0.0 and above. This feature is in experimental support mode. Delta Lake table periodically and automatically compacts all the incremental updates to the Delta log into a Parquet file. This “checkpointing” allows read queries to quickly reconstruct the current state of the table (that is, which files to process, what is the current schema) without reading too many files having incremental updates. Delta Lake protocol allows [splitting the checkpoint](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#checkpoints) into multiple Parquet files. This parallelizes and speeds up writing the checkpoint. In Delta Lake, by default each classic checkpoint is written as a single Parquet file. For V2 checkpoints, the default part size is 50,000 actions, meaning V2 checkpoint sidecar files are automatically split when the number of actions exceeds this threshold. You can tune this for both classic and V2 checkpoints via the SQL configuration `spark.databricks.delta.checkpoint.partSize=`, where `n` is the limit of number of actions (such as `AddFile`) at which Delta Lake on Apache Spark will start parallelizing the checkpoint and attempt to write a maximum of this many actions per checkpoint file. Note This feature requires no reader side configuration changes. The existing reader already supports reading a checkpoint with multiple files. ## Log compactions [Section titled “Log compactions”](#log-compactions) Note This feature is available in Delta Lake 3.0.0 and above. Delta Lake protocol allows new log compaction files with the format `..compact.json`. These files contain the aggregated actions for commit range `[x, y]`. Log compactions reduce the need for frequent checkpoints and minimize the latency spikes caused by them. The read support for the log compaction files is available in Delta Lake 3.0.0 and above. It is enabled by default and can be disabled using the SQL conf `spark.databricks.delta.deltaLog.minorCompaction.useForReads=` where `value` can be `true/false`. The write support for the log compaction will be added in a future version of Delta. ## Optimized Write [Section titled “Optimized Write”](#optimized-write) Note This feature is available in Delta Lake 3.1.0 and above. Optimized writes improve file size as data is written and benefit subsequent reads on the table. Optimized writes are most effective for partitioned tables, as they reduce the number of small files written to each partition. Writing fewer large files is more efficient than writing many small files, but you might still see an increase in write latency because data is shuffled before being written. The following image demonstrates how optimized writes works:  Note You might have code that runs coalesce(n) or repartition(n) just before you write out your data to control the number of files written. Optimized writes eliminates the need to use this pattern. The optimized write feature is **disabled** by default. It can be enabled at the table, SQL session, and/or DataFrameWriter level using the following settings (in order of precedence from low to high): * The `delta.autoOptimize.optimizeWrite` table property (default=None); * The `spark.databricks.delta.optimizeWrite.enabled` SQL configuration (default=None); * The DataFrameWriter option `optimizeWrite` (default=None). Besides the above, the following advanced SQL configurations can be used to further fine-tune the number and size of files written: * `spark.databricks.delta.optimizeWrite.binSize` (default=512MiB), which controls the target in-memory size of each output file; * `spark.databricks.delta.optimizeWrite.numShuffleBlocks` (default=50,000,000), which controls “maximum number of shuffle blocks to target”; * `spark.databricks.delta.optimizeWrite.maxShufflePartitions` (default=2,000), which controls “max number of output buckets (reducers) that can be used by optimized writes”. # Migration guide > Learn how to migrate existing workloads to Delta Lake. ## Migrate workloads to Delta Lake [Section titled “Migrate workloads to Delta Lake”](#migrate-workloads-to-delta-lake) When you migrate workloads to Delta Lake, you should be aware of the following simplifications and differences compared with the data sources provided by Apache Spark and Apache Hive. Delta Lake handles the following operations automatically, which you should never perform manually: * **Add and remove partitions**: Delta Lake automatically tracks the set of partitions present in a table and updates the list as data is added or removed. As a result, there is no need to run `ALTER TABLE [ADD|DROP] PARTITION` or `MSCK`. * **Load a single partition**: As an optimization, you may sometimes directly load the partition of data you are interested in. For example, `spark.read.format("parquet").load("/data/date=2017-01-01")`. This is unnecessary with Delta Lake, since it can quickly read the list of files from the transaction log to find the relevant ones. If you are interested in a single partition, specify it using a `WHERE` clause. For example, `spark.read.delta("/data").where("date = '2017-01-01'")`. For large tables with many files in the partition, this can be much faster than loading a single partition (with direct partition path, or with `WHERE`) from a Parquet table because listing the files in the directory is often slower than reading the list of files from the transaction log. When you port an existing application to Delta Lake, you should avoid the following operations, which bypass the transaction log: * **Manually modify data**: Delta Lake uses the transaction log to atomically commit changes to the table. Because the log is the source of truth, files that are written out but not added to the transaction log are not read by Spark. Similarly, even if you manually delete a file, a pointer to the file is still present in the transaction log. Instead of manually modifying files stored in a Delta table, always use the commands that are described in this guide. * **External readers**: Directly reading the data stored in Delta Lake. For information on how to read Delta tables, see [read a table](/delta-batch/#read-a-table). ### Example [Section titled “Example”](#example) Suppose you have Parquet data stored in a directory named `/data-pipeline`, and you want to create a Delta table named `events`. The [first example](#save-as-delta-table) shows how to: * Read the Parquet data from its original location, `/data-pipeline`, into a DataFrame. * Save the DataFrame’s contents in Delta format in a separate location, `/tmp/delta/data-pipeline/`. * Create the `events` table based on that separate location, `/tmp/delta/data-pipeline/`. The [second example](#convert-to-delta-table) shows how to use `CONVERT TO TABLE` to convert data from Parquet to Delta format without changing its original location, `/data-pipeline/`. #### Save as Delta table [Section titled “Save as Delta table”](#save-as-delta-table) 1. Read the Parquet data into a DataFrame and then save the DataFrame’s contents to a new directory in `delta` format: ```python data = spark.read.format("parquet").load("/data-pipeline") data.write.format("delta").save("/tmp/delta/data-pipeline/") ``` 2. Create a Delta table named `events` that refers to the files in the new directory: ```python spark.sql("CREATE TABLE events USING DELTA LOCATION '/tmp/delta/data-pipeline/'") ``` #### Convert to Delta table [Section titled “Convert to Delta table”](#convert-to-delta-table) You have two options for converting a Parquet table to a Delta table: * Convert files to Delta Lake format and then create a Delta table: ```sql CONVERT TO DELTA parquet.`/data-pipeline/` CREATE TABLE events USING DELTA LOCATION '/data-pipeline/' ``` * Create a Parquet table and then convert it to a Delta table: ```sql CREATE TABLE events USING PARQUET OPTIONS (path '/data-pipeline/') CONVERT TO DELTA events ``` For details, see [Convert a Parquet table to a Delta table](/delta-utility/#convert-a-parquet-table-to-a-delta-table). ## Migrate Delta Lake workloads to newer versions [Section titled “Migrate Delta Lake workloads to newer versions”](#migrate-delta-lake-workloads-to-newer-versions) This section discusses any changes that may be required in the user code when migrating from older to newer versions of Delta Lake. ### Below Delta Lake 3.0 to Delta Lake 3.0 or above [Section titled “Below Delta Lake 3.0 to Delta Lake 3.0 or above”](#below-delta-lake-30-to-delta-lake-30-or-above) Please note that the Delta Lake on Spark Maven artifact has been renamed from `delta-core` (before 3.0) to `delta-spark` (3.0 and above). ### Delta Lake 2.1.1 or below to Delta Lake 2.2 or above [Section titled “Delta Lake 2.1.1 or below to Delta Lake 2.2 or above”](#delta-lake-211-or-below-to-delta-lake-22-or-above) Delta Lake 2.2 collects statistics by default when converting a parquet table to a Delta Lake table (e.g. using the `CONVERT TO DELTA` command). To opt out of statistics collection and revert to the 2.1.1 or below default behavior, use the `NO STATISTICS` SQL API (e.g. `CONVERT TO DELTA parquet.`/path-to-table` NO STATISTICS`) ### Delta Lake 1.2.1, 2.0.0, or 2.1.0 to Delta Lake 2.0.1, 2.1.1 or above [Section titled “Delta Lake 1.2.1, 2.0.0, or 2.1.0 to Delta Lake 2.0.1, 2.1.1 or above”](#delta-lake-121-200-or-210-to-delta-lake-201-211-or-above) Delta Lake 1.2.1, 2.0.0 and 2.1.0 have a bug in their DynamoDB-based S3 multi-cluster configuration implementations where an incorrect timestamp value was written to DynamoDB. This caused [DynamoDB’s TTL](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html) feature to cleanup completed items before it was safe to do so. This has been fixed in Delta Lake versions 2.0.1 and 2.1.1, and the TTL attribute has been renamed from `commitTime` to `expireTime`. If you *already* have TTL enabled on your DynamoDB table using the old attribute, you need to disable TTL for that attribute and then enable it for the new one. You may need to wait an hour between these two operations, as TTL settings changes may take some time to propagate. See the DynamoDB docs [here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-before-you-start.html). If you don’t do this, DyanmoDB’s TTL feature will not remove any new and expired entries. There is no risk of data loss. ```bash # Disable TTL on old attribute aws dynamodb update-time-to-live \ --region \ --table-name \ --time-to-live-specification "Enabled=false, AttributeName=commitTime" # Enable TTL on new attribute aws dynamodb update-time-to-live \ --region \ --table-name \ --time-to-live-specification "Enabled=true, AttributeName=expireTime" ``` ### Delta Lake 2.0 or below to Delta Lake 2.1 or above [Section titled “Delta Lake 2.0 or below to Delta Lake 2.1 or above”](#delta-lake-20-or-below-to-delta-lake-21-or-above) When calling `CONVERT TO DELTA` on a catalog table Delta Lake 2.1 infers the data schema from the catalog. In version 2.0 and below, Delta Lake infers the data schema from the data. This means in Delta 2.1 data columns that are not defined in the original catalog table will not be present in the converted Delta table. This behavior can be disabled by setting the Spark session configuration `spark.databricks.delta.convert.useCatalogSchema=false`. ### Delta Lake 1.2 or below to Delta Lake 2.0 or above [Section titled “Delta Lake 1.2 or below to Delta Lake 2.0 or above”](#delta-lake-12-or-below-to-delta-lake-20-or-above) Delta Lake 2.0.0 introduced a behavior change for [DROP CONSTRAINT](/delta-constraints/#check-constraint). In version 1.2 and below, no error was thrown when trying to drop a non-existent constraint. In version 2.0.0 and above, the behavior is changed to throw a constraint not exists error. To avoid the error, use `IF EXISTS` construct (for example, `ALTER TABLE events DROP CONSTRAINT IF EXISTS constraint_name`). There is no change in behavior in dropping an existing constraint. Delta Lake 2.0.0 introduced support for [Dynamic Partition Overwrites](/delta-batch/#overwrite). In version 1.2 and below, enabling dynamic partition overwrite mode in either the Spark session configuration or a `DataFrameWriter` option was a no-op, and writes in `overwrite` mode replaced all existing data in every partition of the table. In version 2.0.0 and above, when dynamic partition overwrite mode is enabled, Delta Lake replaces all existing data in each logical partition for which the write will commit new data. ### Delta Lake 1.1 or below to Delta Lake 1.2 or above [Section titled “Delta Lake 1.1 or below to Delta Lake 1.2 or above”](#delta-lake-11-or-below-to-delta-lake-12-or-above) The [LogStore](/api/latest/java/index.html) related code is extracted out from the `delta-core` Maven module into a new module `delta-storage` as part of the issue [#951](https://github.com/delta-io/delta/issues/951) for better code manageability. This results in an additional JAR `delta-storage-.jar` dependency for `delta-core`. By default, the additional JAR is downloaded as part of the `delta-core-_.jar` dependency. In clusters where there is *no internet connectivity*, `delta-storage-.jar` cannot be downloaded. It is advised to download the `delta-storage-.jar` manually and place it in the Java classpath. ### Delta Lake 1.0 or below to Delta Lake 1.1 or above [Section titled “Delta Lake 1.0 or below to Delta Lake 1.1 or above”](#delta-lake-10-or-below-to-delta-lake-11-or-above) If the name of a partition column in a Delta table contains invalid characters (` ,;{}()\n\t=`), you cannot read it in Delta Lake 1.1 and above, due to [SPARK-36271](https://issues.apache.org/jira/browse/SPARK-36271). However, this should be rare as you cannot create such tables by using Delta Lake 0.6 and above. If you still have such legacy tables, you can overwrite your tables with new valid column names by using Delta Lake 1.0 and below before upgrading Delta Lake to 1.1 and above, such as the following: * Python ```python spark.read \ .format("delta") \ .load("/the/delta/table/path") \ .withColumnRenamed("column name", "column-name") \ .write \ .format("delta")\ .mode("overwrite") \ .option("overwriteSchema", "true") \ .save("/the/delta/table/path") ``` * Scala ```scala spark.read .format("delta") .load("/the/delta/table/path") .withColumnRenamed("column name", "column-name") .write .format("delta") .mode("overwrite") .option("overwriteSchema", "true") .save("/the/delta/table/path") ``` ### Delta Lake 0.6 or below to Delta Lake 0.7 or above [Section titled “Delta Lake 0.6 or below to Delta Lake 0.7 or above”](#delta-lake-06-or-below-to-delta-lake-07-or-above) If you are using `DeltaTable` APIs in Scala, Java, or Python to [update](/delta-update/) or [run utility operations](/delta-utility/) on them, then you may have to add the following configurations when creating the `SparkSession` used to perform those operations. * Python ```python from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName("...") \ .master("...") \ .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \ .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \ .getOrCreate() ``` * Scala ```scala import org.apache.spark.sql.SparkSession val spark = SparkSession .builder() .appName("...") .master("...") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate() ``` * Java ```java import org.apache.spark.sql.SparkSession; SparkSession spark = SparkSession .builder() .appName("...") .master("...") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate(); ``` Alternatively, you can add additional configurations when submitting you Spark application using `spark-submit` or when starting `spark-shell`/`pyspark` by specifying them as command line parameters. ```bash spark-submit \ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \ --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \ ... ``` ```bash pyspark \ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \ --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \ ... ``` # Presto, Trino, and Athena to Delta Lake integration using manifests > Learn how to set up an integration to enable you to read Delta tables from Presto, Trino, and Athena. Important Presto, Trino and Athena all have native support for Delta Lake. Support is as follows: * Presto [version 0.269](https://prestodb.io/docs/0.269/release/release-0.269.html#delta-lake-connector-changes) and above natively supports reading the Delta Lake tables. For details on using the native Delta Lake connector, see [Delta Lake Connector - Presto](https://prestodb.io/docs/current/connector/deltalake.html). For Presto versions lower than [0.269](https://prestodb.io/docs/0.269/release/release-0.269.html#delta-lake-connector-changes), you can use the manifest-based approach in this article. * Trino [version 373](https://trino.io/docs/current/release/release-373.html) and above natively supports reading and writing the Delta Lake tables. For details on using the native Delta Lake connector, see [Delta Lake Connector - Trino](https://trino.io/docs/current/connector/delta-lake.html). For Trino versions lower than [version 373](https://trino.io/docs/current/release/release-373.html), you can use the manifest-based approach detailed in this article. * Athena [version 3](https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html) and above natively supports reading Delta Lake tables. For details on using the native Delta Lake connector, see [Querying Delta Lake tables](https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html). For Athena versions lower than [version 3](https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html), you can use the manifest-based approach detailed in this article. Presto, Trino, and Athena support reading from external tables using a *manifest file*, which is a text file containing the list of data files to read for querying a table. When an external table is defined in the Hive metastore using manifest files, Presto, Trino, and Athena can use the list of files in the manifest rather than finding the files by directory listing. This article describes how to set up a Presto, Trino, and Athena to Delta Lake integration using manifest files and query Delta tables. ## Set up the Presto, Trino, or Athena to Delta Lake integration and query Delta tables [Section titled “Set up the Presto, Trino, or Athena to Delta Lake integration and query Delta tables”](#set-up-the-presto-trino-or-athena-to-delta-lake-integration-and-query-delta-tables) You set up a Presto, Trino, or Athena to Delta Lake integration using the following steps. ### Step 1: Generate manifests of a Delta table using Apache Spark [Section titled “Step 1: Generate manifests of a Delta table using Apache Spark”](#step-1-generate-manifests-of-a-delta-table-using-apache-spark) Using Spark [configured](/quick-start#set-up-apache-spark-with-delta-lake) with Delta Lake, run any of the following commands on a Delta table at location ``: * SQL ```sql GENERATE symlink_format_manifest FOR TABLE delta.`` ``` * Scala ```scala val deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` * Java ```java DeltaTable deltaTable = DeltaTable.forPath(); deltaTable.generate("symlink_format_manifest"); ``` * Python ```python deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` See [Generate a manifest file](/delta-utility/#generate-a-manifest-file) for details. The `generate` command generates manifest files at `/_symlink_format_manifest/`. In other words, the files in this directory will contain the names of the data files (that is, Parquet files) that should be read for reading a snapshot of the Delta table. ### Step 2: Configure Presto, Trino, or Athena to read the generated manifests [Section titled “Step 2: Configure Presto, Trino, or Athena to read the generated manifests”](#step-2-configure-presto-trino-or-athena-to-read-the-generated-manifests) 1. Define a new table in the Hive metastore connected to Presto, Trino, or Athena using the format `SymlinkTextInputFormat` and the manifest location `/_symlink_format_manifest/`. ```sql CREATE EXTERNAL TABLE mytable ([(col_name1 col_datatype1, ...)]) [PARTITIONED BY (col_name2 col_datatype2, ...)] ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION '/_symlink_format_manifest/' -- location of the generated manifest ``` `SymlinkTextInputFormat` configures Presto, Trino, or Athena to compute file splits for `mytable` by reading the manifest file instead of using a directory listing to find data files. Replace `mytable` with the name of the external table and `` with the absolute path to the Delta table. Important * `mytable` must be the same schema and have the same partitions as the Delta table. * The set of `PARTITIONED BY` columns must be distinct from the set of non-partitioned columns. Furthermore, you cannot specify partitioned columns with `AS `. - You cannot use this table definition in Apache Spark; it can be used only by Presto, Trino, and Athena. The tool you use to run the command depends on whether Apache Spark and Presto, Trino, or Athena use the same Hive metastore. * **Same metastore**: If both Apache Spark and Presto, Trino, or Athena use the same Hive metastore, you can define the table using Apache Spark. - **Different metastores**: If Apache Spark and Presto, Trino, or Athena use different metastores, you must define the table using other tools. * Athena: You can define the external table in Athena. * Presto: Presto does not support the syntax `CREATE EXTERNAL TABLE ... STORED AS ...`, so you must use another tool (for example, Spark or Hive) connected to the same metastore as Presto to create the table. 2. If the Delta table is partitioned, run `MSCK REPAIR TABLE mytable` after generating the manifests to force the metastore (connected to Presto, Trino, or Athena) to discover the partitions. This is needed because the manifest of a partitioned table is itself partitioned in the same directory structure as the table. Run this command using *the same tool* used to create the table. Furthermore, you should run this command: * **After every manifest generation**: New partitions are likely to be visible immediately after the manifest files have been updated. However, doing this too frequently can cause high load for the Hive metastore. * **As frequently as new partitions are expected**: For example, if a table is partitioned by date, then you can run repair once after every midnight, after the new partition has been created in the table and its corresponding manifest files have been generated. ### Step 3: Update manifests [Section titled “Step 3: Update manifests”](#step-3-update-manifests) When the data in a Delta table is updated you must regenerate the manifests using either of the following approaches: * **Update explicitly**: After all the data updates, you can run the `generate` operation to update the manifests. * **Update automatically**: You can configure a Delta table so that all write operations on the table automatically update the manifests. To enable this automatic mode, set the corresponding table property using the following SQL command. ```sql ALTER TABLE delta.`` SET TBLPROPERTIES(delta.compatibility.symlinkFormatManifest.enabled=true) ``` To disable this automatic mode, set this property to `false`. In addition, for partitioned tables, you have to run `MSCK REPAIR` to ensure the metastore connected to Presto, Trino, or Athena to update partitions. Note After enabling automatic mode on a partitioned table, each write operation updates only manifests corresponding to the partitions that operation wrote to. This incremental update ensures that the overhead of manifest generation is low for write operations. However, this also means that if the manifests in other partitions are stale, enabling automatic mode will not automatically fix it. Therefore, it is recommended that you explicitly run `GENERATE` to update manifests for the entire table immediately after enabling automatic mode. Whether to update automatically or explicitly depends on the concurrent nature of write operations on the Delta table and the desired data consistency. For example, if automatic mode is enabled, concurrent write operations lead to concurrent overwrites to the manifest files. With such unordered writes, the manifest files are not guaranteed to point to the latest version of the table after the write operations complete. Hence, if concurrent writes are expected and you want to avoid stale manifests, you should consider explicitly updating the manifest after the expected write operations have completed. ## Limitations [Section titled “Limitations”](#limitations) The Presto, Trino, and Athena integration has known limitations in its behavior. ### Data consistency [Section titled “Data consistency”](#data-consistency) Whenever Delta Lake generates updated manifests, it atomically overwrites existing manifest files. Therefore, Presto, Trino, and Athena will always see a consistent view of the data files; it will see all of the old version files or all of the new version files. However, the granularity of the consistency guarantees depends on whether or not the table is partitioned. * **Unpartitioned tables**: All the files names are written in one manifest file which is updated atomically. In this case Presto, Trino, and Athena will see full table snapshot consistency. * **Partitioned tables**: A manifest file is partitioned in the same Hive-partitioning-style directory structure as the original Delta table. This means that each partition is updated atomically, and Presto, Trino, or Athena will see a consistent view of each partition but not a consistent view across partitions. Furthermore, since all manifests of all partitions cannot be updated together, concurrent attempts to generate manifests can lead to different partitions having manifests of different versions. While this consistency guarantee under data change is weaker than that of reading Delta tables with Spark, it is still stronger than formats like Parquet as they do not provide partition-level consistency. Depending on what storage system you are using for Delta tables, it is possible to get incorrect results when Presto, Trino, or Athena concurrently queries the manifest while the manifest files are being rewritten. In file system implementations that lack atomic file overwrites, a manifest file may be momentarily unavailable. Hence, use manifests with caution if their updates are likely to coincide with queries from Presto, Trino, or Athena. ### Performance [Section titled “Performance”](#performance) Very large numbers of files can hurt the performance of Presto, Trino, and Athena. Hence it is recommended that you [compact the files](/best-practices#compact-files) of the table before generating the manifests. The number of files should not exceed 1000 (for the entire unpartitioned table or for each partition in a partitioned table). ### Schema evolution [Section titled “Schema evolution”](#schema-evolution) Delta Lake supports schema evolution and queries on a Delta table automatically use the latest schema regardless of the schema defined in the table in the Hive metastore. However, Presto, Trino, or Athena uses the schema defined in the Hive metastore and will not query with the updated schema until the table used by Presto, Trino, or Athena is redefined to have the updated schema. ### Encrypted tables [Section titled “Encrypted tables”](#encrypted-tables) Athena does not support reading manifests from [CSE-KMS](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-emrfs-encryption-cse.html) encrypted tables. See the AWS documentation for the latest information. # Quick Start > Learn how to get started quickly with Delta Lake. This guide helps you quickly explore the main features of Delta Lake. It provides code snippets that show how to read from and write to Delta tables from interactive, batch, and streaming queries. ## Set up Apache Spark with Delta Lake [Section titled “Set up Apache Spark with Delta Lake”](#set-up-apache-spark-with-delta-lake) Follow these instructions to set up Delta Lake with Spark. You can run the steps in this guide on your local machine in the following two ways: 1. **Run interactively**: Start the Spark shell (Scala or Python) with Delta Lake and run the code snippets interactively in the shell. 2. **Run as a project**: Set up a Maven or SBT project (Scala or Java) with Delta Lake, copy the code snippets into a source file, and run the project. Alternatively, you can use the [examples provided in the Github repository](https://github.com/delta-io/delta/tree/master/examples). Important! For all of the following instructions, make sure to install the correct version of Spark or PySpark that is compatible with Delta Lake `4.0.0`. See the [release compatibility matrix](/releases) for details. ### Prerequisite: set up Java [Section titled “Prerequisite: set up Java”](#prerequisite-set-up-java) As mentioned in the official Apache Spark installation instructions [here](https://spark.apache.org/docs/latest/index.html#downloading), make sure you have a valid Java version installed (8, 11, or 17) and that Java is configured correctly on your system using either the system `PATH` or `JAVA_HOME` environmental variable. Windows users should follow the instructions in this [blog](https://phoenixnap.com/kb/install-spark-on-windows-10), making sure to use the correct version of Apache Spark that is compatible with Delta Lake `4.0.0`. ### Set up interactive shell [Section titled “Set up interactive shell”](#set-up-interactive-shell) To use Delta Lake interactively within the Spark SQL, Scala, or Python shell, you need a local installation of Apache Spark. Depending on whether you want to use SQL, Python, or Scala, you can set up either the SQL, PySpark, or Spark shell, respectively. #### Spark SQL Shell [Section titled “Spark SQL Shell”](#spark-sql-shell) Download the [compatible version](/releases) of Apache Spark by following instructions from [Downloading Spark](https://spark.apache.org/downloads.html), either using `pip` or by downloading and extracting the archive and running `spark-sql` in the extracted directory. ```bash bin/spark-sql --packages io.delta:delta-spark_2.13:4.0.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ``` #### PySpark Shell [Section titled “PySpark Shell”](#pyspark-shell) 1. Install the PySpark version that is [compatible](/releases) with the Delta Lake version by running the following: ```bash pip install pyspark== ``` 2. Run PySpark with the Delta Lake package and additional configurations: ```bash pyspark --packages io.delta:delta-spark_2.13:4.0.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ``` #### Spark Scala Shell [Section titled “Spark Scala Shell”](#spark-scala-shell) Download the [compatible version](/releases/) of Apache Spark by following instructions from [Downloading Spark](https://spark.apache.org/downloads.html), either using `pip` or by downloading and extracting the archive and running `spark-shell` in the extracted directory. ```bash bin/spark-shell --packages io.delta:delta-spark_2.13:4.0.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" ``` ### Set up project [Section titled “Set up project”](#set-up-project) If you want to build a project using Delta Lake binaries from Maven Central Repository, you can use the following Maven coordinates. #### Maven [Section titled “Maven”](#maven) You include Delta Lake in your Maven project by adding it as a dependency in your POM file. Delta Lake compiled with Scala 2.13. * XML ```xml io.delta delta-spark_2.13 4.0.0 ``` #### SBT [Section titled “SBT”](#sbt) You include Delta Lake in your SBT project by adding the following line to your `build.sbt` file: * Scala ```scala libraryDependencies += "io.delta" %% "delta-spark" % "4.0.0" ``` #### Python [Section titled “Python”](#python) To set up a Python project (for example, for unit testing), you can install Delta Lake using `pip install delta-spark==4.0.0` and then configure the SparkSession with the `configure_spark_with_delta_pip()` utility function in Delta Lake. * Python ```python import pyspark from delta import * builder = pyspark.sql.SparkSession.builder.appName("MyApp") \ .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \ .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") spark = configure_spark_with_delta_pip(builder).getOrCreate() ``` ## Create a table [Section titled “Create a table”](#create-a-table) To create a Delta table, write a DataFrame out in the `delta` format. You can use existing Spark SQL code and change the format from `parquet`, `csv`, `json`, and so on, to `delta`. * SQL ```sql CREATE TABLE delta.`/tmp/delta-table` USING DELTA AS SELECT col1 as id FROM VALUES 0,1,2,3,4; ``` * Python ```python data = spark.range(0, 5) data.write.format("delta").save("/tmp/delta-table") ``` * Scala ```scala val data = spark.range(0, 5) data.write.format("delta").save("/tmp/delta-table") ``` * Java ```java import org.apache.spark.sql.SparkSession; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; SparkSession spark = ... // create SparkSession Dataset data = spark.range(0, 5); data.write().format("delta").save("/tmp/delta-table"); ``` These operations create a new Delta table using the schema that was *inferred* from your DataFrame. For the full set of options available when you create a new Delta table, see [Create a table](/delta-batch/#create-a-table) and [Write to a table](/delta-batch/#write-to-table). Note This quickstart uses local paths for Delta table locations. For configuring HDFS or cloud storage for Delta tables, see [Storage configuration](/delta-storage). ## Read data [Section titled “Read data”](#read-data) You read data in your Delta table by specifying the path to the files: `"/tmp/delta-table"`: * SQL ```sql SELECT * FROM delta.`/tmp/delta-table`; ``` * Python ```python df = spark.read.format("delta").load("/tmp/delta-table") df.show() ``` * Scala ```scala val df = spark.read.format("delta").load("/tmp/delta-table") df.show() ``` * Java ```java Dataset df = spark.read().format("delta").load("/tmp/delta-table"); df.show(); ``` ## Update table data [Section titled “Update table data”](#update-table-data) Delta Lake supports several operations to modify tables using standard DataFrame APIs. This example runs a batch job to overwrite the data in the table: ### Overwrite [Section titled “Overwrite”](#overwrite) * SQL ```sql INSERT OVERWRITE delta.`/tmp/delta-table` SELECT col1 as id FROM VALUES 5,6,7,8,9; ``` * Python ```python data = spark.range(5, 10) data.write.format("delta").mode("overwrite").save("/tmp/delta-table") ``` * Scala ```scala val data = spark.range(5, 10) data.write.format("delta").mode("overwrite").save("/tmp/delta-table") df.show() ``` * Java ```java Dataset data = spark.range(5, 10); data.write().format("delta").mode("overwrite").save("/tmp/delta-table"); ``` If you read this table again, you should see only the values `5-9` you have added because you overwrote the previous data. ### Conditional update without overwrite [Section titled “Conditional update without overwrite”](#conditional-update-without-overwrite) Delta Lake provides programmatic APIs to conditional update, delete, and merge (upsert) data into tables. Here are a few examples. * SQL ```sql -- Update every even value by adding 100 to it UPDATE delta.`/tmp/delta-table` SET id = id + 100 WHERE id % 2 == 0; -- Delete every even value DELETE FROM delta.`/tmp/delta-table` WHERE id % 2 == 0; -- Upsert (merge) new data CREATE TEMP VIEW newData AS SELECT col1 AS id FROM VALUES 1,3,5,7,9,11,13,15,17,19; MERGE INTO delta.`/tmp/delta-table` AS oldData USING newData ON oldData.id = newData.id WHEN MATCHED THEN UPDATE SET id = newData.id WHEN NOT MATCHED THEN INSERT (id) VALUES (newData.id); SELECT * FROM delta.`/tmp/delta-table`; ``` * Python ```python from delta.tables import * from pyspark.sql.functions import * deltaTable = DeltaTable.forPath(spark, "/tmp/delta-table") # Update every even value by adding 100 to it deltaTable.update( condition = expr("id % 2 == 0"), set = { "id": expr("id + 100") }) # Delete every even value deltaTable.delete(condition = expr("id % 2 == 0")) # Upsert (merge) new data newData = spark.range(0, 20) deltaTable.alias("oldData") \ .merge( newData.alias("newData"), "oldData.id = newData.id") \ .whenMatchedUpdate(set = { "id": col("newData.id") }) \ .whenNotMatchedInsert(values = { "id": col("newData.id") }) \ .execute() deltaTable.toDF().show() ``` * Scala ```scala import io.delta.tables._ import org.apache.spark.sql.functions._ val deltaTable = DeltaTable.forPath("/tmp/delta-table") // Update every even value by adding 100 to it deltaTable.update( condition = expr("id % 2 == 0"), set = Map("id" -> expr("id + 100"))) // Delete every even value deltaTable.delete(condition = expr("id % 2 == 0")) // Upsert (merge) new data val newData = spark.range(0, 20).toDF deltaTable.as("oldData") .merge( newData.as("newData"), "oldData.id = newData.id") .whenMatched .update(Map("id" -> col("newData.id"))) .whenNotMatched .insert(Map("id" -> col("newData.id"))) .execute() deltaTable.toDF.show() ``` * Java ```java import io.delta.tables.*; import org.apache.spark.sql.functions; import java.util.HashMap; DeltaTable deltaTable = DeltaTable.forPath("/tmp/delta-table"); // Update every even value by adding 100 to it deltaTable.update( functions.expr("id % 2 == 0"), new HashMap() {{ put("id", functions.expr("id + 100")); }} ); // Delete every even value deltaTable.delete(condition = functions.expr("id % 2 == 0")); // Upsert (merge) new data Dataset newData = spark.range(0, 20).toDF(); deltaTable.as("oldData") .merge( newData.as("newData"), "oldData.id = newData.id") .whenMatched() .update( new HashMap() {{ put("id", functions.col("newData.id")); }}) .whenNotMatched() .insertExpr( new HashMap() {{ put("id", functions.col("newData.id")); }}) .execute(); deltaTable.toDF().show(); ``` You should see that some of the existing rows have been updated and new rows have been inserted. For more information on these operations, see [Table deletes, updates, and merges](/delta-update). ## Read older versions of data using time travel [Section titled “Read older versions of data using time travel”](#read-older-versions-of-data-using-time-travel) You can query previous snapshots of your Delta table by using time travel. If you want to access the data that you overwrote, you can query a snapshot of the table before you overwrote the first set of data using the `versionAsOf` option. * SQL ```sql SELECT * FROM delta.`/tmp/delta-table` VERSION AS OF 0; ``` * Python ```python df = spark.read.format("delta").option("versionAsOf", 0).load("/tmp/delta-table") df.show() ``` * Scala ```scala val df = spark.read.format("delta").option("versionAsOf", 0).load("/tmp/delta-table") df.show() ``` * Java ```java Dataset df = spark.read().format("delta").option("versionAsOf", 0).load("/tmp/delta-table"); df.show(); ``` You should see the first set of data, from before you overwrote it. Time travel takes advantage of the power of the Delta Lake transaction log to access data that is no longer in the table. Removing the version `0` option (or specifying version `1`) would let you see the newer data again. For more information, see [Query an older snapshot of a table (time travel)](/delta-batch#query-an-older-snapshot-of-a-table-time-travel). ## Write a stream of data to a table [Section titled “Write a stream of data to a table”](#write-a-stream-of-data-to-a-table) You can also write to a Delta table using Structured Streaming. The Delta Lake transaction log guarantees exactly-once processing, even when there are other streams or batch queries running concurrently against the table. By default, streams run in append mode, which adds new records to the table: * Python ```python streamingDf = spark.readStream.format("rate").load() stream = streamingDf.selectExpr("value as id").writeStream.format("delta").option("checkpointLocation", "/tmp/checkpoint").start("/tmp/delta-table") ``` * Scala ```scala val streamingDf = spark.readStream.format("rate").load() val stream = streamingDf.select($"value" as "id").writeStream.format("delta").option("checkpointLocation", "/tmp/checkpoint").start("/tmp/delta-table") ``` * Java ```java import org.apache.spark.sql.streaming.StreamingQuery; Dataset streamingDf = spark.readStream().format("rate").load(); StreamingQuery stream = streamingDf.selectExpr("value as id").writeStream().format("delta").option("checkpointLocation", "/tmp/checkpoint").start("/tmp/delta-table"); ``` While the stream is running, you can read the table using the earlier commands. Note If you’re running this in a shell, you may see the streaming task progress, which make it hard to type commands in that shell. It may be useful to start another shell in a new terminal for querying the table. You can stop the stream by running `stream.stop()` in the same terminal that started the stream. For more information about Delta Lake integration with Structured Streaming, see [Table streaming reads and writes](/delta-streaming). See also the [Structured Streaming Programming Guide](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) on the Apache Spark website. ## Read a stream of changes from a table [Section titled “Read a stream of changes from a table”](#read-a-stream-of-changes-from-a-table) While the stream is writing to the Delta table, you can also read from that table as streaming source. For example, you can start another streaming query that prints all the changes made to the Delta table. You can specify which version Structured Streaming should start from by providing the `startingVersion` or `startingTimestamp` option to get changes from that point onwards. See [Structured Streaming](/delta-streaming/#specify-initial-position) for details. * Python ```python stream2 = spark.readStream.format("delta").load("/tmp/delta-table").writeStream.format("console").start() ``` * Scala ```scala val stream2 = spark.readStream.format("delta").load("/tmp/delta-table").writeStream.format("console").start() ``` * Java ```java StreamingQuery stream2 = spark.readStream().format("delta").load("/tmp/delta-table").writeStream().format("console").start(); ``` # AWS Redshift Spectrum connector > Learn how to set up an integration to enable you to read Delta tables from AWS Redshift. Experimental This is an experimental integration. Use with caution. A Delta table can be read by AWS Redshift Spectrum using a *manifest file*, which is a text file containing the list of data files to read for querying a Delta table. This article describes how to set up a AWS Redshift Spectrum to Delta Lake integration using manifest files and query Delta tables. ## Set up a AWS Redshift Spectrum to Delta Lake integration and query Delta tables [Section titled “Set up a AWS Redshift Spectrum to Delta Lake integration and query Delta tables”](#set-up-a-aws-redshift-spectrum-to-delta-lake-integration-and-query-delta-tables) You set up a AWS Redshift Spectrum to Delta Lake integration using the following steps. ### Step 1: Generate manifests of a Delta table using Apache Spark [Section titled “Step 1: Generate manifests of a Delta table using Apache Spark”](#step-1-generate-manifests-of-a-delta-table-using-apache-spark) Run the `generate` operation on a Delta table at location ``: * SQL ```sql GENERATE symlink_format_manifest FOR TABLE delta.`` ``` * Scala ```scala val deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` * Java ```java DeltaTable deltaTable = DeltaTable.forPath(); deltaTable.generate("symlink_format_manifest"); ``` * Python ```python deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` See [Generate a manifest file](/delta-utility/#generate-a-manifest-file) for details. The `generate` operation generates manifest files at `/_symlink_format_manifest/`. In other words, the files in this directory contain the names of the data files (that is, Parquet files) that should be read for reading a snapshot of the Delta table. Note We recommend that you define the Delta table in a location that AWS Redshift Spectrum can read directly. ### Step 2: Configure AWS Redshift Spectrum to read the generated manifests [Section titled “Step 2: Configure AWS Redshift Spectrum to read the generated manifests”](#step-2-configure-aws-redshift-spectrum-to-read-the-generated-manifests) Run the following commands in your AWS Redshift Spectrum environment. 1. Define a new external table in AWS Redshift Spectrum using the format `SymlinkTextInputFormat` and the manifest location `/_symlink_format_manifest/`. * SQL ```sql CREATE EXTERNAL TABLE mytable ([(col_name1 col_datatype1, ...)]) [PARTITIONED BY (col_name2 col_datatype2, ...)] ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION '/_symlink_format_manifest/' -- location of the generated manifest ``` `SymlinkTextInputFormat` configures AWS Redshift Spectrum to compute file splits for `mytable` by reading the manifest file instead of using a directory listing to find data files. Replace `mytable` with the name of the external table and `` with the absolute path to the Delta table. Important * `mytable` must be the same schema and have the same partitions as the Delta table. * The set of `PARTITIONED BY` columns must be distinct from the set of non-partitioned columns. Furthermore, you cannot specify partitioned columns with `AS `. - You cannot use this table definition in Apache Spark; it can be used only by AWS Redshift Spectrum. 2. If the Delta table is partitioned, you must add the partitions explicitly to the AWS Redshift Spectrum table. This is needed because the manifest of a partitioned table is itself partitioned in the same directory structure as the table. * For every partition in the table, run the following in AWS Redshift Spectrum, either directly in AWS Redshift Spectrum, or using the AWS CLI or [Data API](https://docs.aws.amazon.com/redshift/latest/mgmt/data-api.html): * SQL ```sql ALTER TABLE mytable.redshiftdeltatable ADD IF NOT EXISTS PARTITION (col_name=col_value) LOCATION '/_symlink_format_manifest/col_name=col_value' ``` This steps will provide you with a [consistent](#data-consistency) view of the Delta table. ### Step 3: Update manifests [Section titled “Step 3: Update manifests”](#step-3-update-manifests) When data in a Delta table is updated, you must regenerate the manifests using either of the following approaches: * **Update explicitly**: After all the data updates, you can run the `generate` operation to update the manifests. * **Update automatically**: You can configure a Delta table so that all write operations on the table automatically update the manifests. To enable this automatic mode, set the corresponding table property using the following SQL command. - SQL ```sql ALTER TABLE delta.`` SET TBLPROPERTIES(delta.compatibility.symlinkFormatManifest.enabled=true) ``` To disable this automatic mode, set this property to `false`. Note After enabling automatic mode on a partitioned table, each write operation updates only manifests corresponding to the partitions that operation wrote to. This incremental update ensures that the overhead of manifest generation is low for write operations. However, this also means that if the manifests in other partitions are stale, enabling automatic mode will not automatically fix it. Therefore, you should explicitly run `GENERATE` to update manifests for the entire table immediately after enabling automatic mode. Whether to update automatically or explicitly depends on the concurrent nature of write operations on the Delta table and the desired data consistency. For example, if automatic mode is enabled, concurrent write operations lead to concurrent overwrites to the manifest files. With such unordered writes, the manifest files are not guaranteed to point to the latest version of the table after the write operations complete. Hence, if concurrent writes are expected and you want to avoid stale manifests, you should consider explicitly updating the manifest after the expected write operations have completed. In addition, if your table is partitioned, then you must add any new partitions or remove deleted partitions by following the same process as described in the preceding step. ## Limitations [Section titled “Limitations”](#limitations) The AWS Redshift Spectrum integration has known limitations in its behavior. ### Data consistency [Section titled “Data consistency”](#data-consistency) Whenever Delta Lake generates updated manifests, it atomically overwrites existing manifest files. Therefore, AWS Redshift Spectrum will always see a consistent view of the data files; it will see all of the old version files or all of the new version files. However, the granularity of the consistency guarantees depends on whether or not the table is partitioned. * **Unpartitioned tables**: All the files names are written in one manifest file which is updated atomically. In this case AWS Redshift Spectrum will see full table snapshot consistency. * **Partitioned tables**: A manifest file is partitioned in the same Hive-partitioning-style directory structure as the original Delta table. This means that each partition is updated atomically, and AWS Redshift Spectrum will see a consistent view of each partition but not a consistent view across partitions. Furthermore, since all manifests of all partitions cannot be updated together, concurrent attempts to generate manifests can lead to different partitions having manifests of different versions. While this consistency guarantee under data change is weaker than that of reading Delta tables with Spark, it is still stronger than formats like Parquet as they do not provide partition-level consistency. Depending on what storage system you are using for Delta tables, it is possible to get incorrect results when AWS Redshift Spectrum concurrently queries the manifest while the manifest files are being rewritten. In file system implementations that lack atomic file overwrites, a manifest file may be momentarily unavailable. Hence, use manifests with caution if their updates are likely to coincide with queries from AWS Redshift Spectrum. ### Performance [Section titled “Performance”](#performance) This is an experimental integration and its performance and scalability characteristics have not yet been tested. ### Schema evolution [Section titled “Schema evolution”](#schema-evolution) Delta Lake supports schema evolution and queries on a Delta table automatically use the latest schema regardless of the schema defined in the table in the Hive metastore. However, AWS Redshift Spectrum uses the schema defined in its table definition, and will not query with the updated schema until the table definition is updated to the new schema. # Releases > Learn about Delta Lake releases. ## Release notes [Section titled “Release notes”](#release-notes) The [GitHub releases page](https://github.com/delta-io/delta/releases/) describes features of each release. ## Compatibility with Apache Spark [Section titled “Compatibility with Apache Spark”](#compatibility-with-apache-spark) The following table lists Delta Lake versions and their compatible Apache Spark versions. | Delta Lake version | Apache Spark version | | ------------------ | ----------------------- | | 4.0.x | 4.0.x | | 3.3.x | 3.5.x | | 3.2.x | 3.5.x | | 3.1.x | 3.5.x | | 3.0.x | 3.5.x | | 2.4.x | 3.4.x | | 2.3.x | 3.3.x | | 2.2.x | 3.3.x | | 2.1.x | 3.3.x | | 2.0.x | 3.2.x | | 1.2.x | 3.2.x | | 1.1.x | 3.2.x | | 1.0.x | 3.1.x | | 0.7.x and 0.8.x | 3.0.x | | Below 0.7.0 | 2.4.2 - 2.4.*\* | # Snowflake connector > Learn how to set up an integration to enable you to read Delta tables from Snowflake. Visit the [Snowflake Delta Lake support](https://docs.snowflake.com/en/user-guide/tables-external-intro.html#delta-lake-support) documentation to use the connector. Important Some users in the community have reported that Snowflake, unlike [Trino](https://trino.io/docs/current/connector/delta-lake.html) or [Spark](/delta-batch/), is not using [Delta statistics](/optimizations-oss/#data-skipping) to do data skipping when reading Delta tables. Due to this bug, Snowflake may read a lot of unnecessary parquet files resulting in poor query performance and increased API call requests from cloud providers. If you are a Snowflake customer and have subscribed to their enterprise support, please [open a support case](https://community.snowflake.com/s/article/How-To-Submit-a-Support-Case-in-Snowflake-Lodge). As a workaround, you can enable [Delta UniForm](/delta-uniform/) to generate Iceberg metadata and read these tables as Iceberg tables from Snowflake. A Delta table can be read by Snowflake using a *manifest file*, which is a text file containing the list of data files to read for querying a Delta table. This article describes how to set up a Delta Lake to Snowflake integration using manifest files and query Delta tables. ## Set up a Delta Lake to Snowflake integration and query Delta tables [Section titled “Set up a Delta Lake to Snowflake integration and query Delta tables”](#set-up-a-delta-lake-to-snowflake-integration-and-query-delta-tables) You set up a Delta Lake to Snowflake integration using the following steps. ### Step 1: Generate manifests of a Delta table using Apache Spark [Section titled “Step 1: Generate manifests of a Delta table using Apache Spark”](#step-1-generate-manifests-of-a-delta-table-using-apache-spark) Run the `generate` operation on a Delta table at location ``: * SQL ```sql GENERATE symlink_format_manifest FOR TABLE delta.`` ``` * Scala ```scala val deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` * Java ```java DeltaTable deltaTable = DeltaTable.forPath(); deltaTable.generate("symlink_format_manifest"); ``` * Python ```python deltaTable = DeltaTable.forPath() deltaTable.generate("symlink_format_manifest") ``` See [Generate a manifest file](/delta-utility/#generate-a-manifest-file) for details. The `generate` operation generates manifest files at `/_symlink_format_manifest/`. In other words, the files in this directory contain the names of the data files (that is, Parquet files) that should be read for reading a snapshot of the Delta table. Note We recommend that you define the Delta table in a location that Snowflake can read directly. ### Step 2: Configure Snowflake to read the generated manifests [Section titled “Step 2: Configure Snowflake to read the generated manifests”](#step-2-configure-snowflake-to-read-the-generated-manifests) Run the following commands in your Snowflake environment. #### Define an external table on the manifest files [Section titled “Define an external table on the manifest files”](#define-an-external-table-on-the-manifest-files) To define an external table in Snowflake, you must first [define a external stage](https://docs.snowflake.net/manuals/user-guide/data-load-s3-create-stage.html) `my_staged_table` that points to the Delta table. In Snowflake, run the following. * SQL ```sql create or replace stage my_staged_table url='' ``` Replace `` with the full path to the Delta table. Using this stage, you can [define a table](https://docs.snowflake.net/manuals/sql-reference/sql/create-external-table.html) `delta_manifest_table` that reads the file names specified in the manifest files as follows: * SQL ````sql VARCHAR AS split_part(VALUE:c1, '/', -1) ) WITH LOCATION = @my_staged_table/_symlink_format_manifest/ FILE_FORMAT = (TYPE = CSV) PATTERN = '.*[/]manifest' AUTO_REFRESH = true; ``` #### Define an external table on Parquet files You can define a table `my_parquet_data_table` that reads all the Parquet files in the Delta table. ```sql CREATE OR REPLACE EXTERNAL TABLE my_parquet_data_table( id INT AS (VALUE:id::INT), part INT AS (VALUE:part::INT), ..., parquet_filename VARCHAR AS split_part(metadata$filename, '/', -1) ) WITH LOCATION = @my_staged_table/ FILE_FORMAT = (TYPE = PARQUET) PATTERN = '.*[/]part-[^/]*[.]parquet' AUTO_REFRESH = true; ```` Note In this query: * The location is the Delta table path. * The `parquet_filename` column contains the name of the file that contains each row of the table. If your Delta table is partitioned, then you will have to explicitly extract the partition values in the table definition. For example, if the table was partitioned by a single integer column named `part`, you can extract the values as follows: * SQL ```sql CREATE OR REPLACE EXTERNAL TABLE my_parquet_data_partitioned_table( id INT AS (VALUE:id::INT), part INT AS ( nullif( regexp_replace(metadata$filename, '.*part\\=(.*)\\/.*', '\\1'), '__HIVE_DEFAULT_PARTITION__' )::INT ), ..., parquet_filename VARCHAR AS split_part(metadata$filename, '/', -1) ) WITH LOCATION = @my_staged_partitioned_table/ FILE_FORMAT = (TYPE = PARQUET) PATTERN = '.*[/]part-[^/]*[.]parquet' AUTO_REFRESH = true; ``` The regular expression is used to extract the partition value for the column `part`. Querying the Delta table as this Parquet table will produce incorrect results because this query will read all the Parquet files in this table rather than only those that define a consistent snapshot of the table. You can use the manifest table to get a consistent snapshot data. #### Define view to get correct contents of the Delta table using the manifest table [Section titled “Define view to get correct contents of the Delta table using the manifest table”](#define-view-to-get-correct-contents-of-the-delta-table-using-the-manifest-table) To read only the rows belonging to the consistent snapshot defined in the generated manifests, you can apply a filter to keep only the rows in the Parquet table that came from the files defined in the manifest table. * SQL ```sql CREATE OR REPLACE VIEW my_delta_table AS SELECT id, part, ... FROM my_parquet_data_table WHERE parquet_filename IN ( SELECT filename FROM delta-manifest-table ); ``` Querying this view will provide you with a [consistent](#data-consistency) view of the Delta table. ### Step 3: Update manifests [Section titled “Step 3: Update manifests”](#step-3-update-manifests) When data in a Delta table is updated, you must regenerate the manifests using either of the following approaches: * **Update explicitly**: After all the data updates, you can run the `generate` operation to update the manifests. * **Update automatically**: You can configure a Delta table so that all write operations on the table automatically update the manifests. To enable this automatic mode, set the corresponding table property using the following SQL command. - SQL ```sql ALTER TABLE delta.`` SET TBLPROPERTIES(delta.compatibility.symlinkFormatManifest.enabled=true) ``` ## Limitations [Section titled “Limitations”](#limitations) The Snowflake integration has known limitations in its behavior. ### Data consistency [Section titled “Data consistency”](#data-consistency) Whenever Delta Lake generates updated manifests, it atomically overwrites existing manifest files. Therefore, Snowflake will always see a consistent view of the data files; it will see all of the old version files or all of the new version files. However, the granularity of the consistency guarantees depends on whether the table is partitioned or not. * **Unpartitioned tables**: All the files names are written in one manifest file which is updated atomically. In this case Snowflake will see full table snapshot consistency. * **Partitioned tables**: A manifest file is partitioned in the same Hive-partitioning-style directory structure as the original Delta table. This means that each partition is updated atomically, and Snowflake will see a consistent view of each partition but not a consistent view across partitions. Furthermore, since all manifests of all partitions cannot be updated together, concurrent attempts to generate manifests can lead to different partitions having manifests of different versions. Depending on what storage system you are using for Delta tables, it is possible to get incorrect results when Snowflake concurrently queries the manifest while the manifest files are being rewritten. In file system implementations that lack atomic file overwrites, a manifest file may be momentarily unavailable. Hence, use manifests with caution if their updates are likely to coincide with queries from Snowflake. ### Performance [Section titled “Performance”](#performance) This is an experimental integration and its performance and scalability characteristics have not yet been tested. ### Schema evolution [Section titled “Schema evolution”](#schema-evolution) Delta Lake supports schema evolution and queries on a Delta table automatically use the latest schema regardless of the schema defined in the table in the Hive metastore. However, Snowflake uses the schema defined in its table definition, and will not query with the updated schema until the table definition is updated to the new schema. # Delta Table Properties Reference > Access the list of available Delta table properties. | Property | Description | Data type | Default | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------ | | `delta.appendOnly` | `true` for this Delta table to be append-only. If append-only, existing records cannot be deleted, and existing values cannot be updated. See [Table properties](/delta-batch/#table-properties). | `Boolean` | `false` | | `delta.checkpoint.writeStatsAsJson` | `true` for Delta Lake to write file statistics in checkpoints in JSON format for the `stats` column. | `Boolean` | `true` | | `delta.checkpoint.writeStatsAsStruct` | `true` for Delta Lake to write file statistics to checkpoints in struct format for the `stats_parsed` column and to write partition values as a struct for `partitionValues_parsed`. | `Boolean` | (none) | | `delta.compatibility.symlinkFormatManifest.enabled` | `true` for Delta Lake to configure the Delta table so that all write operations on the table automatically update the manifests. See [Update manifests](/presto-integration/#step-3-update-manifests). | `Boolean` | `false` | | `delta.dataSkippingNumIndexedCols` | The number of columns for Delta Lake to collect statistics about for data skipping. A value of `-1` means to collect statistics for all columns. Updating this property does not automatically collect statistics again; instead, it redefines the statistics schema of the Delta table. For example, it changes the behavior of future statistics collection (such as during appends and optimizations) as well as data skipping (such as ignoring column statistics beyond this number, even when such statistics exist). See [Data skipping](/optimizations-oss/#data-skipping). | `Int` | `32` | | `delta.deletedFileRetentionDuration` | The shortest duration for Delta Lake to keep logically deleted data files before deleting them physically. This is to prevent failures in stale readers after compactions or partition overwrites. This value should be large enough to ensure that: - It is larger than the longest possible duration of a job if you run `VACUUM` when there are concurrent readers or writers accessing the Delta table. - If you run a streaming query that reads from the table, that the query does not stop for longer than this value. Otherwise, the query may not be able to restart, as it must still read old files. See [Data retention](/delta-batch/#data-retention). | `CalendarInterval` | `interval 1 week` | | `delta.enableChangeDataFeed` | `true` to enable change data feed. See [Enable change data feed](/delta-change-data-feed/#enable-change-data-feed). | `Boolean` | `false` | | `delta.logRetentionDuration` | How long the history for a Delta table is kept. Each time a checkpoint is written, Delta Lake automatically cleans up log entries older than the retention interval. If you set this property to a large enough value, many log entries are retained. This should not impact performance as operations against the log are constant time. Operations on history are parallel but will become more expensive as the log size increases. See [Data retention](/delta-batch/#data-retention). | `CalendarInterval` | `interval 30 days` | | `delta.minReaderVersion` | The minimum required protocol reader version for a reader that allows to read from this Delta table. See [Versioning](/versioning). | `Int` | `1` | | `delta.minWriterVersion` | The minimum required protocol writer version for a writer that allows to write to this Delta table. See [Versioning](/versioning). | `Int` | `2` | | `delta.setTransactionRetentionDuration` | The shortest duration within which new snapshots will retain transaction identifiers (for example, `SetTransaction`s). When a new snapshot sees a transaction identifier older than or equal to the duration specified by this property, the snapshot considers it expired and ignores it. The `SetTransaction` identifier is used when making the writes idempotent. See [Idempotent table writes in foreachBatch](/delta-streaming/#idempotent-table-writes-in-foreachbatch) for details. | `CalendarInterval` | (none) | | `delta.checkpointPolicy` | `classic` for classic Delta Lake checkpoints. `v2` for v2 checkpoints. See [V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec) for details. See [Versioning](/versioning) for details around compatibility. | `String` | `classic` | # How does Delta Lake manage feature compatibility? > Learn how Delta table protocols are versioned. Many Delta Lake optimizations require enabling Delta Lake features on a table. Delta Lake features are always backwards compatible, so tables written by a lower Delta Lake version can always be read and written by a higher Delta Lake version. Enabling some features breaks forward compatibility with workloads running in a lower Delta Lake version. For features that break forward compatibility, you must update all workloads that reference the upgraded tables to use a compliant Delta Lake version. ## What Delta Lake features require client upgrades? [Section titled “What Delta Lake features require client upgrades?”](#what-delta-lake-features-require-client-upgrades) The following Delta Lake features break forward compatibility. Features are enabled on a table-by-table basis. | Feature | Requires Delta Lake version or later | Documentation | | --------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `CHECK` constraints | [Delta Lake 0.8.0](https://github.com/delta-io/delta/releases/tag/v0.8.0) | [CHECK constraint](/delta-constraints/#check-constraint) | | Generated columns | [Delta Lake 1.0.0](https://github.com/delta-io/delta/releases/tag/v1.0.0) | [Use generated columns](/delta-batch/#use-generated-columns) | | Column mapping | [Delta Lake 1.2.0](https://github.com/delta-io/delta/releases/tag/v1.2.0) | [Delta column mapping](/delta-column-mapping/) | | Change data feed | [Delta Lake 2.0.0](https://github.com/delta-io/delta/releases/tag/v2.0.0) | [Change data feed](/delta-change-data-feed/) | | Deletion vectors | [Delta Lake 2.3.0](https://github.com/delta-io/delta/releases/tag/v2.3.0) | [What are deletion vectors?](/delta-deletion-vectors/) | | Table features | [Delta Lake 2.3.0](https://github.com/delta-io/delta/releases/tag/v2.3.0) | [What are table features?](#what-are-table-features) | | Timestamp without Timezone | [Delta Lake 2.4.0](https://github.com/delta-io/delta/releases/tag/v2.4.0) | [TimestampNTZType](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) | | Iceberg Compatibility V1 | [Delta Lake 3.0.0](https://github.com/delta-io/delta/releases/tag/v3.0.0) | [IcebergCompatV1](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#iceberg-compatibility-v1) | | Iceberg Compatibility V2 | [Delta Lake 3.1.0](https://github.com/delta-io/delta/releases/tag/v3.1.0) | [IcebergCompatV2](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#iceberg-compatibility-v2) | | V2 Checkpoints | [Delta Lake 3.0.0](https://github.com/delta-io/delta/releases/tag/v3.0.0) | [V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec) | | Domain metadata | [Delta Lake 3.0.0](https://github.com/delta-io/delta/releases/tag/v3.0.0) | [Domain Metadata Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#domain-metadata) | | Clustering | [Delta Lake 3.1.0](https://github.com/delta-io/delta/releases/tag/v3.1.0) | [Use liquid clustering for Delta tables](/delta-clustering/) | | Row Tracking | [Delta Lake 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0) | [Use row tracking for Delta tables](/delta-row-tracking/) | | Type widening (Preview) | [Delta Lake 3.2.0](https://github.com/delta-io/delta/releases/tag/v3.2.0) | [Delta type widening](/delta-type-widening/) | | Type widening | [Delta Lake 4.0.0](https://github.com/delta-io/delta/releases/tag/v4.0.0) | [Delta type widening](/delta-type-widening/) | | Identity columns | [Delta Lake 3.3.0](https://github.com/delta-io/delta/releases/tag/v3.3.0) | [Use identity columns](/delta-batch/#use-identity-columns) | | Variant Type | [Delta Lake 4.0.0](https://github.com/delta-io/delta/releases/tag/v4.0.0) | [Delta type widening](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#variant-data-type) | | Variant Shredding (Preview) | [Delta Lake 4.0.0](https://github.com/delta-io/delta/releases/tag/v4.0.0) | [Variant Shredding](https://github.com/delta-io/delta/blob/master/protocol_rfcs/accepted/variant-shredding.md) | | Checkpoint Protection | [Delta Lake 4.0.0](https://github.com/delta-io/delta/releases/tag/v4.0.0) | [Checkpoint Protection](https://github.com/delta-io/delta/blob/master/protocol_rfcs/checkpoint-protection.md) | ## What is a table protocol specification? [Section titled “What is a table protocol specification?”](#what-is-a-table-protocol-specification) Every Delta table has a protocol specification which indicates the set of features that the table supports. The protocol specification is used by applications that read or write the table to determine if they can handle all the features that the table supports. If an application does not know how to handle a feature that is listed as supported in the protocol of a table, then that application is not be able to read or write that table. The protocol specification is separated into two components: the *read protocol* and the *write protocol*. ### Read protocol [Section titled “Read protocol”](#read-protocol) The read protocol lists all features that a table supports and that an application must understand in order to read the table correctly. Upgrading the read protocol of a table requires that all reader applications support the added features. Important All applications that write to a Delta table must be able to construct a snapshot of the table. As such, workloads that write to Delta tables must respect both reader and writer protocol requirements. If you encounter a protocol that is unsupported by a workload on Delta Lake, you must upgrade to a higher Delta Lake implementation with more comprehensive support. ### Write protocol [Section titled “Write protocol”](#write-protocol) The write protocol lists all features that a table supports and that an application must understand in order to write to the table correctly. Upgrading the write protocol of a table requires that all writer applications support the added features. It does not affect read-only applications, unless the read protocol is also upgraded. ## Which protocols must be upgraded? [Section titled “Which protocols must be upgraded?”](#which-protocols-must-be-upgraded) Some features require upgrading both the read protocol and the write protocol. Other features only require upgrading the write protocol. As an example, support for `CHECK` constraints is a write protocol feature: only writing applications need to know about `CHECK` constraints and enforce them. In contrast, column mapping requires upgrading both the read and write protocols. Because the data is stored differently in the table, reader applications must understand column mapping so they can read the data correctly. For more on upgrading, see [Upgrading protocol versions](#upgrading-protocol-versions). ## What are table features? [Section titled “What are table features?”](#what-are-table-features) In Delta Lake 2.3.0 and above, Delta Lake table features introduce granular flags specifying which features are supported by a given table. Table features are the successor to protocol versions and are designed with the goal of improved flexibility for clients that read and write Delta Lake. See [What is a protocol version?](#what-is-a-protocol-version). Note Table features have protocol version requirements. See [Features by protocol version](#features-by-protocol-version). A Delta table feature is a marker that indicates that the table supports a particular feature. Every feature is either a write protocol feature (meaning it only upgrades the write protocol) or a read/write protocol feature (meaning both read and write protocols are upgraded to enable the feature). To learn more about supported table features in Delta Lake, see the [Delta Lake protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#valid-feature-names-in-table-features). ## Do table features change how Delta Lake features are enabled? [Section titled “Do table features change how Delta Lake features are enabled?”](#do-table-features-change-how-delta-lake-features-are-enabled) If you only interact with Delta tables through Delta Lake, you can continue to track support for Delta Lake features using minimum Delta Lake requirements. If you read and write from Delta tables using other systems, you might need to consider how table features impact compatibility, because there is a risk that the system could not understand the upgraded protocol versions. ## What is a protocol version? [Section titled “What is a protocol version?”](#what-is-a-protocol-version) A protocol version is a protocol number that indicates a particular grouping of table features. In Delta Lake 2.3.0 and below, you cannot enable table features individually. Protocol versions bundle a group of features. Delta tables specify a separate protocol version for read protocol and write protocol. The transaction log for a Delta table contains protocol versioning information that supports Delta Lake evolution. The protocol versions bundle all features from previous protocols. See [Features by protocol version](#features-by-protocol-version). Note Starting with writer version 7 and reader version 3, Delta Lake has introduced the concept of table features. Using table features, you can now choose to only enable those features that are supported by other clients in your data ecosystem. See [What are table features?](#what-are-table-features). ## Features by protocol version [Section titled “Features by protocol version”](#features-by-protocol-version) The following table shows minimum protocol versions required for Delta Lake features. | Feature | `minWriterVersion` | `minReaderVersion` | Documentation | | --------------------------- | ------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------- | | Basic functionality | 2 | 1 | [Welcome to the Delta Lake documentation](/) | | `CHECK` constraints | 3 | 1 | [CHECK constraint](/delta-constraints/#check-constraint) | | Change data feed | 4 | 1 | [Change data feed](/delta-change-data-feed/) | | Generated columns | 4 | 1 | [Use generated columns](/delta-batch/#use-generated-columns) | | Column mapping | 5 | 2 | [Delta column mapping](/delta-column-mapping/) | | Identity columns | 6 | 1 | [Use identity columns](/delta-batch/#use-identity-columns) | | Table features read | 7 | 1 | [What are table features?](#what-are-table-features) | | Table features write | 7 | 3 | [What are table features?](#what-are-table-features) | | Deletion vectors | 7 | 3 | [What are deletion vectors?](/delta-deletion-vectors/) | | Timestamp without Timezone | 7 | 3 | [TimestampNTZType](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) | | Iceberg Compatibility V1 | 7 | 2 | [IcebergCompatV1](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#iceberg-compatibility-v1) | | V2 Checkpoints | 7 | 3 | [V2 Checkpoint Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec) | | Vacuum Protocol Check | 7 | 3 | [Vacuum Protocol Check Spec](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#vacuum-protocol-check) | | Row Tracking | 7 | 3 | [Use row tracking for Delta tables](/delta-row-tracking/) | | Type widening (Preview) | 7 | 3 | [Delta type widening](/delta-type-widening/) | | Type widening | 7 | 3 | [Delta type widening](/delta-type-widening/) | | Variant Type | 7 | 3 | [Variant Type](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#variant-data-type) | | Variant Shredding (Preview) | 7 | 3 | [Variant Shredding](https://github.com/delta-io/delta/blob/master/protocol_rfcs/accepted/variant-shredding.md) | ## Upgrading protocol versions [Section titled “Upgrading protocol versions”](#upgrading-protocol-versions) You can choose to manually update a table to a newer protocol version. We recommend using the lowest protocol versions that support the Delta Lake features required for your table. Upgrading the writer protocol might cause less disruption than upgrading the reader protocol since systems and workloads using older Delta Lake versions can still read from tables, even if they do not support the updated writer protocol. Warning Protocol version upgrades are irreversible, and upgrading the protocol version might break the existing Delta Lake table readers, writers, or both. We recommend you upgrade specific tables only when needed, such as to opt-in to new features in Delta Lake. You should also check to make sure that all of your current and future production tools support Delta Lake tables with the new protocol version. To upgrade a table to a newer protocol version, use the `DeltaTable.upgradeTableProtocol` method: * SQL ```sql -- Upgrades the reader protocol version to 1 and the writer protocol version to 3. ALTER TABLE SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3') ``` * Python ```python from delta.tables import DeltaTable delta = DeltaTable.forPath(spark, "path_to_table") # or DeltaTable.forName delta.upgradeTableProtocol(1, 3) # upgrades to readerVersion=1, writerVersion=3 ``` * Scala ```scala import io.delta.tables.DeltaTable val delta = DeltaTable.forPath(spark, "path_to_table") // or DeltaTable.forName delta.upgradeTableProtocol(1, 3) // Upgrades to readerVersion=1, writerVersion=3. ```