Modify the protocol to add a supported feature, and if the table does not support table features, upgrade the protocol automatically. In such a case when the provided feature is writer-only, the table's writer version will be upgraded to 7, and when the provided feature is reader-writer, both reader and writer versions will be upgraded, to (3, 7).

See online documentation and Delta's protocol specification at PROTOCOL.md for more details.

Apply an alias to the DeltaTable. This is similar to Dataset.as(alias) or SQL tableName AS alias.

Apply an alias to the DeltaTable. This is similar to Dataset.as(alias) or SQL tableName AS alias.

Clone a DeltaTable to a given destination to mirror the existing table's data and metadata.

An example would be

io.delta.tables.DeltaTable.clone(
  "/some/path/to/table",
  true)

Clone a DeltaTable to a given destination to mirror the existing table's data and metadata.

An example would be

io.delta.tables.DeltaTable.clone(
  "/some/path/to/table",
  true,
  true)

clone used by Python implementation using java.util.HashMap for the properties argument.

Specifying properties here means that the target will override any properties with the same key in the source table with the user-defined properties.

An example would be

io.delta.tables.DeltaTable.clone(
  "/some/path/to/table",
  true,
  true,
  Map("foo" -> "bar"))

Clone a DeltaTable to a given destination to mirror the existing table's data and metadata.

Specifying properties here means that the target will override any properties with the same key in the source table with the user-defined properties.

An example would be

io.delta.tables.DeltaTable.clone(
 "/some/path/to/table",
 true,
 true,
 Map("foo" -> "bar"))

Clone a DeltaTable at a specific timestamp to a given destination to mirror the existing table's data and metadata at that timestamp.

Timestamp can be of the format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.

An example would be

io.delta.tables.DeltaTable.cloneAtTimestamp(
  "2019-01-01",
  "/some/path/to/table",
  true)

Clone a DeltaTable at a specific timestamp to a given destination to mirror the existing table's data and metadata at that timestamp.

Timestamp can be of the format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.

An example would be

io.delta.tables.DeltaTable.cloneAtTimestamp(
  "2019-01-01",
  "/some/path/to/table",
  true,
  true)

cloneAtTimestamp used by Python implementation using java.util.HashMap for the properties argument.

Clone a DeltaTable at a specific timestamp to a given destination to mirror the existing table's data and metadata at that version. Specifying properties here means that the target will override any properties with the same key in the source table with the user-defined properties.

An example would be

io.delta.tables.DeltaTable.cloneAtVersion(
  5,
  "/some/path/to/table",
  true,
  true,
  new java.util.HashMap[String, String](Map("foo" -> "bar").asJava)

Clone a DeltaTable at a specific timestamp to a given destination to mirror the existing table's data and metadata at that timestamp.

Timestamp can be of the format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.

Specifying properties here means that the target will override any properties with the same key in the source table with the user-defined properties.

An example would be

io.delta.tables.DeltaTable.cloneAtTimestamp(
  "2019-01-01",
  "/some/path/to/table",
  true,
  true,
  Map("foo" -> "bar"))

Clone a DeltaTable at a specific version to a given destination to mirror the existing table's data and metadata at that version.

An example would be

io.delta.tables.DeltaTable.cloneAtVersion(
  5,
  "/some/path/to/table",
  true)

Clone a DeltaTable at a specific version to a given destination to mirror the existing table's data and metadata at that version.

An example would be

io.delta.tables.DeltaTable.cloneAtVersion(
  5,
  "/some/path/to/table",
  true,
  true)

cloneAtVersion used by Python implementation using java.util.HashMap for the properties argument.

Specifying properties here means that the target will override any properties with the same key in the source table with the user-defined properties.

An example would be

io.delta.tables.DeltaTable.cloneAtVersion(
  5,
  "/some/path/to/table",
  true,
  true,
  new java.util.HashMap[String, String](Map("foo" -> "bar").asJava))

Clone a DeltaTable at a specific version to a given destination to mirror the existing table's data and metadata at that version.

Specifying properties here means that the target will override any properties with the same key in the source table with the user-defined properties.

An example would be

io.delta.tables.DeltaTable.cloneAtVersion(
  5,
  "/some/path/to/table",
  true,
  true,
  Map("foo" -> "bar"))

Delete data from the table.

Delete data from the table that match the given condition.

Delete data from the table that match the given condition.

:: Evolving ::

Get the details of a Delta table such as the format, name, and size.

Modify the protocol to drop a supported feature. The operation always normalizes the resulting protocol. Protocol normalization is the process of converting a table features protocol to the weakest possible form. This primarily refers to converting a table features protocol to a legacy protocol. A table features protocol can be represented with the legacy representation only when the feature set of the former exactly matches a legacy protocol. Normalization can also decrease the reader version of a table features protocol when it is higher than necessary. For example:

(1, 7, None, {AppendOnly, Invariants, CheckConstraints}) -> (1, 3) (3, 7, None, {RowTracking}) -> (1, 7, RowTracking)

The dropFeatureSupport method can be used as follows:

io.delta.tables.DeltaTable.dropFeatureSupport("rowTracking")

Note, this command will not truncate history.

See online documentation for more details.

Modify the protocol to drop a supported feature. The operation always normalizes the resulting protocol. Protocol normalization is the process of converting a table features protocol to the weakest possible form. This primarily refers to converting a table features protocol to a legacy protocol. A table features protocol can be represented with the legacy representation only when the feature set of the former exactly matches a legacy protocol. Normalization can also decrease the reader version of a table features protocol when it is higher than necessary. For example:

(1, 7, None, {AppendOnly, Invariants, CheckConstraints}) -> (1, 3) (3, 7, None, {RowTracking}) -> (1, 7, RowTracking)

The dropFeatureSupport method can be used as follows:

io.delta.tables.DeltaTable.dropFeatureSupport("rowTracking")

See online documentation for more details.

Generate a manifest for the given Delta Table

Get the information available commits on this table as a Spark DataFrame. The information is in reverse chronological order.

Get the information of the latest limit commits on this table as a Spark DataFrame. The information is in reverse chronological order.

Merge data from the source DataFrame based on the given merge condition. This returns a DeltaMergeBuilder object that can be used to specify the update, delete, or insert actions to be performed on rows based on whether the rows matched the condition or not.

See the DeltaMergeBuilder for a full description of this operation and what combinations of update, delete and insert operations are allowed.

Scala example to update a key-value Delta table with new key-values from a source DataFrame:

deltaTable
 .as("target")
 .merge(
   source.as("source"),
   "target.key = source.key")
 .whenMatched
 .updateExpr(Map(
   "value" -> "source.value"))
 .whenNotMatched
 .insertExpr(Map(
   "key" -> "source.key",
   "value" -> "source.value"))
 .execute()

Java example to update a key-value Delta table with new key-values from a source DataFrame:

deltaTable
 .as("target")
 .merge(
   source.as("source"),
   "target.key = source.key")
 .whenMatched
 .updateExpr(
    new HashMap<String, String>() {{
      put("value" -> "source.value")
    }})
 .whenNotMatched
 .insertExpr(
    new HashMap<String, String>() {{
     put("key", "source.key");
     put("value", "source.value");
   }})
 .execute()

Merge data from the source DataFrame based on the given merge condition. This returns a DeltaMergeBuilder object that can be used to specify the update, delete, or insert actions to be performed on rows based on whether the rows matched the condition or not.

See the DeltaMergeBuilder for a full description of this operation and what combinations of update, delete and insert operations are allowed.

Scala example to update a key-value Delta table with new key-values from a source DataFrame:

deltaTable
 .as("target")
 .merge(
   source.as("source"),
   "target.key = source.key")
 .whenMatched
 .updateExpr(Map(
   "value" -> "source.value"))
 .whenNotMatched
 .insertExpr(Map(
   "key" -> "source.key",
   "value" -> "source.value"))
 .execute()

Java example to update a key-value Delta table with new key-values from a source DataFrame:

deltaTable
 .as("target")
 .merge(
   source.as("source"),
   "target.key = source.key")
 .whenMatched
 .updateExpr(
    new HashMap<String, String>() {{
      put("value" -> "source.value");
    }})
 .whenNotMatched
 .insertExpr(
    new HashMap<String, String>() {{
     put("key", "source.key");
     put("value", "source.value");
   }})
 .execute();

Optimize the data layout of the table. This returns a DeltaOptimizeBuilder object that can be used to specify the partition filter to limit the scope of optimize and also execute different optimization techniques such as file compaction or order data using Z-Order curves.

See the DeltaOptimizeBuilder for a full description of this operation.

Scala example to run file compaction on a subset of partitions in the table:

deltaTable
 .optimize()
 .where("date='2021-11-18'")
 .executeCompaction();

Restore the DeltaTable to an older version of the table specified by a timestamp.

Timestamp can be of the format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss

An example would be

io.delta.tables.DeltaTable.restoreToTimestamp("2019-01-01")

Restore the DeltaTable to an older version of the table specified by version number.

An example would be

io.delta.tables.DeltaTable.restoreToVersion(7)

Get a DataFrame (that is, Dataset[Row]) representation of this Delta table.

Update data from the table on the rows that match the given condition based on the rules defined by set.

Java example to increment the column data.

import org.apache.spark.sql.Column;
import org.apache.spark.sql.functions;

deltaTable.update(
  functions.col("date").gt("2018-01-01"),
  new HashMap<String, Column>() {{
    put("data", functions.col("data").plus(1));
  }}
);

Update data from the table on the rows that match the given condition based on the rules defined by set.

Scala example to increment the column data.

import org.apache.spark.sql.functions._

deltaTable.update(
  col("date") > "2018-01-01",
  Map("data" -> col("data") + 1))

Update rows in the table based on the rules defined by set.

Java example to increment the column data.

import org.apache.spark.sql.Column;
import org.apache.spark.sql.functions;

deltaTable.update(
  new HashMap<String, Column>() {{
    put("data", functions.col("data").plus(1));
  }}
);

Update rows in the table based on the rules defined by set.

Scala example to increment the column data.

import org.apache.spark.sql.functions._

deltaTable.update(Map("data" -> col("data") + 1))

Update data from the table on the rows that match the given condition, which performs the rules defined by set.

Java example to increment the column data.

deltaTable.update(
  "date > '2018-01-01'",
  new HashMap<String, String>() {{
    put("data", "data + 1");
  }}
);

Update data from the table on the rows that match the given condition, which performs the rules defined by set.

Scala example to increment the column data.

deltaTable.update(
  "date > '2018-01-01'",
  Map("data" -> "data + 1"))

Update rows in the table based on the rules defined by set.

Java example to increment the column data.

deltaTable.updateExpr(
  new HashMap<String, String>() {{
    put("data", "data + 1");
  }}
);

Update rows in the table based on the rules defined by set.

Scala example to increment the column data.

deltaTable.updateExpr(Map("data" -> "data + 1")))

Updates the protocol version of the table to leverage new features. Upgrading the reader version will prevent all clients that have an older version of Delta Lake from accessing this table. Upgrading the writer version will prevent older versions of Delta Lake to write to this table. The reader or writer version cannot be downgraded.

See online documentation and Delta's protocol specification at PROTOCOL.md for more details.

Recursively delete files and directories in the table that are not needed by the table for maintaining older versions up to the given retention threshold. This method will return an empty DataFrame on successful completion.

note: This will use the default retention period of 7 days.

Recursively delete files and directories in the table that are not needed by the table for maintaining older versions up to the given retention threshold. This method will return an empty DataFrame on successful completion.