Type Members

1.  class DeltaColumnBuilder extends AnyRef

   :: Evolving ::

   :: Evolving ::

   Builder to specify a table column.

   See DeltaTableBuilder for examples.

   Annotations

   @Evolving()

   Since

   1.0.0

2.  class DeltaMergeBuilder extends AnalysisHelper with Logging

   Builder to specify how to merge data from source DataFrame into the target Delta table.

   Builder to specify how to merge data from source DataFrame into the target Delta table. You can specify any number of whenMatched and whenNotMatched clauses. Here are the constraints on these clauses.

   • whenMatched clauses:
     • The condition in a whenMatched clause is optional. However, if there are multiple whenMatched clauses, then only the last one may omit the condition.
     • When there are more than one whenMatched clauses and there are conditions (or the lack of) such that a row satisfies multiple clauses, then the action for the first clause satisfied is executed. In other words, the order of the whenMatched clauses matters.
     • If none of the whenMatched clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted.
     • If you want to update all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use the whenMatched(...).updateAll(). This is equivalent to

               whenMatched(...).updateExpr(Map(
                 ("col1", "source.col1"),
                 ("col2", "source.col2"),
                 ...))
       

   • whenNotMatched clauses:
     • The condition in a whenNotMatched clause is optional. However, if there are multiple whenNotMatched clauses, then only the last one may omit the condition.
     • When there are more than one whenNotMatched clauses and there are conditions (or the lack of) such that a row satisfies multiple clauses, then the action for the first clause satisfied is executed. In other words, the order of the whenNotMatched clauses matters.
     • If no whenNotMatched clause is present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted.
     • If you want to insert all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use whenNotMatched(...).insertAll(). This is equivalent to

               whenNotMatched(...).insertExpr(Map(
                 ("col1", "source.col1"),
                 ("col2", "source.col2"),
                 ...))
       

   • whenNotMatchedBySource clauses:
     • The condition in a whenNotMatchedBySource clause is optional. However, if there are multiple whenNotMatchedBySource clauses, then only the last one may omit the condition.
     • When there are more than one whenNotMatchedBySource clauses and there are conditions (or the lack of) such that a row satisfies multiple clauses, then the action for the first clause satisfied is executed. In other words, the order of the whenNotMatchedBySource clauses matters.
     • If no whenNotMatchedBySource clause is present or if it is present but the non-matching target row does not satisfy any of the whenNotMatchedBySource clause condition, then the target row will not be updated or deleted.

   Scala example to update a key-value Delta table with new key-values from a source DataFrame:

   deltaTable
    .as("target")
    .merge(
      source.as("source"),
      "target.key = source.key")
    .withSchemaEvolution()
    .whenMatched()
    .updateExpr(Map(
      "value" -> "source.value"))
    .whenNotMatched()
    .insertExpr(Map(
      "key" -> "source.key",
      "value" -> "source.value"))
    .whenNotMatchedBySource()
    .updateExpr(Map(
      "value" -> "target.value + 1"))
    .execute()

   Java example to update a key-value Delta table with new key-values from a source DataFrame:

   deltaTable
    .as("target")
    .merge(
      source.as("source"),
      "target.key = source.key")
    .withSchemaEvolution()
    .whenMatched()
    .updateExpr(
       new HashMap<String, String>() {{
         put("value", "source.value");
       }})
    .whenNotMatched()
    .insertExpr(
       new HashMap<String, String>() {{
        put("key", "source.key");
        put("value", "source.value");
      }})
    .whenNotMatchedBySource()
    .updateExpr(
       new HashMap<String, String>() {{
        put("value", "target.value + 1");
      }})
    .execute();

   Since

   0.3.0

3.  class DeltaMergeMatchedActionBuilder extends AnyRef

   Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.

   Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.

   See DeltaMergeBuilder for more information.

   Since

   0.3.0

4.  class DeltaMergeNotMatchedActionBuilder extends AnyRef

   Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.

   Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.

   See DeltaMergeBuilder for more information.

   Since

   0.3.0

5.  class DeltaMergeNotMatchedBySourceActionBuilder extends AnyRef

   Builder class to specify the actions to perform when a target table row has no match in the source table based on the given merge condition and optional match condition.

   Builder class to specify the actions to perform when a target table row has no match in the source table based on the given merge condition and optional match condition.

   See DeltaMergeBuilder for more information.

   Since

   2.3.0

6.  class DeltaOptimizeBuilder extends AnalysisHelper

   Builder class for constructing OPTIMIZE command and executing.

   Builder class for constructing OPTIMIZE command and executing.

   Since

   2.0.0

7.  class DeltaTable extends DeltaTableOperations with Serializable

   Main class for programmatically interacting with Delta tables.

   Main class for programmatically interacting with Delta tables. You can create DeltaTable instances using the static methods.

   DeltaTable.forPath(sparkSession, pathToTheDeltaTable)

   Since

   0.3.0

8.  class DeltaTableBuilder extends AnyRef

   :: Evolving ::

   :: Evolving ::

   Builder to specify how to create / replace a Delta table. You must specify the table name or the path before executing the builder. You can specify the table columns, the partitioning columns, the location of the data, the table comment and the property, and how you want to create / replace the Delta table.

   After executing the builder, an instance of DeltaTable is returned.

   Scala example to create a Delta table with generated columns, using the table name:

   val table: DeltaTable = DeltaTable.create()
     .tableName("testTable")
     .addColumn("c1",  dataType = "INT", nullable = false)
     .addColumn(
       DeltaTable.columnBuilder("c2")
         .dataType("INT")
         .generatedAlwaysAs("c1 + 10")
         .build()
     )
     .addColumn(
       DeltaTable.columnBuilder("c3")
         .dataType("INT")
         .comment("comment")
         .nullable(true)
         .build()
     )
     .partitionedBy("c1", "c2")
     .execute()

   Scala example to create a delta table using the location:

   val table: DeltaTable = DeltaTable.createIfNotExists(spark)
     .location("/foo/`bar`")
     .addColumn("c1", dataType = "INT", nullable = false)
     .addColumn(
       DeltaTable.columnBuilder(spark, "c2")
         .dataType("INT")
         .generatedAlwaysAs("c1 + 10")
         .build()
     )
     .addColumn(
       DeltaTable.columnBuilder(spark, "c3")
         .dataType("INT")
         .comment("comment")
         .nullable(true)
         .build()
     )
     .partitionedBy("c1", "c2")
     .execute()

   Java Example to replace a table:

   DeltaTable table = DeltaTable.replace()
     .tableName("db.table")
     .addColumn("c1",  "INT", false)
     .addColumn(
       DeltaTable.columnBuilder("c2")
         .dataType("INT")
         .generatedAlwaysBy("c1 + 10")
         .build()
     )
     .execute();

   Annotations

   @Evolving()

   Since

   1.0.0