Type Members

1.  class DeltaMergeBuilder extends AnalysisHelper with Logging

   :: Evolving ::

   :: Evolving ::

   Builder to specify how to merge data from source DataFrame into the target Delta table. You can specify 1, 2 or 3 when clauses of which there can be at most 2 whenMatched clauses and at most 1 whenNotMatched clause. Here are the constraints on these clauses.

   • whenMatched clauses:
     • There can be at most one update action and one delete action in whenMatched clauses.
     • Each whenMatched clause can have an optional condition. However, if there are two whenMatched clauses, then the first one must have a condition.
     • When there are two whenMatched clauses and there are conditions (or the lack of) such that a row matches both clauses, then the first clause/action is executed. In other words, the order of the whenMatched clauses matter.
     • If none of the whenMatched clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted.
     • If you want to update all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use the whenMatched(...).updateAll(). This is equivalent to

               whenMatched(...).updateExpr(Map(
                 ("col1", "source.col1"),
                 ("col2", "source.col2"),
                 ...))
       

   • whenNotMatched clauses:
     • This clause can have only an insert action, which can have an optional condition.
     • If the whenNotMatched clause is not present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted.
     • If you want to insert all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use whenMatched(...).insertAll(). This is equivalent to

               whenMatched(...).insertExpr(Map(
                 ("col1", "source.col1"),
                 ("col2", "source.col2"),
                 ...))
       

   Scala example to update a key-value Delta table with new key-values from a source DataFrame:

   deltaTable
    .as("target")
    .merge(
      source.as("source"),
      "target.key = source.key")
    .whenMatched
    .updateExpr(Map(
      "value" -> "source.value"))
    .whenNotMatched
    .insertExpr(Map(
      "key" -> "source.key",
      "value" -> "source.value"))
    .execute()

   Java example to update a key-value Delta table with new key-values from a source DataFrame:

   deltaTable
    .as("target")
    .merge(
      source.as("source"),
      "target.key = source.key")
    .whenMatched
    .updateExpr(
       new HashMap<String, String>() {{
         put("value", "source.value");
       }})
    .whenNotMatched
    .insertExpr(
       new HashMap<String, String>() {{
        put("key", "source.key");
        put("value", "source.value");
      }})
    .execute();

   Annotations

   @Evolving()

   Since

   0.3.0

2.  class DeltaMergeMatchedActionBuilder extends AnyRef

   :: Evolving ::

   :: Evolving ::

   Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.

   See DeltaMergeBuilder for more information.

   Annotations

   @Evolving()

   Since

   0.3.0

3.  class DeltaMergeNotMatchedActionBuilder extends AnyRef

   :: Evolving ::

   :: Evolving ::

   Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.

   See DeltaMergeBuilder for more information.

   Annotations

   @Evolving()

   Since

   0.3.0

4.  class DeltaTable extends DeltaTableOperations with Serializable

   :: Evolving ::

   :: Evolving ::

   Main class for programmatically interacting with Delta tables. You can create DeltaTable instances using the static methods.

   DeltaTable.forPath(sparkSession, pathToTheDeltaTable)

   Annotations

   @Evolving()

   Since

   0.3.0