class DeltaMergeBuilder extends AnalysisHelper with Logging

Builder to specify how to merge data from source DataFrame into the target Delta table. You can specify any number of whenMatched and whenNotMatched clauses. Here are the constraints on these clauses.

  • whenMatched clauses:
    • The condition in a whenMatched clause is optional. However, if there are multiple whenMatched clauses, then only the last one may omit the condition.
    • When there are more than one whenMatched clauses and there are conditions (or the lack of) such that a row satisfies multiple clauses, then the action for the first clause satisfied is executed. In other words, the order of the whenMatched clauses matters.
    • If none of the whenMatched clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted.
    • If you want to update all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use the whenMatched(...).updateAll(). This is equivalent to
              whenMatched(...).updateExpr(Map(
                ("col1", "source.col1"),
                ("col2", "source.col2"),
                ...))
      
  • whenNotMatched clauses:
    • The condition in a whenNotMatched clause is optional. However, if there are multiple whenNotMatched clauses, then only the last one may omit the condition.
    • When there are more than one whenNotMatched clauses and there are conditions (or the lack of) such that a row satisfies multiple clauses, then the action for the first clause satisfied is executed. In other words, the order of the whenNotMatched clauses matters.
    • If no whenNotMatched clause is present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted.
    • If you want to insert all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use whenNotMatched(...).insertAll(). This is equivalent to
              whenNotMatched(...).insertExpr(Map(
                ("col1", "source.col1"),
                ("col2", "source.col2"),
                ...))
      
  • whenNotMatchedBySource clauses:
    • The condition in a whenNotMatchedBySource clause is optional. However, if there are multiple whenNotMatchedBySource clauses, then only the last one may omit the condition.
    • When there are more than one whenNotMatchedBySource clauses and there are conditions (or the lack of) such that a row satisfies multiple clauses, then the action for the first clause satisfied is executed. In other words, the order of the whenNotMatchedBySource clauses matters.
    • If no whenNotMatchedBySource clause is present or if it is present but the non-matching target row does not satisfy any of the whenNotMatchedBySource clause condition, then the target row will not be updated or deleted.

Scala example to update a key-value Delta table with new key-values from a source DataFrame:

deltaTable
 .as("target")
 .merge(
   source.as("source"),
   "target.key = source.key")
 .withSchemaEvolution()
 .whenMatched()
 .updateExpr(Map(
   "value" -> "source.value"))
 .whenNotMatched()
 .insertExpr(Map(
   "key" -> "source.key",
   "value" -> "source.value"))
 .whenNotMatchedBySource()
 .updateExpr(Map(
   "value" -> "target.value + 1"))
 .execute()

Java example to update a key-value Delta table with new key-values from a source DataFrame:

deltaTable
 .as("target")
 .merge(
   source.as("source"),
   "target.key = source.key")
 .withSchemaEvolution()
 .whenMatched()
 .updateExpr(
    new HashMap<String, String>() {{
      put("value", "source.value");
    }})
 .whenNotMatched()
 .insertExpr(
    new HashMap<String, String>() {{
     put("key", "source.key");
     put("value", "source.value");
   }})
 .whenNotMatchedBySource()
 .updateExpr(
    new HashMap<String, String>() {{
     put("value", "target.value + 1");
   }})
 .execute();
Since

0.3.0

Linear Supertypes
Logging, AnalysisHelper, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaMergeBuilder
  2. Logging
  3. AnalysisHelper
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DeltaMergeBuilder(targetTable: DeltaTable, source: DataFrame, onCondition: Column, whenClauses: Seq[DeltaMergeIntoClause])

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def execute(): DataFrame

    Execute the merge operation based on the built matched and not matched actions.

    Execute the merge operation based on the built matched and not matched actions.

    Since

    0.3.0

  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. def improveUnsupportedOpError[T](f: => T): T
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  12. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  13. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  16. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  17. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  30. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  39. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  40. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  41. def resolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planProvidingAttrs: LogicalPlan): Seq[Expression]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  42. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  43. def toDataset(sparkSession: SparkSession, logicalPlan: LogicalPlan): Dataset[Row]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  44. def toString(): String
    Definition Classes
    AnyRef → Any
  45. def tryResolveReferences(sparkSession: SparkSession)(expr: Expression, planContainingExpr: LogicalPlan): Expression
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  46. def tryResolveReferencesForExpressions(sparkSession: SparkSession)(exprs: Seq[Expression], plansProvidingAttrs: Seq[LogicalPlan]): Seq[Expression]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  47. def tryResolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planContainingExpr: LogicalPlan): Seq[Expression]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  48. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  49. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  50. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  51. def whenMatched(condition: Column): DeltaMergeMatchedActionBuilder

    Build the actions to perform when the merge condition was matched and the given condition is true.

    Build the actions to perform when the merge condition was matched and the given condition is true. This returns a DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.

    condition

    boolean expression as a Column object

    Since

    0.3.0

  52. def whenMatched(condition: String): DeltaMergeMatchedActionBuilder

    Build the actions to perform when the merge condition was matched and the given condition is true.

    Build the actions to perform when the merge condition was matched and the given condition is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.

    condition

    boolean expression as a SQL formatted string

    Since

    0.3.0

  53. def whenMatched(): DeltaMergeMatchedActionBuilder

    Build the actions to perform when the merge condition was matched.

    Build the actions to perform when the merge condition was matched. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.

    Since

    0.3.0

  54. def whenNotMatched(condition: Column): DeltaMergeNotMatchedActionBuilder

    Build the actions to perform when the merge condition was not matched and the given condition is true.

    Build the actions to perform when the merge condition was not matched and the given condition is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.

    condition

    boolean expression as a Column object

    Since

    0.3.0

  55. def whenNotMatched(condition: String): DeltaMergeNotMatchedActionBuilder

    Build the actions to perform when the merge condition was not matched and the given condition is true.

    Build the actions to perform when the merge condition was not matched and the given condition is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.

    condition

    boolean expression as a SQL formatted string

    Since

    0.3.0

  56. def whenNotMatched(): DeltaMergeNotMatchedActionBuilder

    Build the action to perform when the merge condition was not matched.

    Build the action to perform when the merge condition was not matched. This returns DeltaMergeNotMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.

    Since

    0.3.0

  57. def whenNotMatchedBySource(condition: Column): DeltaMergeNotMatchedBySourceActionBuilder

    Build the actions to perform when the merge condition was not matched by the source and the given condition is true.

    Build the actions to perform when the merge condition was not matched by the source and the given condition is true. This returns DeltaMergeNotMatchedBySourceActionBuilder object which can be used to specify how to update or delete the target table row .

    condition

    boolean expression as a Column object

    Since

    2.3.0

  58. def whenNotMatchedBySource(condition: String): DeltaMergeNotMatchedBySourceActionBuilder

    Build the actions to perform when the merge condition was not matched by the source and the given condition is true.

    Build the actions to perform when the merge condition was not matched by the source and the given condition is true. This returns DeltaMergeNotMatchedBySourceActionBuilder object which can be used to specify how to update or delete the target table row.

    condition

    boolean expression as a SQL formatted string

    Since

    2.3.0

  59. def whenNotMatchedBySource(): DeltaMergeNotMatchedBySourceActionBuilder

    Build the actions to perform when the merge condition was not matched by the source.

    Build the actions to perform when the merge condition was not matched by the source. This returns DeltaMergeNotMatchedBySourceActionBuilder object which can be used to specify how to update or delete the target table row.

    Since

    2.3.0

  60. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  61. def withSchemaEvolution(): DeltaMergeBuilder

    Enable schema evolution for the merge operation.

    Enable schema evolution for the merge operation. This allows the schema of the target table/columns to be automatically updated based on the schema of the source table/columns.

    Since

    3.2.0

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from AnalysisHelper

Inherited from AnyRef

Inherited from Any

Ungrouped