class DeltaMergeBuilder extends AnalysisHelper with Logging
Builder to specify how to merge data from source DataFrame into the target Delta table.
You can specify 1, 2 or 3 when
clauses of which there can be at most 2 whenMatched
clauses
and at most 1 whenNotMatched
clause. Here are the constraints on these clauses.
whenMatched
clauses:- There can be at most one
update
action and onedelete
action inwhenMatched
clauses. - Each
whenMatched
clause can have an optional condition. However, if there are twowhenMatched
clauses, then the first one must have a condition. - When there are two
whenMatched
clauses and there are conditions (or the lack of) such that a row matches both clauses, then the first clause/action is executed. In other words, the order of thewhenMatched
clauses matter. - If none of the
whenMatched
clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted. - If you want to update all the columns of the target Delta table with the
corresponding column of the source DataFrame, then you can use the
whenMatched(...).updateAll()
. This is equivalent towhenMatched(...).updateExpr(Map( ("col1", "source.col1"), ("col2", "source.col2"), ...))
- There can be at most one
whenNotMatched
clauses:- This clause can have only an
insert
action, which can have an optional condition. - If the
whenNotMatched
clause is not present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted. - If you want to insert all the columns of the target Delta table with the
corresponding column of the source DataFrame, then you can use
whenMatched(...).insertAll()
. This is equivalent towhenMatched(...).insertExpr(Map( ("col1", "source.col1"), ("col2", "source.col2"), ...))
- This clause can have only an
Scala example to update a key-value Delta table with new key-values from a source DataFrame:
deltaTable .as("target") .merge( source.as("source"), "target.key = source.key") .whenMatched .updateExpr(Map( "value" -> "source.value")) .whenNotMatched .insertExpr(Map( "key" -> "source.key", "value" -> "source.value")) .execute()
Java example to update a key-value Delta table with new key-values from a source DataFrame:
deltaTable .as("target") .merge( source.as("source"), "target.key = source.key") .whenMatched .updateExpr( new HashMap<String, String>() {{ put("value", "source.value"); }}) .whenNotMatched .insertExpr( new HashMap<String, String>() {{ put("key", "source.key"); put("value", "source.value"); }}) .execute();
- Since
0.3.0
- Alphabetic
- By Inheritance
- DeltaMergeBuilder
- Logging
- AnalysisHelper
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
execute(): Unit
Execute the merge operation based on the built matched and not matched actions.
Execute the merge operation based on the built matched and not matched actions.
- Since
0.3.0
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
improveUnsupportedOpError(f: ⇒ Unit): Unit
- Attributes
- protected
- Definition Classes
- AnalysisHelper
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
resolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planProvidingAttrs: LogicalPlan): Seq[Expression]
Resolve expressions using the attributes provided by
planProvidingAttrs
.Resolve expressions using the attributes provided by
planProvidingAttrs
. Throw an error if failing to resolve any expressions.- Attributes
- protected
- Definition Classes
- AnalysisHelper
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toDataset(sparkSession: SparkSession, logicalPlan: LogicalPlan): Dataset[Row]
- Attributes
- protected
- Definition Classes
- AnalysisHelper
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
tryResolveReferences(sparkSession: SparkSession)(expr: Expression, planContainingExpr: LogicalPlan): Expression
- Attributes
- protected
- Definition Classes
- AnalysisHelper
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
whenMatched(condition: Column): DeltaMergeMatchedActionBuilder
Build the actions to perform when the merge condition was matched and the given
condition
is true.Build the actions to perform when the merge condition was matched and the given
condition
is true. This returns a DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.- condition
boolean expression as a Column object
- Since
0.3.0
-
def
whenMatched(condition: String): DeltaMergeMatchedActionBuilder
Build the actions to perform when the merge condition was matched and the given
condition
is true.Build the actions to perform when the merge condition was matched and the given
condition
is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.- condition
boolean expression as a SQL formatted string
- Since
0.3.0
-
def
whenMatched(): DeltaMergeMatchedActionBuilder
Build the actions to perform when the merge condition was matched.
Build the actions to perform when the merge condition was matched. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.
- Since
0.3.0
-
def
whenNotMatched(condition: Column): DeltaMergeNotMatchedActionBuilder
Build the actions to perform when the merge condition was not matched and the given
condition
is true.Build the actions to perform when the merge condition was not matched and the given
condition
is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.- condition
boolean expression as a Column object
- Since
0.3.0
-
def
whenNotMatched(condition: String): DeltaMergeNotMatchedActionBuilder
Build the actions to perform when the merge condition was not matched and the given
condition
is true.Build the actions to perform when the merge condition was not matched and the given
condition
is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.- condition
boolean expression as a SQL formatted string
- Since
0.3.0
-
def
whenNotMatched(): DeltaMergeNotMatchedActionBuilder
Build the action to perform when the merge condition was not matched.
Build the action to perform when the merge condition was not matched. This returns DeltaMergeNotMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.
- Since
0.3.0