io.delta.tables

Class DeltaMergeBuilder

Object

io.delta.tables.DeltaMergeBuilder

public class DeltaMergeBuilder
extends Object

:: Evolving ::

Builder to specify how to merge data from source DataFrame into the target Delta table. You can specify 1, 2 or 3 when clauses of which there can be at most 2 whenMatched clauses and at most 1 whenNotMatched clause. Here are the constraints on these clauses.

- whenMatched clauses:

- There can be at most one update action and one delete action in whenMatched clauses.

- Each whenMatched clause can have an optional condition. However, if there are two whenMatched clauses, then the first one must have a condition.

- When there are two whenMatched clauses and there are conditions (or the lack of) such that a row matches both clauses, then the first clause/action is executed. In other words, the order of the whenMatched clauses matter.

- If none of the whenMatched clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted.

- If you want to update all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use the whenMatched(...).updateAll(). This is equivalent to

         whenMatched(...).updateExpr(Map(
           ("col1", "source.col1"),
           ("col2", "source.col2"),
           ...))
       

- whenNotMatched clauses:

- This clause can have only an insert action, which can have an optional condition.

- If the whenNotMatched clause is not present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted.

- If you want to insert all the columns of the target Delta table with the corresponding column of the source DataFrame, then you can use whenMatched(...).insertAll(). This is equivalent to

         whenMatched(...).insertExpr(Map(
           ("col1", "source.col1"),
           ("col2", "source.col2"),
           ...))
       

Scala example to update a key-value Delta table with new key-values from a source DataFrame:

    deltaTable
     .as("target")
     .merge(
       source.as("source"),
       "target.key = source.key")
     .whenMatched
     .updateExpr(Map(
       "value" -> "source.value"))
     .whenNotMatched
     .insertExpr(Map(
       "key" -> "source.key",
       "value" -> "source.value"))
     .execute()
 

Java example to update a key-value Delta table with new key-values from a source DataFrame:

    deltaTable
     .as("target")
     .merge(
       source.as("source"),
       "target.key = source.key")
     .whenMatched
     .updateExpr(
        new HashMap<String, String>() {{
          put("value", "source.value");
        }})
     .whenNotMatched
     .insertExpr(
        new HashMap<String, String>() {{
         put("key", "source.key");
         put("value", "source.value");
       }})
     .execute();
 

Since:

0.3.0

Method Summary

Modifier and Type	Method and Description
void	execute() :: Evolving ::
static void	logConsole(String line)
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
static void	recordEvent(com.databricks.spark.util.MetricDefinition metric, scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> additionalTags, String blob, boolean trimBlob)
static scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String>	recordEvent$default$2()
static String	recordEvent$default$3()
static boolean	recordEvent$default$4()
static <S> S	recordOperation(com.databricks.spark.util.OpType opType, String opTarget, scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> extraTags, boolean isSynchronous, boolean alwaysRecordStats, boolean allowAuthTags, boolean killJvmIfStuck, com.databricks.spark.util.MetricDefinition outputMetric, boolean silent, scala.Function0<S> thunk)
static <S> String	recordOperation$default$2()
static <S> boolean	recordOperation$default$4()
static <S> boolean	recordOperation$default$5()
static <S> boolean	recordOperation$default$6()
static <S> boolean	recordOperation$default$7()
static <S> com.databricks.spark.util.MetricDefinition	recordOperation$default$8()
static <S> boolean	recordOperation$default$9()
static void	recordUsage(com.databricks.spark.util.MetricDefinition metric, double quantity, scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> additionalTags, String blob, boolean forceSample, boolean trimBlob, boolean silent)
static scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String>	recordUsage$default$3()
static String	recordUsage$default$4()
static boolean	recordUsage$default$5()
static boolean	recordUsage$default$6()
static boolean	recordUsage$default$7()
DeltaMergeMatchedActionBuilder	whenMatched() :: Evolving ::
DeltaMergeMatchedActionBuilder	whenMatched(org.apache.spark.sql.Column condition) :: Evolving ::
DeltaMergeMatchedActionBuilder	whenMatched(String condition) :: Evolving ::
DeltaMergeNotMatchedActionBuilder	whenNotMatched() :: Evolving ::
DeltaMergeNotMatchedActionBuilder	whenNotMatched(org.apache.spark.sql.Column condition) :: Evolving ::
DeltaMergeNotMatchedActionBuilder	whenNotMatched(String condition) :: Evolving ::
static <T> T	withStatusCode(String statusCode, String defaultMessage, scala.collection.immutable.Map<String,Object> data, scala.Function0<T> body)
static <T> scala.collection.immutable.Map<String,Object>	withStatusCode$default$3()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

withStatusCode

public static <T> T withStatusCode(String statusCode,
                                   String defaultMessage,
                                   scala.collection.immutable.Map<String,Object> data,
                                   scala.Function0<T> body)

withStatusCode$default$3

public static <T> scala.collection.immutable.Map<String,Object> withStatusCode$default$3()

logConsole

public static void logConsole(String line)

recordUsage

public static void recordUsage(com.databricks.spark.util.MetricDefinition metric,
                               double quantity,
                               scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> additionalTags,
                               String blob,
                               boolean forceSample,
                               boolean trimBlob,
                               boolean silent)

recordUsage$default$3

public static scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> recordUsage$default$3()

recordUsage$default$4

public static String recordUsage$default$4()

recordUsage$default$5

public static boolean recordUsage$default$5()

recordUsage$default$6

public static boolean recordUsage$default$6()

recordUsage$default$7

public static boolean recordUsage$default$7()

recordEvent

public static void recordEvent(com.databricks.spark.util.MetricDefinition metric,
                               scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> additionalTags,
                               String blob,
                               boolean trimBlob)

recordEvent$default$2

public static scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> recordEvent$default$2()

recordEvent$default$3

public static String recordEvent$default$3()

recordEvent$default$4

public static boolean recordEvent$default$4()

recordOperation

public static <S> S recordOperation(com.databricks.spark.util.OpType opType,
                                    String opTarget,
                                    scala.collection.immutable.Map<com.databricks.spark.util.TagDefinition,String> extraTags,
                                    boolean isSynchronous,
                                    boolean alwaysRecordStats,
                                    boolean allowAuthTags,
                                    boolean killJvmIfStuck,
                                    com.databricks.spark.util.MetricDefinition outputMetric,
                                    boolean silent,
                                    scala.Function0<S> thunk)

recordOperation$default$2

public static <S> String recordOperation$default$2()

recordOperation$default$4

public static <S> boolean recordOperation$default$4()

recordOperation$default$5

public static <S> boolean recordOperation$default$5()

recordOperation$default$6

public static <S> boolean recordOperation$default$6()

recordOperation$default$7

public static <S> boolean recordOperation$default$7()

recordOperation$default$8

public static <S> com.databricks.spark.util.MetricDefinition recordOperation$default$8()

recordOperation$default$9

public static <S> boolean recordOperation$default$9()

whenMatched

public DeltaMergeMatchedActionBuilder whenMatched()

:: Evolving ::

Build the actions to perform when the merge condition was matched. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.

Since:

0.3.0

whenMatched

public DeltaMergeMatchedActionBuilder whenMatched(String condition)

:: Evolving ::

Build the actions to perform when the merge condition was matched and the given condition is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.

Parameters:

condition - boolean expression as a SQL formatted string

Since:

0.3.0

whenMatched

public DeltaMergeMatchedActionBuilder whenMatched(org.apache.spark.sql.Column condition)

:: Evolving ::

Build the actions to perform when the merge condition was matched and the given condition is true. This returns a DeltaMergeMatchedActionBuilder object which can be used to specify how to update or delete the matched target table row with the source row.

Parameters:

condition - boolean expression as a Column object

Since:

0.3.0

whenNotMatched

public DeltaMergeNotMatchedActionBuilder whenNotMatched()

:: Evolving ::

Build the action to perform when the merge condition was not matched. This returns DeltaMergeNotMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.

Since:

0.3.0

whenNotMatched

public DeltaMergeNotMatchedActionBuilder whenNotMatched(String condition)

:: Evolving ::

Build the actions to perform when the merge condition was not matched and the given condition is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.

Parameters:

condition - boolean expression as a SQL formatted string

Since:

0.3.0

whenNotMatched

public DeltaMergeNotMatchedActionBuilder whenNotMatched(org.apache.spark.sql.Column condition)

:: Evolving ::

Build the actions to perform when the merge condition was not matched and the given condition is true. This returns DeltaMergeMatchedActionBuilder object which can be used to specify how to insert the new sourced row into the target table.

Parameters:

condition - boolean expression as a Column object

Since:

0.3.0

execute

public void execute()

:: Evolving ::

Execute the merge operation based on the built matched and not matched actions.

Since:

0.3.0