package tables
- Alphabetic
- Public
- All
Type Members
-
class
DeltaColumnBuilder extends AnyRef
:: Evolving ::
- Annotations
- @Evolving()
- Since
1.0.0
-
class
DeltaMergeBuilder extends AnalysisHelper with Logging
Builder to specify how to merge data from source DataFrame into the target Delta table.
Builder to specify how to merge data from source DataFrame into the target Delta table. You can specify 1, 2 or 3
when
clauses of which there can be at most 2whenMatched
clauses and at most 1whenNotMatched
clause. Here are the constraints on these clauses.whenMatched
clauses:- There can be at most one
update
action and onedelete
action inwhenMatched
clauses. - Each
whenMatched
clause can have an optional condition. However, if there are twowhenMatched
clauses, then the first one must have a condition. - When there are two
whenMatched
clauses and there are conditions (or the lack of) such that a row matches both clauses, then the first clause/action is executed. In other words, the order of thewhenMatched
clauses matter. - If none of the
whenMatched
clauses match a source-target row pair that satisfy the merge condition, then the target rows will not be updated or deleted. - If you want to update all the columns of the target Delta table with the
corresponding column of the source DataFrame, then you can use the
whenMatched(...).updateAll()
. This is equivalent towhenMatched(...).updateExpr(Map( ("col1", "source.col1"), ("col2", "source.col2"), ...))
- There can be at most one
whenNotMatched
clauses:- This clause can have only an
insert
action, which can have an optional condition. - If the
whenNotMatched
clause is not present or if it is present but the non-matching source row does not satisfy the condition, then the source row is not inserted. - If you want to insert all the columns of the target Delta table with the
corresponding column of the source DataFrame, then you can use
whenMatched(...).insertAll()
. This is equivalent towhenMatched(...).insertExpr(Map( ("col1", "source.col1"), ("col2", "source.col2"), ...))
- This clause can have only an
Scala example to update a key-value Delta table with new key-values from a source DataFrame:
deltaTable .as("target") .merge( source.as("source"), "target.key = source.key") .whenMatched .updateExpr(Map( "value" -> "source.value")) .whenNotMatched .insertExpr(Map( "key" -> "source.key", "value" -> "source.value")) .execute()
Java example to update a key-value Delta table with new key-values from a source DataFrame:
deltaTable .as("target") .merge( source.as("source"), "target.key = source.key") .whenMatched .updateExpr( new HashMap<String, String>() {{ put("value", "source.value"); }}) .whenNotMatched .insertExpr( new HashMap<String, String>() {{ put("key", "source.key"); put("value", "source.value"); }}) .execute();
- Since
0.3.0
-
class
DeltaMergeMatchedActionBuilder extends AnyRef
Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.
Builder class to specify the actions to perform when a target table row has matched a source row based on the given merge condition and optional match condition.
See DeltaMergeBuilder for more information.
- Since
0.3.0
-
class
DeltaMergeNotMatchedActionBuilder extends AnyRef
Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.
Builder class to specify the actions to perform when a source row has not matched any target Delta table row based on the merge condition, but has matched the additional condition if specified.
See DeltaMergeBuilder for more information.
- Since
0.3.0
-
class
DeltaTable extends DeltaTableOperations with Serializable
Main class for programmatically interacting with Delta tables.
Main class for programmatically interacting with Delta tables. You can create DeltaTable instances using the static methods.
DeltaTable.forPath(sparkSession, pathToTheDeltaTable)
- Since
0.3.0
-
class
DeltaTableBuilder extends AnyRef
:: Evolving ::
:: Evolving ::
Builder to specify how to create / replace a Delta table. You must specify the table name or the path before executing the builder. You can specify the table columns, the partitioning columns, the location of the data, the table comment and the property, and how you want to create / replace the Delta table.
After executing the builder, an instance of DeltaTable is returned.
Scala example to create a Delta table with generated columns, using the table name:
val table: DeltaTable = DeltaTable.create() .tableName("testTable") .addColumn("c1", dataType = "INT", nullable = false) .addColumn( DeltaTable.columnBuilder("c2") .dataType("INT") .generatedAlwaysAs("c1 + 10") .build() ) .addColumn( DeltaTable.columnBuilder("c3") .dataType("INT") .comment("comment") .nullable(true) .build() ) .partitionedBy("c1", "c2") .execute()
Scala example to create a delta table using the location:
val table: DeltaTable = DeltaTable.createIfNotExists(spark) .location("/foo/`bar`") .addColumn("c1", dataType = "INT", nullable = false) .addColumn( DeltaTable.columnBuilder(spark, "c2") .dataType("INT") .generatedAlwaysAs("c1 + 10") .build() ) .addColumn( DeltaTable.columnBuilder(spark, "c3") .dataType("INT") .comment("comment") .nullable(true) .build() ) .partitionedBy("c1", "c2") .execute()
Java Example to replace a table:
DeltaTable table = DeltaTable.replace() .tableName("db.table") .addColumn("c1", "INT", false) .addColumn( DeltaTable.columnBuilder("c2") .dataType("INT") .generatedAlwaysBy("c1 + 10") .build() ) .execute();
- Annotations
- @Evolving()
- Since
1.0.0
Value Members
- object DeltaMergeBuilder
- object DeltaMergeMatchedActionBuilder
- object DeltaMergeNotMatchedActionBuilder
-
object
DeltaTable extends Serializable
Companion object to create DeltaTable instances.
Companion object to create DeltaTable instances.
DeltaTable.forPath(sparkSession, pathToTheDeltaTable)
- Since
0.3.0