io.delta.flink.source

Class RowDataContinuousDeltaSourceBuilder

Object

<any>

io.delta.flink.source.RowDataContinuousDeltaSourceBuilder

public class RowDataContinuousDeltaSourceBuilder
extends <any>

A builder class for DeltaSource for a stream of RowData where the created source instance will operate in Continuous mode.

In Continuous mode, the DeltaSource will, by default, load the full state of the latest table version, and then start monitoring for changes. If you use either the startingVersion(java.lang.String) or startingTimestamp(java.lang.String) APIs, then the DeltaSource will start monitoring for changes from that historical version. It will not load the full table state at that historical table version.

For most common use cases use DeltaSource.forContinuousRowData(org.apache.flink.core.fs.Path, org.apache.hadoop.conf.Configuration) utility method to instantiate the source. After instantiation of this builder you can either call RowDataBoundedDeltaSourceBuilder.build() method to get the instance of a DeltaSource or configure additional options using builder's API.

Method Summary

Modifier and Type	Method and Description
DeltaSource<org.apache.flink.table.data.RowData>	build() Creates an instance of DeltaSource for a stream of RowData.
RowDataContinuousDeltaSourceBuilder	columnNames(java.util.List<String> columnNames) Specifies a List of column names that should be read from Delta table.
RowDataContinuousDeltaSourceBuilder	columnNames(String... columnNames) Specifies an array of column names that should be read from Delta table.
RowDataContinuousDeltaSourceBuilder	ignoreChanges(boolean ignoreChanges) Sets the "ignoreChanges" option.
RowDataContinuousDeltaSourceBuilder	ignoreDeletes(boolean ignoreDeletes) Sets the "ignoreDeletes" option.
RowDataContinuousDeltaSourceBuilder	option(String optionName, boolean optionValue) Sets a configuration option.
RowDataContinuousDeltaSourceBuilder	option(String optionName, int optionValue) Sets a configuration option.
RowDataContinuousDeltaSourceBuilder	option(String optionName, long optionValue) Sets a configuration option.
RowDataContinuousDeltaSourceBuilder	option(String optionName, String optionValue) Sets a configuration option.
RowDataContinuousDeltaSourceBuilder	startingTimestamp(String startingTimestamp) Sets value of "startingTimestamp" option.
RowDataContinuousDeltaSourceBuilder	startingVersion(long startingVersion) Sets value of "startingVersion" option.
RowDataContinuousDeltaSourceBuilder	startingVersion(String startingVersion) Sets value of "startingVersion" option.
RowDataContinuousDeltaSourceBuilder	updateCheckIntervalMillis(long updateCheckInterval) Sets the value for "updateCheckIntervalMillis" option.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

columnNames

public RowDataContinuousDeltaSourceBuilder columnNames(java.util.List<String> columnNames)

Specifies a List of column names that should be read from Delta table. If this method is not used, Source will read all columns from Delta table.

If provided List is null or contains null, empty or blank elements it will throw a DeltaSourceValidationException by builder after calling build() method.

Parameters:

columnNames - column names that should be read.

columnNames

public RowDataContinuousDeltaSourceBuilder columnNames(String... columnNames)

Specifies an array of column names that should be read from Delta table. If this method is not used, Source will read all columns from Delta table.

If provided List is null or contains null, empty or blank elements it will throw a DeltaSourceValidationException by builder after calling build() method.

Parameters:

columnNames - column names that should be read.

startingVersion

public RowDataContinuousDeltaSourceBuilder startingVersion(String startingVersion)

Sets value of "startingVersion" option. This option specifies the starting table version from which we want to start reading changes.

This option is mutually exclusive with startingTimestamp(String) option.

Parameters:

startingVersion - Delta table version to start reading changes from. The values can be string numbers like "1", "10" etc. or keyword "latest", where in that case, changes from the latest Delta table version will be read.

startingVersion

public RowDataContinuousDeltaSourceBuilder startingVersion(long startingVersion)

Sets value of "startingVersion" option. This option specifies the starting table version from which we want to start reading changes.

This option is mutually exclusive with startingTimestamp(String) option.

Parameters:

startingVersion - Delta table version to start reading changes from.

startingTimestamp

public RowDataContinuousDeltaSourceBuilder startingTimestamp(String startingTimestamp)

Sets value of "startingTimestamp" option. This option is used to read only changes starting from the table version that was generated at or after the given timestamp.

This option is mutually exclusive with startingVersion(String) and startingVersion(long) option.

Parameters:

startingTimestamp - The timestamp of the table from which we start reading changes. Supported formats are:

• 2022-02-24
• 2022-02-24 04:55:00
• 2022-02-24 04:55:00.001
• 2022-02-24T04:55:00
• 2022-02-24T04:55:00.001
• 2022-02-24T04:55:00.001Z

updateCheckIntervalMillis

public RowDataContinuousDeltaSourceBuilder updateCheckIntervalMillis(long updateCheckInterval)

Sets the value for "updateCheckIntervalMillis" option. This option is used to specify the check interval (in milliseconds) used for periodic Delta table changes checks.

The default value for this option is 5000 ms.

Parameters:

updateCheckInterval - The update check internal in milliseconds.

ignoreDeletes

public RowDataContinuousDeltaSourceBuilder ignoreDeletes(boolean ignoreDeletes)

Sets the "ignoreDeletes" option. When set to true, this option allows processing Delta table versions where data is deleted.

The default value for this option is false.

ignoreChanges

public RowDataContinuousDeltaSourceBuilder ignoreChanges(boolean ignoreChanges)

Sets the "ignoreChanges" option. When set to true, this option allows processing Delta table versions where data is changed (i.e. updated) or deleted.

Note that setting this option to true can lead to duplicate processing of data, as, in the case of updates, existing rows may be rewritten in new files, and those new files will be treated as new data and be fully reprocessed.

This option subsumes ignoreDeletes(boolean) option. Therefore, if you set "ignoreChanges" to true, your stream will not be disrupted by either deletions or updates to the source table.

The default value for this option is false.

option

public RowDataContinuousDeltaSourceBuilder option(String optionName,
                                                  String optionValue)

Sets a configuration option.

Parameters:

optionName - Option name to set.

optionValue - Option String value to set.

option

public RowDataContinuousDeltaSourceBuilder option(String optionName,
                                                  boolean optionValue)

Sets a configuration option.

Parameters:

optionName - Option name to set.

optionValue - Option boolean value to set.

option

public RowDataContinuousDeltaSourceBuilder option(String optionName,
                                                  int optionValue)

Sets a configuration option.

Parameters:

optionName - Option name to set.

optionValue - Option int value to set.

option

public RowDataContinuousDeltaSourceBuilder option(String optionName,
                                                  long optionValue)

Sets a configuration option.

Parameters:

optionName - Option name to set.

optionValue - Option long value to set.

build

public DeltaSource<org.apache.flink.table.data.RowData> build()

Creates an instance of DeltaSource for a stream of RowData. Created source will work in Continuous mode, actively monitoring Delta table for new changes.

This method can throw DeltaSourceValidationException in case of invalid arguments passed to Delta source builder.

Returns:

New DeltaSource instance.