How does Delta Lake manage feature compatibility?
Many Delta Lake optimizations require enabling Delta Lake features on a table. Delta Lake features are always backwards compatible, so tables written by a lower Delta Lake version can always be read and written by a higher Delta Lake version. Enabling some features breaks forward compatibility with workloads running in a lower Delta Lake version. For features that break forward compatibility, you must update all workloads that reference the upgraded tables to use a compliant Delta Lake version.
What Delta Lake features require client upgrades?
The following Delta Lake features break forward compatibility. Features are enabled on a table-by-table basis.
Feature |
Requires Delta Lake version or later |
Documentation |
|---|---|---|
|
||
Generated columns |
||
Column mapping |
||
Change data feed |
||
Deletion vectors |
||
Table features |
||
Timestamp without Timezone |
What is a table protocol specification?
Every Delta table has a protocol specification which indicates the set of features that the table supports. The protocol specification is used by applications that read or write the table to determine if they can handle all the features that the table supports. If an application does not know how to handle a feature that is listed as supported in the protocol of a table, then that application is not be able to read or write that table.
The protocol specification is separated into two components: the read protocol and the write protocol.
Read protocol
The read protocol lists all features that a table supports and that an application must understand in order to read the table correctly. Upgrading the read protocol of a table requires that all reader applications support the added features.
Important
All applications that write to a Delta table must be able to construct a snapshot of the table. As such, workloads that write to Delta tables must respect both reader and writer protocol requirements.
If you encounter a protocol that is unsupported by a workload on Delta Lake, you must upgrade to a higher Delta Lake implementation with more comprehensive support.
Write protocol
The write protocol lists all features that a table supports and that an application must understand in order to write to the table correctly. Upgrading the write protocol of a table requires that all writer applications support the added features. It does not affect read-only applications, unless the read protocol is also upgraded.
Which protocols must be upgraded?
Some features require upgrading both the read protocol and the write protocol. Other features only require upgrading the write protocol.
As an example, support for CHECK constraints is a write protocol feature: only writing applications need to know about CHECK constraints and enforce them.
In contrast, column mapping requires upgrading both the read and write protocols. Because the data is stored differently in the table, reader applications must understand column mapping so they can read the data correctly.
For more on upgrading, see Upgrading protocol versions.
What are table features?
In Delta Lake 2.3.0 and above, Delta Lake table features introduce granular flags specifying which features are supported by a given table. Table features are the successor to protocol versions and are designed with the goal of improved flexibility for clients that read and write Delta Lake. See What is a protocol version?.
Note
Table features have protocol version requirements. See Features by protocol version.
A Delta table feature is a marker that indicates that the table supports a particular feature. Every feature is either a write protocol feature (meaning it only upgrades the write protocol) or a read/write protocol feature (meaning both read and write protocols are upgraded to enable the feature).
To learn more about supported table features in Delta Lake, see the Delta Lake protocol.
Do table features change how Delta Lake features are enabled?
If you only interact with Delta tables through Delta Lake, you can continue to track support for Delta Lake features using minimum Delta Lake requirements. If you read and write from Delta tables using other systems, you might need to consider how table features impact compatibility, because there is a risk that the system could not understand the upgraded protocol versions.
What is a protocol version?
A protocol version is a protocol number that indicates a particular grouping of table features. In Delta Lake 2.3.0 and below, you cannot enable table features individually. Protocol versions bundle a group of features.
Delta tables specify a separate protocol version for read protocol and write protocol. The transaction log for a Delta table contains protocol versioning information that supports Delta Lake evolution.
The protocol versions bundle all features from previous protocols. See Features by protocol version.
Note
Starting with writer version 7 and reader version 3, Delta Lake has introduced the concept of table features. Using table features, you can now choose to only enable those features that are supported by other clients in your data ecosystem. See What are table features?.
Features by protocol version
The following table shows minimum protocol versions required for Delta Lake features.
Feature |
|
|
Documentation |
|---|---|---|---|
Basic functionality |
2 |
1 |
|
|
3 |
1 |
|
Change data feed |
4 |
1 |
|
Generated columns |
4 |
1 |
|
Column mapping |
5 |
2 |
|
Table features read |
7 |
1 |
|
Table features write |
7 |
3 |
|
Deletion vectors |
7 |
3 |
|
Timestamp without Timezone |
7 |
3 |
Upgrading protocol versions
You can choose to manually update a table to a newer protocol version. We recommend using the lowest protocol versions that support the Delta Lake features required for your table. Upgrading the writer protocol might cause less disruption than upgrading the reader protocol since systems and workloads using older Delta Lake versions can still read from tables, even if they do not support the updated writer protocol.
Warning
Protocol version upgrades are irreversible, and upgrading the protocol version might break the existing Delta Lake table readers, writers, or both. We recommend you upgrade specific tables only when needed, such as to opt-in to new features in Delta Lake. You should also check to make sure that all of your current and future production tools support Delta Lake tables with the new protocol version.
To upgrade a table to a newer protocol version, use the DeltaTable.upgradeTableProtocol method:
-- Upgrades the reader protocol version to 1 and the writer protocol version to 3.
ALTER TABLE <table_identifier> SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3')
from delta.tables import DeltaTable
delta = DeltaTable.forPath(spark, "path_to_table") # or DeltaTable.forName
delta.upgradeTableProtocol(1, 3) # upgrades to readerVersion=1, writerVersion=3
import io.delta.tables.DeltaTable
val delta = DeltaTable.forPath(spark, "path_to_table") // or DeltaTable.forName
delta.upgradeTableProtocol(1, 3) // Upgrades to readerVersion=1, writerVersion=3.