FFI (C/C++ integration)

The delta_kernel_ffi crate exposes delta-kernel-rs to C and C++ through a stable FFI boundary. It uses cbindgen to generate header files (.h and .hpp) at build time. This matters because it lets you build a connector in any language that can call C functions, not only Rust.

Building the FFI crate

The crate can be built as a shared library (cdylib) or static library (staticlib):

# Shared library (e.g. libdelta_kernel_ffi.so / .dylib / .dll)
cargo build -p delta_kernel_ffi --release

# Generated headers are written to target/ffi-headers/
# - delta_kernel_ffi.h   (C)
# - delta_kernel_ffi.hpp (C++)

Feature flags

Note: You must enable exactly one of default-engine-rustls or default-engine-native-tls. The default-engine-base feature contains shared implementation details and is not meant to be enabled directly.

The handle system

Objects that cross the FFI boundary are wrapped in handles. These are opaque pointers that carry ownership semantics. There are two kinds:

• Mutable handles (Box-like) represent exclusive ownership. Dropping the handle drops the underlying object. These are neither Copy nor Clone.
• Shared handles (Arc-like) represent shared ownership. Dropping the handle only drops the underlying object if it was the last reference.

Every handle has a corresponding free_* function that you must call to release it. For example, free_engine, free_snapshot, free_scan, free_transaction.

Several FFI functions consume their handle argument and return a new handle. After calling such a function, you must not use the old handle. The function documentation notes this with “CONSUMES the handle.”

Core API surface

The FFI mirrors the Rust API. A typical read flow looks like (no transaction is needed for reads):

get_default_engine()        ->  Handle<SharedExternEngine>
        |
get_snapshot_builder()      ->  Handle<MutableFfiSnapshotBuilder>
        |
snapshot_builder_build()    ->  Handle<SharedSnapshot>
        |
      scan()                ->  Handle<SharedScan>
        |
scan_metadata_iter_init()   ->  Handle<SharedScanMetadataIterator>
        |
  (read parquet, apply transforms, apply selection vectors)

A typical write flow:

get_default_engine()  ->  Handle<SharedExternEngine>
        |
    transaction()     ->  Handle<ExclusiveTransaction>
        |
  with_engine_info()  ->  Handle<ExclusiveTransaction>
        |
    add_files()
        |
    commit()          ->  ExternResult<u64>  (committed version)

For more control over scans, you can use the scan builder API instead of the convenience scan() function:

scan_builder()                ->  Handle<ExclusiveScanBuilder>
        |
scan_builder_with_predicate() ->  Handle<ExclusiveScanBuilder>
        |
scan_builder_with_schema()    ->  Handle<ExclusiveScanBuilder>
        |
scan_builder_build()          ->  Handle<SharedScan>

Public FFI functions

The tables below group the stable FFI functions by purpose. Unless noted, each function is available with the default feature flags. For the full, authoritative list and signatures, consult the generated delta_kernel_ffi.h header.

Engine creation

Snapshots

Snapshot inspection

Use these on a Handle<SharedSnapshot> to read table metadata, protocol, and partition columns without building a scan.

Schema and metadata visitors

Kernel exposes schemas, protocols, and metadata through visitor callbacks so the engine can materialize them into its own types without Kernel allocating engine-owned memory.

See Visitor callbacks below for the pattern.

Schema construction (projection pushdown)

The build-side counterpart to visit_schema: per-field callbacks that let the engine construct a Kernel StructType from its own type system (for example, to pass to scan_builder_with_schema).

Reading (scans)

Engine data and Arrow interop

Rows marked (requires default-engine-base) are only compiled when that feature is enabled; the rest are always available.

Warning: get_raw_engine_data is always exported (regardless of feature flags) but unimplemented. It calls todo!() and will panic. Do not use it.

Writing (transactions)

Write context and file writing

Use a WriteContext to learn where to write parquet files and what schema to write. For unpartitioned writes, one context serves the whole transaction. Partitioned writes (which would use one context per partition) are tracked in #2355.

Engines must append their own <uuid>.parquet filename (and any subdirectory layout) onto the returned table root. The kernel-side WriteContext::write_dir helper – which produces the recommended directory (Hive-style partition paths for partitioned tables when column mapping is off, or a random 2-char prefix when column mapping is on) – is internal and has no FFI binding.

Domain metadata

Table creation

Change data feed (table changes)

Incremental log reads for change data feed. Mirrors the Rust TableChanges API. The entire table_changes module is gated behind the default-engine-base feature.

Warning

scan_table_changes_next returns *mut ArrowFFIData (a heap-allocated Arrow C Data Interface batch). C callers must release each non-null result with free_arrow_ffi_data exactly once. This is an ABI-breaking change from earlier releases that used a different return shape, so connectors upgrading across that boundary need to update both the type and the cleanup path.

Checkpointing

Unity Catalog integration

Requires the delta-kernel-unity-catalog feature. Lets the engine provide a catalog-aware committer without implementing the committer trait from scratch.

Expressions and predicates

Kernel’s expression system is exposed through two parallel surfaces:

• Build (engine AST -> Kernel expression): the engine calls visit_engine_expression / visit_engine_predicate, providing an engine-side iterator. From inside that callback, the engine uses the visit_expression_* / visit_predicate_* builder functions to append Kernel-side nodes (columns, literals, operators) into an internal KernelExpressionVisitorState. The final handle is a SharedExpression or SharedPredicate.
• Walk (Kernel expression -> engine type): the engine calls visit_expression / visit_predicate, supplying an EngineExpressionVisitor struct whose function pointers Kernel invokes as it traverses the tree.

Most engines need only one of the two surfaces.

Tracing

Enable Kernel’s internal tracing instrumentation. Requires the tracing feature flag. Call at most one of these during a process lifetime; later calls return false.

Common utilities

Visitor callbacks

Several FFI entry points take a visitor struct (for example, EngineSchemaVisitor for visit_schema, EngineExpressionVisitor for visit_expression) or individual callbacks (as in visit_metadata). The pattern is the same in every case:

1. You allocate a context (a NullableCvoid you own) that the callbacks can write into. Kernel does not interpret this value.
2. You fill in one callback per node kind. Kernel invokes the callbacks passing the context plus node-specific arguments (names, types, literal values, child handles). For tree-structured inputs (schemas, expressions, predicates), callbacks fire in depth-first order; for flat inputs (e.g. visit_metadata), the callback fires once.
3. When visit_* returns, the context holds your engine-side representation.

Callbacks run synchronously on the same thread that called visit_*. Strings passed to callbacks (KernelStringSlice) are borrowed for the duration of the call; copy them if you need to retain them beyond the callback.

Error handling

Kernel functions that can fail return an ExternResult<T>, which is a tagged union:

// C representation (simplified)
typedef enum { Ok, Err } ExternResultTag;
typedef struct {
    ExternResultTag tag;
    union {
        T ok;
        EngineError* err;
    };
} ExternResult;

You provide an allocate_error callback when creating the engine. Kernel calls this callback to allocate error objects in your memory space whenever an operation fails. Because the engine allocates these errors, the engine is also responsible for freeing them. Kernel returns the error pointer immediately and does not retain it.

The EngineError struct contains a KernelError enum that classifies the error type (e.g., GenericError, FileNotFoundError, InvalidUrlError). The error message string passed to allocate_error is only valid for the duration of the callback, so you must copy it if you need to keep it.

C examples

The repository ships four runnable C examples under ffi/examples/. Each is a complete program that links against delta_kernel_ffi and exercises a different slice of the API.

The high-level flow in the read-table example:

// 1. Create an engine
ExternResultHandleSharedExternEngine engine_res =
    get_default_engine(table_path, allocate_error);

// 2. Build a snapshot
ExternResultHandleMutableFfiSnapshotBuilder builder_res =
    get_snapshot_builder(table_path, engine);
ExternResultHandleSharedSnapshot snap_res =
    snapshot_builder_build(builder);

// 3. Create a scan
ExternResultHandleSharedScan scan_res =
    scan(snap, engine, NULL, NULL);

// 4. Iterate over scan metadata and read data
// ... (see the full example for details)

// 5. Clean up
free_scan(the_scan);
free_snapshot(snap);
free_engine(engine);

What’s next

• Building a Connector: implementing a custom engine in Rust
• The Engine Trait: understanding the engine interface that the FFI wraps