The Engine trait

The Engine trait is the integration point between Delta Kernel and your connector. Kernel implements the Delta protocol and gives you scan and write APIs, but it needs help with the mechanics: listing files, reading and writing JSON and Parquet, and evaluating expressions for data skipping and logical-to-physical transformations. Kernel never does any of this directly. Instead, it calls into the Engine trait, which your connector implements.

This matters because it lets Kernel stay format-agnostic and runtime-agnostic. You control how I/O happens, what columnar format you use, and how expressions are evaluated.

A DefaultEngine is provided that you can use out of the box. If you need better performance or want to use your own data formats, you can build a custom engine. See Building a Connector for details.

The trait

trait Engine: AsAny {
    fn evaluation_handler(&self) -> Arc<dyn EvaluationHandler>;
    fn storage_handler(&self) -> Arc<dyn StorageHandler>;
    fn json_handler(&self) -> Arc<dyn JsonHandler>;
    fn parquet_handler(&self) -> Arc<dyn ParquetHandler>;
}

Kernel calls these methods whenever it needs to interact with the outside world. Each returns a handler trait object that Kernel uses for a specific category of work. The four handlers cover storage, JSON, Parquet, and expression evaluation. For observability, see Observability.

The four handlers

StorageHandler

File system operations. Kernel calls this to list and read files from the Delta log, and to write commit and checkpoint files.

JsonHandler

Reads and writes JSON. Kernel uses this for Delta log commits (the _delta_log/*.json files) and the JSON checkpoint manifest that references parquet sidecars. The sidecar files themselves are parquet and are read by the ParquetHandler below.

ParquetHandler

Reads and writes Parquet. Kernel uses this for checkpoint files (including parquet checkpoint sidecars) and for reading data files during scans.

The Parquet handler also supports metadata columns (row index, file path) and field-ID-based column matching for column mapping.

EvaluationHandler

Expression evaluation. Kernel uses this for data skipping (evaluating predicates against file statistics) and for applying per-file transformations (partition value injection, column mapping).

The expression and predicate evaluators are reusable objects that you can call repeatedly on different batches of EngineData.

The Default Engine

The DefaultEngine is a batteries-included implementation that works out of the box:

• Uses Apache Arrow as the in-memory data format
• Uses object_store for I/O (supports local FS, S3, GCS, Azure)
• Runs async I/O on a Tokio thread pool
• Supports multiple Arrow versions (see Feature Flags)

To construct one, create an object store and pass it to the builder:

#![allow(unused)]
fn main() {
extern crate delta_kernel;
use delta_kernel::engine::default::DefaultEngine;
use delta_kernel::engine::default::storage::store_from_url;
use delta_kernel::DeltaResult;
fn example() -> DeltaResult<()> {
let url = url::Url::parse("file:///path/to/table")?;
let store = store_from_url(&url)?;
let engine = DefaultEngine::builder(store).build();
Ok(())
}
}

DefaultEngine also provides a convenience method for writing Parquet files:

let engine_data = engine
    .write_parquet(&data, &write_context)
    .await?;

This is not part of the Engine trait itself. It’s a helper on DefaultEngine that orchestrates the lower-level ParquetHandler methods with a WriteContext (write directory, schema, stats columns, partition values).

Configuring the Default Engine

DefaultEngine uses a builder pattern that lets you customize the task executor and plug in a metrics reporter. The builder starts from DefaultEngine::builder(store) and chains optional configuration before calling build():

let engine = DefaultEngine::builder(store)
    .with_metrics_reporter(reporter)
    .with_task_executor(executor)
    .build();

All builder methods are optional. Calling DefaultEngine::builder(store).build() gives you a fully functional engine with sensible defaults.

The TaskExecutor trait

DefaultEngine uses asynchronous I/O internally, but Kernel’s public APIs are synchronous. The TaskExecutor trait bridges this gap by defining how async work gets scheduled and awaited. It has four methods:

You don’t need to implement TaskExecutor yourself unless you have a non-Tokio async runtime. Kernel ships two Tokio-based implementations behind the tokio feature flag.

TokioBackgroundExecutor

TokioBackgroundExecutor is the default executor. It spawns a dedicated background thread running a single-threaded Tokio runtime. All async work is dispatched to that thread over a channel.

use delta_kernel::engine::default::executor::tokio::TokioBackgroundExecutor;

let executor = TokioBackgroundExecutor::new();

This is the right choice when:

• You don’t already have a Tokio runtime in your application.
• You want Kernel’s I/O isolated from the rest of your process.
• You’re building a standalone connector or CLI tool.

Because it owns its runtime, TokioBackgroundExecutor works even when no external Tokio runtime is active. On drop, it shuts down the background thread cleanly.

TokioMultiThreadExecutor

TokioMultiThreadExecutor runs async work on a multi-threaded Tokio runtime. It comes in two flavors.

Share an existing runtime. If your application already has a Tokio runtime (for example, a web server or a query engine), pass its handle so Kernel’s I/O shares the same thread pool:

use delta_kernel::engine::default::executor::tokio::TokioMultiThreadExecutor;

let handle = tokio::runtime::Handle::current();
let executor = TokioMultiThreadExecutor::new(handle);

The handle must come from a multi-threaded runtime. Passing a current-thread handle causes a panic.

Own a dedicated runtime. If you want a multi-threaded runtime that Kernel manages, use new_owned_runtime. You can optionally set the number of worker threads and the maximum number of blocking threads. Pass None for either to use Tokio’s defaults:

let executor = TokioMultiThreadExecutor::new_owned_runtime(
    Some(4),   // 4 worker threads
    Some(64),  // up to 64 blocking threads
)?;

Warning

Deeply nested block_on calls can exhaust Tokio’s blocking thread pool and deadlock. If you set a small max_blocking_threads, keep nesting depth low.

Choosing an executor

To use a custom executor, pass it to the builder with with_task_executor:

let executor = Arc::new(TokioMultiThreadExecutor::new(handle));
let engine = DefaultEngine::builder(store)
    .with_task_executor(executor)
    .build();

Entering the runtime context

Some libraries (for example, object_store or reqwest) require an active Tokio runtime context to construct clients or resolve configuration. If you call such code outside of an async function, tokio::runtime::Handle::current() will panic because no runtime is active.

DefaultEngine::enter() solves this by entering the executor’s runtime context:

let guard = engine.enter();
// Code here can call Handle::current() safely.
// The guard must be dropped before acquiring another.
drop(guard);

The returned guard keeps the runtime context active until it is dropped. If you acquire multiple guards, you must drop them in reverse order. Dropping out of order causes a panic.

When to implement your own Engine

You should implement Engine if:

• You have your own columnar data format (not Arrow)
• You need custom I/O (e.g. your own distributed file system client)
• You want to use your own expression evaluation engine
• You need to control parallelism or resource usage beyond what DefaultEngine offers

You do not need a custom engine to use different storage (S3, Azure, etc.). The DefaultEngine supports all object_store backends. See Configuring Storage.

For a guide on implementing Engine, see Implementing the Engine Trait.

What’s next

• Building a Connector explains the role of a connector and how the Engine fits into the bigger picture.
• Implementing the Engine Trait walks through building a custom Engine from scratch.
• Configuring Storage shows how to point DefaultEngine at S3, GCS, or Azure storage.