Runtime

Estuary's runtime is the engine that powers the platform. In general, you will not need to know how the runtime works to use it. This document provides optional advanced information about low-level Estuary components for those interested in the platform's inner workings.

warning

Estuary is in the process of migrating tasks to a newer version of the runtime. This especially can affect users who automate task specifications. See the migration section below for details.

The runtime orchestrates end-to-end transaction boundaries that scale from milliseconds to hours, and from kilobytes to hundreds of gigabytes. It uses destination back-pressure to optimize the size of transactions, and within a transaction it reduces source changes into their cumulative effects.

In standard-update mode, the destination sees one load and one store per key per transaction even when that value reflects many upstream documents, which eliminates hot-key skew and allows, for example, a busy PostgresSQL CDC workload to be safely aggregated into Google Sheets despite the later accepting updates only every few seconds.

All of this is automatic; none of it requires specialist configuration.

Architecture

While many of Estuary's low-level concepts (such as journals and shards) originate in the Gazette project, Estuary's main runtime transaction loop diverges from the Gazette consumer framework's transaction lifecycle.

Which transaction loop model a task uses depends on the task type.

Task Type	Transaction Loop
Capture	Independent per-shard transaction loop
Derivation/Materialization	Shuffle leader coordinates multi-shard transactions

The runtime's architecture allows scaling without re-partitioning collections: storage layout and grouping is decoupled so storage and compute can scale separately.

Estuary manages this with a shuffle leader to coordinate multi-shard transactions. Instead of fixing storage partitions at write time, the shuffle service allows Estuary to decide which key belongs to which consumer at read time. This lets Estuary manage stateful transactions at scale.

For more information, see Estuary's runtime-next and shuffle crates.

Runtime Migration

As of Q3 2026, Estuary is currently rolling out runtime changes to improve parallelization in journals and shards. For many users, this rollout will occur automatically without any action required.

However, some users may wish to manage the runtime version their tasks are on themselves, and users who manage their own task specifications with flowctl will need to account for specification changes for the new runtime version. In these cases, work with Estuary support to ensure your tasks are migrated correctly.

New runtime benefits

The runtime was rewritten with several key benefits in mind. The following benefits are all currently available for tasks on the new runtime, with additional planned features to follow.

Handling large and variable-size documents

The new runtime replaces a fixed-size ring buffer measured by document count with a disk-backed buffer sized in bytes. It reads ahead continuously up to a byte limit, so it adapts to whatever document sizes a collection actually contains.

This allows tasks to better handle:

• Variable-sized documents without hand-tuning buffer settings
• Oversize documents without running out of memory

Higher throughput during backfills

While the new runtime maintains the rate at which it reads source collections, backfill throughput has increased.

The new runtime manages this by reading ahead while the previous transaction is still committing. Each transaction opens with far more data already accumulated, and the combiner reduces many more documents together before anything is written. The destination then receives fewer, larger writes for the same input, which also lowers cost on destinations that charge per query or per load operation.

Transactions are sized based on the backlog. A task with a lot of work waiting, such as one working through a backfill or a historical dataset, automatically runs longer transactions to clear that backlog faster. As it catches up and less data is waiting, transactions become shorter again.

Faster restarts

Tasks restart as a matter of course: a spec edit, a schema inference update, a connector release.

On a restart, the new runtime keeps reading until the next transaction cutoff and defers post-commit work to the following session. Its read-ahead buffer also survives the restart, so the task can come back and commit a large transaction straight away rather than needing to rebuild.

More precise sync schedules

The new runtime reduces staleness in sync schedules, allowing users to more precisely rely on their configured interval.

The runtime will read new documents continuously so that documents that arrive late in the interval window will still make the next commit. Reporting catches up correspondingly.

Additional sync schedule updates are planned to bring the feature fully within the runtime rather than a connector-level configuration.

Backlog visibility

Materializations on the new runtime report how many bytes of each source collection they have yet to read. The dashboard shows this as Backlog on the materialization's Overview tab: a byte figure while the task is catching up, or current once it has nothing left to read.

The same figure is available per binding in the bytesBehind field of your stats collection, alongside lastSourcePublishedAt, which tells you how far behind the task is in time.

Transaction grouping across collections

A single transaction at your source often writes to several tables, and therefore to several Estuary collections. Captures on the new runtime record which collection writes belong to the same source transaction. Materializations reading those collections can then apply the whole group in one destination transaction.

In practice: a single order return at a PostgreSQL source writes a sale reversal, an inventory update, and a loyalty point adjustment. Those three writes will land in your warehouse as one commit. A query cannot see the reversal without the inventory change, or the inventory change without the loyalty adjustment.

Automatic journal splitting

Every collection is stored in one or more journals, and each journal accepts writes at up to 4 MiB/s, or roughly 14 GB per hour. That per-journal limit keeps a single high-volume collection from overloading a storage broker, and it is the ceiling a fast capture usually reaches first.

The new runtime automatically splits journals as needed. When a task sees sustained write throttling on a journal, it splits that journal at the midpoint of its key range. Splitting repeats as needed up to 8 journals per logical partition, which raises a collection's write ceiling to about 32 MiB/s, or roughly 115 GB per hour, without requiring manual intervention.

Webhook trigger debounce

If your materialization fires a webhook trigger after each commit, the new runtime adds an optional interval on each trigger config. A burst of transactions then collapses into at most one delivery per interval, with a payload covering everything the collapsed transactions wrote.

This matters when the webhook starts something expensive. A dbt job that previously ran once per commit can be debounced to once every 30 minutes, while a Slack notification on the same materialization keeps firing per commit. The setting is per config.

Leaving interval unset preserves the existing fire-every-transaction behavior.

Scale-out within a task

The new runtime allows one task to spread its work across multiple shards running in parallel, coordinated so that they still commit as a single transaction. That removes the reason to fan out into multiple tasks.

Individual connectors are being updated to take full advantage of this parallelization: updates for Databricks have been released with more connectors to follow.

Order of migration

Both capture and materialization tasks can currently support the new runtime version. Capture tasks should always be migrated before associated materialization tasks.

Derivations will be migrated at a later date. Derivation task specs do not need to be updated at this time.

Task specification changes

Tasks on the new runtime version may include a new shards stanza in the specification. This stanza sits at the endpoint/bindings level:

{
  "endpoint": {...},
  "bindings": [...],
  "shards": {
    "flags": {
      "enable-runtime-v2": "true"
    }
  }
}

If the enable-runtime-v2 flag is set to true, the task uses the new runtime. If set to false, the task still uses the old runtime.

If the flag or the shards stanza as a whole is not present, the task may either be on the old or new runtime depending on the current stage of the runtime migration. If you're unsure of your task's status, you can check in with us.

While Estuary will be managing this flag for most task specs, you can also set it yourself following the migration order above.

Either way, you should ensure you have the latest version of your task specs from Estuary before publishing any new changes to them. This will make sure you don't accidentally rewrite the runtime version by removing the shards stanza or resetting the runtime flag.

In the long run, the enable-runtime-v2 flag is meant to be temporary. Once all tasks are fully migrated, the old runtime version will eventually be retired and the flag will no longer be needed.

Managing automated task specs

Users with automated task specs must be particularly cautious, as automated workflows may accidentally overwrite Estuary's changes. For example, Estuary may add the shards stanza and, since the CI/CD version of the spec doesn't that stanza, the CI/CD workflow may remove it again.

To avoid this scenario, you may either:

• Include pulling a fresh specification from Estuary in your workflow.

  This will ensure you start with any Estuary-side changes before making your own changes.

• Migrate your tasks to the new runtime in advance.

  Add a shards stanza with the enable-runtime-v2 flag to your specs following migration order.