Schema Evolution

When schemas change, Estuary's default behavior is to automatically evolve their specifications to avoid errors. This guide covers what to expect when using automatic schema evolution.

See related resources:

• To handle schema changes if automatic evolution is turned off, see manually updating schemas.
• For a conceptual overview of schema evolution, see the schema evolution concepts page.

Introduction

Estuary collections serve not only as your real-time data storage, but also as a contract between tasks that produce and consume their data. Captures are producers, materializations are consumers, and derivations act as both.

This contract helps prevent data loss and error in your Data Flows, and is defined in terms of the collection specification, or spec, which includes:

• The JSON schema
• The collection key
• Projections, if any

There are many reasons a collection spec might change. Often, it's due to a change in the source data. Estuary will automatically infer schema changes. You can then control whether your pipeline automatically handles evolution tasks with:

• autoDiscover capture settings
• onIncompatibleSchemaChange materialization settings

Automatic schema evolution

Estuary's default behavior is to automatically discover changes and evolve schemas. This is especially useful for data you expect to change shape frequently, or for data that includes outliers which initial schema inference may not capture.

You can set your desired auto-discover and evolution behavior from the capture.

This comprises several related functions:

• Automatically keep schemas up to date
• Automatically add new collections
• Changing primary keys resets collections

Schema updates can be inferred in the first place in one of several ways:

	Auto-discover	Sourced schemas	Continuous schema inference
Reads	Source metadata	Source metadata	Captured documents
Runs in	Control plane, every 2h	Connector, 15 min default (SQL CDC)	Runtime, every transaction
Lands in	flow://connector-schema (write schema)	flow://inferred-schema (read schema)	flow://inferred-schema (read schema)
Update rule	Replaces	Widen only	Widen only

Auto-discover

With auto-discover, Estuary will periodically run background operations to check for changes to a source and re-publish the capture if necessary. This discovers newly available bindings, as well as collection schema updates. These schemas are defined as the flow://connector-schema in the write schema.

Auto-discover runs on a 2-hour cadence and applies to most captures.

Sourced schemas

For sources that keep a strict schema (such as SQL-based sources), Estuary scans the source at predefined intervals to check if there are any updates to the source's tables. This informs the runtime about types and metadata that aren't easily inferred by inspecting individual documents.

For SQL CDC captures, this interval defaults to 15 minutes and is configurable via the rediscovery interval.

Other captures, such as batch SQL captures and some SaaS captures like Salesforce and HubSpot, check source metadata each time the capture starts or in response to specific events.

These schemas are defined as the flow://inferred-schema in the read schema.

Continuous schema inference

With continuous schema inference, Estuary automatically infers and updates collection schemas based on the data read in from the capture.

Estuary accounts for schema changes (such as changes to a column's data type, or a new field) by widening the schema. This is defined as the flow://inferred-schema in the read schema.

Schema inference will only ever widen data types. To narrow a column's data type instead, use a dataflow reset to reset inferred schemas and backfill data.

Schema evolution scenarios

Collection specifications can change for many reasons. Common changes requiring evolution can include:

• The key pointers have changed
• The logical partitioning configuration has changed
• The collection schema has changed
  • A new field is added
  • A field's data type has changed
  • A field was removed

The sections below provide additional details on these scenarios.

tip

See troubleshooting schemas if you encounter issues with evolutions or manual evolution if you need to make updates manually.

Collection key changes

Example:

An inventory table at the source used to be keyed on a sku field and is now keyed on a uuid field, changing Estuary's collection key pointer.

Changing the collection key is an incompatible schema change. This means that the new schema cannot work simultaneously with the old schema, but Estuary can still automatically recover. evolveIncompatibleCollections (on by default) triggers a dataflow reset to recover.

The reset backfills all associated resources, automatically carrying over to materialization bindings unless the materialization's onIncompatibleSchemaChange setting is abort.

Logical partition changes

Example:

A customers table used to contain no logical partitioning and is now being partitioned by geo_region.

Changing logical partitioning is an incompatible schema change. This means that the new schema cannot work simultaneously with the old schema, but Estuary can still automatically recover. evolveIncompatibleCollections (on by default) triggers a dataflow reset to recover.

The reset backfills all associated resources, automatically carrying over to materialization bindings unless the materialization's onIncompatibleSchemaChange setting is abort.

A new field is added

Example:

An inventory table in the source originally consisted of name, sku, and quantity fields. Later, a new location field is added to the table.

Estuary's schema inference will automatically pick up new fields and add them to the collection schema.

When a new field appears in the collection schema, it may automatically be added to any materializations that use recommended fields. Recommended fields are enabled by default in each binding. See the materialization docs for more info about how to enable or disable recommended fields.

When recommended fields are enabled, new fields are added automatically if they meet the criteria for the particular materialization connector. For example, scalar fields (strings, numbers, and booleans) are considered "recommended" fields when materializing to database tables.

If your materialization binding is set to recommended: false, or if the new field is not recommended, you can manually add it to the materialization.

A field's data type has changed

Example:

A field quantity had previously been inferred as an integer type. New records include float values, so Estuary widens the data type to number.

Schema inference will generally handle this type of evolution for you by widening the field's data type. This allows both historic and future values to fit the collection schema.

A field was removed

Example:

A customers table and its corresponding data stream used to include a birthday field and no longer does.

Removing fields is generally allowed by all connectors, and does not require new tables or collections. Note that for database materializations, the existing column will not be dropped, and will just be ignored by the materialization going forward. A NOT NULL constraint would be removed from that column, but it will otherwise be left in place.