Flow stores the documents that comprise your collections in a cloud storage bucket. Your storage mapping tells Flow which bucket to use.
When you first register for Flow, your storage mapping is Estuary's secure Google Cloud Storage bucket. Data in Flow's cloud storage bucket is deleted 30 days after collection.
For production workflows, you should set up your own cloud storage bucket as a storage mapping.
You may also use apply different storage mappings to different catalog prefixes within your organization's prefix.
You can set up a bucket lifecycle policy to manage data retention in your storage mapping; for example, to remove data after six months.
In addition to collection data, Flow uses your storage mapping to temporarily store recovery logs.
Flow tasks — captures, derivations, and materializations — use recovery logs to durably store their processing context as a backup. Recovery logs are an opaque binary log, but may contain user data.
The recovery logs of a task are always prefixed by
so a task named
acmeCo/produce-TNT would have a recovery log called
Flow prunes data from recovery logs once it is no longer required.
Deleting data from recovery logs while it is still in use can cause Flow processing tasks to fail permanently.