Skip to main content

Configuring Cloud Storage for Flow

New Flow accounts are connected to Flow's secure cloud storage bucket to store collection data. To switch to your own bucket, you will need to:

  1. Choose a data plane.

  2. Then choose a cloud provider for your storage bucket and complete the setup steps:

  3. Add the bucket in Estuary.

Once your collections begin to persist data, you can view your storage bucket to confirm that the bucket is receiving data as expected. This should indicate that your bucket has been set up correctly.

You can also set a bucket lifecycle policy to limit how long collections keep data. Note that lifecycle policies should be specific to the collection-data/ sub-directory that Estuary creates, not cover the entire bucket.

Choose a Data Plane​

In Estuary, data planes handle connector tasks and data movement. While there is one control plane that orchestrates tasks, there are multiple data planes to choose from, even private data planes.

Choosing a data plane with the same cloud provider or region as your existing infrastructure can reduce latency or you can ensure data stays within certain regional bounds for compliance purposes.

Data planes with running tasks must be able to talk to your cloud storage to save collection and recovery data. You must therefore grant access to your storage bucket to the data plane's IAM user or service account.

You can find your available data planes and their authentication details from the dashboard:

  1. Go to the Admin page in the Estuary dashboard and select the Settings tab.

  2. Scroll down to the Data Planes table.

  3. Find your desired data plane by the cloud provider and region listing. Copy its information for use when setting up your storage bucket permissions:

Google Cloud Storage buckets​

You'll need to grant Estuary Flow access to your GCS bucket.

  1. Create a bucket to use with Flow, if you haven't already.

  2. Follow the steps to add a principal to a bucket level policy. As you do so:

Amazon S3 buckets​

You'll need to grant Estuary Flow access to your S3 bucket.

  1. Create a bucket to use with Flow, if you haven't already.

  2. Follow the steps to add a bucket policy. Use the example policy below, making the following modifications:

    • Replace YOUR-S3-BUCKET with the actual name of your bucket.

    • Replace DATA-PLANE-IAM-ARN with your data plane's AWS IAM User ARN.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowUsersToAccessObjectsUnderPrefix",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "DATA-PLANE-IAM-ARN",
          "arn:aws:iam::789740162118:user/flow-aws"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "DATA-PLANE-IAM-ARN",
          "arn:aws:iam::789740162118:user/flow-aws"
        ]
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "DATA-PLANE-IAM-ARN",
          "arn:aws:iam::789740162118:user/flow-aws"
        ]
      },
      "Action": "s3:GetBucketPolicy",
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET"
    }
  ]
}

Azure Blob Storage​

You'll need to grant Estuary Flow access to your storage account and container. You'll also need to provide some identifying information.

  1. Create an Azure storage account if you haven't already.

    Make sure your storage account has the Hierarchical Namespace option disabled. During storage account creation, you can review this setting in the Advanced tab. Under the Data Lake Storage section, ensure the hierarchical namespace option is unchecked.

  2. Within the storage account, create an Azure Blob Storage container to use with Flow, if you haven't already.

  3. Gather the following information. You'll need this when you contact us to complete setup.

    • Your Azure AD tenant ID. You can find this in the Azure Active Directory page. Azure AD Tenant ID

    • Your Azure Blob Storage account ID. You can find this in the Storage Accounts page. Azure Storage Account Name

    • Your Azure Blob Storage container ID. You can find this inside your storage account. Azure Container ID

    You'll grant Flow access to your storage resources by connecting to Estuary's Azure application.

  4. Add Estuary's Azure application to your tenant.


If you're having trouble using the input field above, you may also modify this OAuth link with your Azure tenant ID and paste it directly into your browser:

https://login.microsoftonline.com/<YOUR_AZURE_TENANT>/oauth2/authorize?client_id=42cb0c6c-dab0-411f-9c21-16d5a2b1b025&response_type=code&redirect_uri=https%3A%2F%2Feyrcnmuzzyriypdajwdk.supabase.co%2Ffunctions%2Fv1%2Fazure-dpc-oauth&resource_id=https://storage.azure.com

  1. Grant the application access to your storage account via the Storage Blob Data Owner IAM role.

    • Inside your storage account's Access Control (IAM) tab, click Add Role Assignment.

    • Search for Storage Blob Data Owner and select it.

    • On the next page, make sure User, group, or service principal is selected, then click + Select Members.

    • You must search for the exact name of the application, otherwise it won't show up.

      • GCP US Central-1 Data Plane: Estuary Storage Mappings Prod
      • AWS EU West-1 Data Plane: data-plane-aws-eu-west-1-c1.dp.estuary-data.com

    • Once you've selected the application, finish granting the role.

    For more help, see the Azure docs.

Add the Bucket​

If your bucket is for Google Cloud Storage or AWS S3, you can add it yourself. Once you've finished the above steps, head to "Admin", "Settings" then "Configure Cloud Storage" and enter the relevant information there and we'll start to use your bucket for all data going forward.

If your bucket is for Azure, send support@estuary.dev an email with the name of the storage bucket and any other information you gathered per the steps above. Let us know whether you want to use this storage bucket to for your whole Flow account, or just a specific prefix. We'll be in touch when it's done!

Migrating your existing data to the new storage mapping​

Once you've created your new storage mapping, your collections will be updated to pick up the new storage mapping, so new data will be written there. Existing data from your previous storage mapping is not automatically migrated, to do so you should backfill all of your captures after configuring a storage mapping. Most tenants will have been using the estuary-public storage mapping on the Free plan to begin with, which expires data after 20 days. By backfilling all of your captures you guarantee that even though the data from estuary-public will expire, you’ll still have a full view of your data available on your new storage mapping.