Skip to main content

Configuring Cloud Storage for Flow

New Flow accounts are connected to Flow's secure cloud storage bucket to store collection data. To switch to your own bucket, choose a cloud provider and complete the setup steps:

Once you're done, get in touch.

Google Cloud Storage buckets​

You'll need to grant Estuary Flow access to your GCS bucket.

  1. Create a bucket to use with Flow, if you haven't already.

  2. Follow the steps to add a principal to a bucket level policy. As you do so:

    • For the principal, enter flow-258@helpful-kingdom-273219.iam.gserviceaccount.com

    • Select the roles/storage.admin role.

Amazon S3 buckets​

You'll need to grant Estuary Flow access to your S3 bucket.

  1. Create a bucket to use with Flow, if you haven't already.

  2. Follow the steps to add a bucket policy, pasting the policy below. The policy below will work for the US Data Plane. If you're on the EU Data Plane, [get in touch]. Be sure to replace YOUR-S3-BUCKET with the actual name of your bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowUsersToAccessObjectsUnderPrefix",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::789740162118:user/flow-aws"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::789740162118:user/flow-aws"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::789740162118:user/flow-aws"
      },
      "Action": "s3:GetBucketPolicy",
      "Resource": "arn:aws:s3:::YOUR-S3-BUCKET"
    }
  ]
}

Azure Blob Storage​

You'll need to grant Estuary Flow access to your storage account and container. You'll also need to provide some identifying information.

  1. Create an Azure Blob Storage container to use with Flow, if you haven't already.

  2. Gather the following information. You'll need this when you contact us to complete setup.

    • Your Azure AD tenant ID. You can find this in the Azure Active Directory page. Azure AD Tenant ID

    • Your Azure Blob Storage account ID. You can find this in the Storage Accounts page. Azure Storage Account Name

    • Your Azure Blob Storage container ID. You can find this inside your storage account. Azure Container ID

    You'll grant Flow access to your storage resources by connecting to Estuary's Azure application.

  3. Add Estuary's Azure application to your tenant.

  1. Grant the application access to your storage account via the Storage Blob Data Owner IAM role.

    • Inside your storage account's Access Control (IAM) tab, click Add Role Assignment.

    • Search for Storage Blob Data Owner and select it.

    • On the next page, make sure User, group, or service principal is selected, then click + Select Members.

    • You must search for the exact name of the application, otherwise it won't show up: Estuary Storage Mappings Prod

    • Once you've selected the application, finish granting the role.

    For more help, see the Azure docs.

Add the Bucket​

If your bucket is for Google Cloud Storage or AWS S3, you can add it yourself. Once you've finished the above steps, head to "Admin", "Settings" then "Configure Cloud Storage" and enter the relevant information there and we'll start to use your bucket for all data going forward.

If your bucket is for Azure, send support@estuary.dev an email with the name of the storage bucket and any other information you gathered per the steps above. Let us know whether you want to use this storage bucket to for your whole Flow account, or just a specific prefix. We'll be in touch when it's done!