Skip to main content

Configure Connections with PrivateLink

If you use Estuary in a private deployment or Bring Your Own Cloud setup, you can securely connect services using PrivateLink. This lets you transfer data between your private or BYOC deployment and external services without exposing it to the public internet.

Different cloud providers have different setup steps to start using PrivateLink.

For AWS private or BYOC deployments, we can establish connections to your endpoints using AWS PrivateLink.

To do so, you will first need to create a Virtual Private Cloud (VPC) endpoint service and associated Network Load Balancer (NLB). The NLB configuration will depend on the service you're working with.

You can create or manage your endpoint services on the AWS VPC dashboard. As you do so:

  • Specify the NLB you created.
  • Safelist your AWS VPC Account ID (such as arn:aws:iam::12345:root) to allow access to your VPC endpoint service.
  • By default, the endpoint service should be in the same region as your private deployment. To connect across regions, see Cross-region setup below.
tip

See the AWS documentation for the most up-to-date, detailed instructions on working with PrivateLink or contact your AWS representative for assistance.

We will then need the following information to establish the connection. Send these details to your Estuary point of contact:

  • The endpoint service name, such as com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef
  • The Availability Zone IDs it offers, such as [use1-az4, use1-az6]

To activate the connection, accept the interface endpoint connection request from Estuary. We will then provide a DNS name which you can use as the hostname when connecting to the database.

Network Load Balancer Setup​

To set up your Virtual Private Cloud endpoint service in AWS, you will first need to create a Network Load Balancer in your VPC. How you configure your NLB will depend on the service you're working with:

  • Static IP

    If your service is accessed through a single static IP (like EC2, or a non-RDS database), simply ensure that the NLB availability zones match the target availability zones or enable cross-zone load balancing.

  • Dynamic IP

    If your service has a dynamic IP (such as an Amazon Aurora or RDS database), you will need to configure your Network Load Balancer further to ensure uninterrupted service. There are essentially two different ways to do this.

    • Use a port forwarding instance

      Deploy an EC2 instance that is configured to do port forwarding, such as accepting requests from the NLB and forwarding them to your RDS database.

    • Use the dynamic IP address

      To work around a database IP that may change without notice, you can deploy a lambda function to periodically check the IP address and update the NLB target group when it changes.

      You can find the IP address by running the nslookup or dig command with the DNS name for your endpoint. For example:

      dig +short <YOUR_RDS_DNS_ENDPOINT>

      Set up your NLB target group with this initial IP address. Your lambda function can then periodically pull the current IP, compare, and update the target group as needed.

Cross-region​

By default, AWS requires a VPC endpoint service to be in the same region as the consumer. Estuary supports the cross-region topology, where your endpoint service (and the database behind it) is in a different AWS region from your Estuary data plane - for example, an Estuary data plane in us-east-1 connecting to an Aurora cluster in eu-west-1. This is useful when a single data plane needs to capture from sources across multiple regions, or when regulatory constraints fix the data plane's region independent of where individual sources live.

Cross-region PrivateLink requires additional configuration on your side beyond the standard setup. The steps below assume you have already followed the steps in the AWS PrivateLink section above and have an NLB-backed VPC endpoint service in the source region.

Requirements on your side​

  1. IAM: the identity used to configure your endpoint service must have the vpce:AllowMultiRegion permission-only action allowed in its identity-based policy, and the action must not be denied by any Service Control Policy. By default, no IAM entity has this permission. See Cross-Region access — Permissions in the AWS documentation.
  2. Supported regions: explicitly add the data plane's region as a supported region on your endpoint service. In the AWS VPC console, open the endpoint service → Actions → Modify supported Regions → select the data plane's region → Save changes. See Modify the supported Regions.
  3. NLB constraints (per AWS cross-region considerations):
    • The NLB must use the IPv4 or dualstack IP address type. Pure-IPv6 NLBs are not supported for cross-region access.
    • The NLB must use the default TCP idle timeout (tcp.idle_timeout.seconds = 350). Custom idle timeouts are not supported for cross-region.
  4. AllowedPrincipals: add arn:aws:iam::770785070253:user/data-plane-ops to the endpoint service's Allow principals tab. This is the specific IAM user Estuary uses to create cross-region VPC endpoints on your service.
  5. Region availability: confirm that both regions are in the same AWS partition (e.g., both aws, not one aws-cn or aws-us-gov). AWS does not support cross-region PrivateLink in a small set of Availability Zones — see the AWS list of unsupported AZs.

What to send Estuary​

Once the above is configured, share the same details as in the standard setup, and explicitly mention that this is a cross-region setup so we configure the link correctly on our side:

  • The endpoint service name, such as com.amazonaws.vpce.eu-west-1.vpce-svc-0123456789abcdef
  • The Availability Zone IDs offered by your service, such as [euw1-az1, euw1-az2, euw1-az3]
  • A note that the setup is cross-region, along with the region of your endpoint service (e.g. "the endpoint service is in eu-west-1; our data plane is in us-east-2")

The AZ IDs refer to the service's region. Estuary provisions the consumer-side VPC endpoint in the data plane's region - AWS handles the AZ mapping internally - so you do not need to share AZ IDs from the data plane's region.

After Estuary creates the endpoint​

Estuary will initiate the connection from the data plane's region. You will see a pending connection request appear under your endpoint service's Endpoint connections tab - accept it from there. Estuary will then provide a DNS name pointing at the cross-region endpoint, which you can use as the hostname when configuring connectors.

tip

When accessing services cross-region, you must use the regional DNS name (e.g. vpce-0123-xyz.vpce-svc-abc.us-east-2.vpce.amazonaws.com), not a zonal name (...-us-east-2a.vpce-svc-...). AWS does not support zonal DNS for cross-region endpoints.

Common pitfalls​

  • Connection request never appears in your console: check that Estuary's principal is on Allow principals and that the data plane region is in your endpoint service's Supported Regions list.
  • Connection accepted but the connector still fails to resolve the host: verify the connector is using the regional DNS name returned by Estuary, not a zonal variant.

Variations​

Certain services may use AWS PrivateLink in unique ways. More detailed instructions for these services are provided below.

MongoDB

A MongoDB connection will use an altered endpoint ID to retrieve the hostname.

Create the endpoint service in AWS and receive back the DNS name from Estuary as for a standard AWS PrivateLink connection. To finish setup:

  1. Take the first 22 characters of the DNS name, such as vpce-0123456789abcdefg.

  2. In your MongoDB dashboard, navigate to the Network Access section.

  3. On the Private Endpoint page in this section, add a new private endpoint.

  4. In the setup modal, choose your cloud provider and Atlas region. You may skip the "Interface Endpoint" step.

  5. As part of the Finalize Endpoint Connection step, add the 22-character DNS string as Your VPC Endpoint ID.

  6. Click Create and wait until the Endpoint Status is Available.

MongoDB will create a URL for this endpoint. You can use this as the hostname in Estuary to connect with your database.

You can find this URL as you would any other Atlas hostname:

  1. Bring up the Connect modal for your MongoDB cluster.

  2. Make sure to choose Private Endpoint as your Connection Type and select the endpoint you created.

  3. Choose the Shell connection method. The shell command will display your MongoDB URL (for example, mongodb+srv://abc-123.mongodb.net/).

For Azure private or BYOC deployments, we can establish connections to your endpoints using Azure Private Link.

You will need to create an Azure Private Link Service which also requires having an Azure Load Balancer in front of the services you intend to expose. After creating these resources, make sure your LoadBalancer is able to route traffic correctly to your instances. You can check this by looking at the Monitoring -> Metrics page of your LoadBalancer and checking for its Health Probe Status.

Once you have your Private Link Service set up, we need these details from you to establish the connection. Send them to your Estuary point of contact:

  • The service URI, like /subscriptions/abcdefg-12345-12cc-1234-1234abcd1234abc/resourceGroups/foo/providers/Microsoft.Network/privateLinkServices/bar-service; this can be found by navigating to the private link service's details page in your Azure Portal and copying the URL
  • Location for the private endpoint, like westus

After establishing the connection we will give you a private IP address which you can use to connect to your endpoint when setting up your task on the Estuary web app.

GCP Private Service Connect​

For GCP private or BYOC deployments, we can establish connections to your endpoints using Private Service Connect.

You will first need to publish a service with Private Service Connect. The exact steps vary depending on whether you're using a managed service like Cloud SQL or a self-hosted service.

tip

See the GCP documentation for the most up-to-date, detailed instructions on publishing services with Private Service Connect.

Once you have your service published, we need these details from you to establish the connection. Send them to your Estuary point of contact:

  • The service attachment URI, such as projects/my-project/regions/us-central1/serviceAttachments/my-attachment
  • The region of the service attachment, such as us-central1
  • Your desired DNS zone name, such as estuary-poc
  • Your desired DNS record name(s), such as my-database

To activate the connection, accept the Private Service Connect connection request from Estuary. We will then configure an endpoint in your data plane, and you can connect using the hostname you selected (such as my-database.estuary-poc).

Cloud SQL​

Cloud SQL for MySQL, PostgreSQL, and SQL Server all support Private Service Connect. You can enable PSC when creating a new instance or configure it on an existing instance.

You must add the Estuary project to your instance's allowed PSC projects list:

  • For private deployments, add helpful-kingdom-273219
  • For BYOC deployments, add your own GCP project name

You can update this list in the Cloud SQL instance settings under Connections > Private Service Connect allowed projects, or via the --allowed-psc-projects flag in gcloud.

To find the service attachment URI for your Cloud SQL instance:

  1. In the Google Cloud console, go to the Cloud SQL Instances page.
  2. Click on your instance to open its details.
  3. In the Connect to this instance section, locate the Service attachment field.

The URI will look like projects/my-project/regions/us-central1/serviceAttachments/my-instance-psc-attachment.

See the Cloud SQL documentation for detailed setup instructions.

Self-Hosted Services​

For self-hosted services running in GCP (such as databases on Compute Engine VMs), you will need to publish a Private Service Connect service. This requires:

See the GCP documentation for detailed setup instructions.