Skip to main content

Configure connections with SSH tunneling

Flow connects to certain types of endpoints — generally databases — using their IP address and port. For added security, you can configure SSH tunneling, also known as port forwarding. You configure this in the networkTunnel section of applicable capture or materialization definitions, but before you can do so, you need a properly configured SSH server on your internal network or cloud hosting platform.

tip

If permitted by your organization, a quicker way to connect to a secure database is to whitelist the Estuary IP address, 34.121.207.128. For help completing this task on different cloud hosting platforms, see the documentation for the connector you're using.

This guide includes setup steps for popular cloud platforms, as well as generalized setup that provides a basic roadmap for on-premise servers or other cloud platforms.

After completing the appropriate setup requirements, proceed to the configuration section to add your SSH server to your capture or materialization definition.

General setup​

  1. Activate an SSH implementation on a server, if you don't have one already. Consult the documentation for your server's operating system and/or cloud service provider, as the steps will vary. Configure the server to your organization's standards, or reference the SSH documentation for basic configuration options.

  2. Referencing the config files and shell output, collect the following information:

  • The SSH user, which will be used to log into the SSH server, for example, sshuser. You may choose to create a new user for this workflow.
  • The SSH endpoint for the SSH server, formatted as ssh://user@hostname[:port]. This may look like the any of following:
    • ssh://sshuser@ec2-198-21-98-1.compute-1.amazonaws.com
    • ssh://sshuser@198.21.98.1
    • ssh://sshuser@198.21.98.1:22
    Hint

    The SSH default port is 22. Depending on where your server is hosted, you may not be required to specify a port, but we recommend specifying :22 in all cases to ensure a connection can be made.

  1. In the .ssh subdirectory of your user home directory, look for the PEM file that contains the private SSH key. Check that it starts with -----BEGIN RSA PRIVATE KEY-----, which indicates it is an RSA-based file.
    • If no such file exists, generate one using the command:
       ssh-keygen -m PEM -t rsa
    • If a PEM file exists, but starts with -----BEGIN OPENSSH PRIVATE KEY-----, convert it with the command:
       ssh-keygen -p -N "" -m pem -f /path/to/key

Taken together, these configuration details would allow you to log into the SSH server from your local machine. They'll allow the connector to do the same.

  1. Configure your internal network to allow the SSH server to access your capture or materialization endpoint.

  2. To grant external access to the SSH server, it's essential to configure your network settings accordingly. The approach you take will be dictated by your organization's IT policies. One recommended step is to whitelist Estuary's IP address, which is 34.121.207.128. This ensures that connections from this specific IP are permitted through your network's firewall or security measures.

Setup for AWS​

To allow SSH tunneling to a database instance hosted on AWS, you'll need to create a virtual computing environment, or instance, in Amazon EC2.

  1. Begin by finding your public SSH key on your local machine. In the .ssh subdirectory of your user home directory, look for the PEM file that contains the private SSH key. Check that it starts with -----BEGIN RSA PRIVATE KEY-----, which indicates it is an RSA-based file.

    • If no such file exists, generate one using the command:
       ssh-keygen -m PEM -t rsa
    • If a PEM file exists, but starts with -----BEGIN OPENSSH PRIVATE KEY-----, convert it with the command:
       ssh-keygen -p -N "" -m pem -f /path/to/key

  2. Import your SSH key into AWS.

  3. Launch a new instance in EC2. During setup:

    • Configure the security group to allow SSH connection from anywhere.
    • When selecting a key pair, choose the key you just imported.

  4. Connect to the instance, setting the user name to ec2-user.

  5. Find and note the instance's public DNS. This will be formatted like: ec2-198-21-98-1.compute-1.amazonaws.com.

Setup for Google Cloud​

To allow SSH tunneling to a database instance hosted on Google Cloud, you must set up a virtual machine (VM).

  1. Begin by finding your public SSH key on your local machine. In the .ssh subdirectory of your user home directory, look for the PEM file that contains the private SSH key. Check that it starts with -----BEGIN RSA PRIVATE KEY-----, which indicates it is an RSA-based file.

    • If no such file exists, generate one using the command:
       ssh-keygen -m PEM -t rsa
    • If a PEM file exists, but starts with -----BEGIN OPENSSH PRIVATE KEY-----, convert it with the command:
       ssh-keygen -p -N "" -m pem -f /path/to/key
    • If your Google login differs from your local username, generate a key that includes your Google email address as a comment:
       ssh-keygen -m PEM -t rsa -C user@domain.com

  2. Create and start a new VM in GCP, choosing an image that supports OS Login.

  3. Add your public key to the VM.

  4. Reserve an external IP address and connect it to the VM during setup. Note the generated address.

Setup for Azure​

To allow SSH tunneling to a database instance hosted on Azure, you'll need to create a virtual machine (VM) in the same virtual network as your endpoint database.

  1. Begin by finding your public SSH key on your local machine. In the .ssh subdirectory of your user home directory, look for the PEM file that contains the private SSH key. Check that it starts with -----BEGIN RSA PRIVATE KEY-----, which indicates it is an RSA-based file.

    • If no such file exists, generate one using the command:
       ssh-keygen -m PEM -t rsa
    • If a PEM file exists, but starts with -----BEGIN OPENSSH PRIVATE KEY-----, convert it with the command:
       ssh-keygen -p -N "" -m pem -f /path/to/key

  2. Create and connect to a VM in a virtual network, and add the endpoint database to the network.

    1. Create a new virtual network and subnet.

    2. Create a Linux or Windows VM within the virtual network, directing the SSH public key source to the public key you generated previously.

    3. Note the VM's public IP; you'll need this later.

  3. Create a service endpoint for your database in the same virtual network as your VM. Instructions for Azure Database For PostgreSQL can be found here; note that instructions for other database engines may be different.

Configuration​

After you've completed the prerequisites, you should have the following parameters:

  • SSH Endpoint / sshEndpoint: the remote SSH server's hostname, or public IP address, formatted as ssh://user@hostname[:port]

    The SSH default port is 22. Depending on where your server is hosted, you may not be required to specify a port, but we recommend specifying :22 in all cases to ensure a connection can be made.

  • Private Key / privateKey: the contents of the SSH private key file

Use these to add SSH tunneling to your capture or materialization definition, either by filling in the corresponding fields in the web app, or by working with the YAML directly. Reference the Connectors page for a YAML sample.