Setting up a development environment
Get set up to run Flow for local development.
There are currently two ways to set up a local development environment for Flow: using a Docker image, and using VS Code devcontainers.
Both methods require Docker. Install Docker now if you don't have it already.

Using a Docker image

Estuary provides a script that installs all the dependencies of Flow for you. It uses a Docker image to wrap Flow's multi-call binary, which allows you to work with Flow in your preferred development tool or directly from the command line.
Due to third-party software issues, Apple silicon (M1) hardware is not fully supported for this setup method. For now, if you are using an Apple silicon machine, we recommend the devcontainer method.
Flow's runtime is provided as an x86 Docker image. Docker for Mac has known issues running this in emulation on Apple silicon. As a result, you may experience occasional crashes, which usually present as segmentation faults.
The tutorials in the documentation are written and tested with VS Code. You may need to adapt the instructions slightly using other tools.

Install the script

Run the following command to put the script in your PATH. The following example uses the location /usr/local/bin, but you may modify as needed.
curl -OL https://raw.githubusercontent.com/estuary/flow/master/scripts/flowctl.sh
chmod 755 flowctl.sh
sudo mv flowctl.sh /usr/local/bin/flowctl.sh
sudo ln -s flowctl.sh /usr/local/bin/flowctl
For more technical details and caveats about the script, see its documentation on GitHub.

Using Visual Studio Code devcontainers

Flow includes a devcontainer, which provides a nice development experience using a self-contained Docker-based environment. This is an easy way to get a great development experience using Flow, with auto-completion set up for your Flow catalog.
Make sure that all of the following components are downloaded and installed:

Create a Git repository from the Flow Template

Visit the Flow template repository on GitHub, click on Use this template, and proceed to create your repository.

Open in VS Code

Clone your repository locally and open it in VS Code. You should see a popup in the lower right corner asking if you'd like to re-open the repository in a container. Click Re-open in container. This may take a minute or two the first time you do this, as it downloads everything and builds the container.

Verify everything works

This repository contains a "hello world" Flow project. To verify that everything is working correctly, open the terminal in VS Code and run flowctl test --source hello-world.flow.yaml. This command will exit successfully, meaning that the test passed.

Next Steps

You're now ready to start using Flow! If you're new to Flow, then we recommend going through the Flow Introductory Tutorial.
Last modified 26d ago