flowctl
There are two ways to work with Flow: through the web app, and using the flowctl command-line interface. With flowctl, you can work on drafts and active catalogs created in the webapp with a higher degree of control. You can also authenticate Flow users and roles and generate Typescript modules to write custom transformations for your derivations.
flowctl is the only Flow binary that you need to work with, so distribution and upgrades are all simple.
Installation
Beta
Simplified installation is coming soon.
For now, you can contact Estuary support for access.
flowctl subcommands
flowctl includes several top-level subcommands representing different functional areas. Each of these include multiple nested subcommands. Important top-level flowctl subcommands are described below.
auth
allows you to authenticate your development session in your local development environment. It's also how you provision Flow roles and users. Learn more about authentication.catalog
allows you to work with your organization's current active catalog. You can investigate the current deployment, or add its specification to a draft, where you can develop it further.draft
allows you to work with draft catalog specifications. You can create, test, develop locally, and then publish, or deploy, them.
You can access full documentation of all flowctl subcommands from the command line by passing the --help
or -h
flag, for example:
flowctl --help
lists top-level flowctl subcommandsflowctl catalog --help
lists subcommands ofcatalog
Build directory
When building Flow catalogs, flowctl
uses a build directory
which is typically the root directory of your project, and is controlled by flag --directory
.
Within this directory, flowctl
creates a number of files and sub-directories.
Except where noted, it's recommended that these outputs be committed within your GitOps project.
flow_generated/
: ♻ Directory of generated files, including TypeScript classes and interfaces. See TypeScript code generation.dist/
: ♻ Holds JavaScript and source map files produced during TypeScript compilation.dist/
should be added to your.gitignore
.node_modules/
: ♻ Location wherenpm
will download your TypeScript dependencies.node_modules/
should be added to your.gitignore
.package.json
andpackage-lock.json
: ♻ Files used bynpm
to manage dependencies and your catalog's associated JavaScript project. You may customizepackage.json
, but itsdependencies
stanza will be overwritten by the npmDependencies of your catalog source files..eslintrc.js
: ⚓ Configures the TypeScript linter that's run as part of the catalog build process. Linting supplements TypeScript compilation to catch additional common mistakes and errors at build time..prettierrc.js
: ⚓ Configures the formatter that's used to format your TypeScript files.
Legend
⚓: Generated only if it does not exist. Never modified or deleted by flowctl
.
♻: flowctl
re-generates and overwrites contents.
TypeScript code generation
TypeScript files are used in the Flow catalog both as part of the automatic build process, and to define lambdas functions for derivations, which requires your input.
As part of the catalog build process, Flow translates your
schemas
into equivalent TypeScript types on your behalf.
These definitions live within flow_generated/
in your catalog build directory,
and are frequently over-written by invocations of flowctl
.
Files in this subdirectory are human-readable and stable.
You may want to commit them as part of a GitOps-managed project, but this isn't required.
Whenever you define a derivation that uses a lambda,
you must define the lambda in an accompanying TypeScript module, and reference that module
in the derivation's definition. To facilitate this,
you can generate a stub of the module using flowctl typescript generate
and simply write the function bodies.
Learn more about this workflow.
If a TypeScript module exists, flowctl
will never overwrite it,
even if you update or expand your catalog sources such that the required interfaces have changed.