There are two ways to work with Flow: through the web app, and using the flowctl command-line interface. With flowctl, you can work on drafts and active catalogs created in the webapp with a higher degree of control. You can also authenticate Flow users and roles and generate Typescript modules to write custom transformations for your derivations.
flowctl is the only Flow binary that you need to work with, so distribution and upgrades are all simple.
Simplified installation is coming soon.
For now, you can contact Estuary support for access.
flowctl includes several top-level subcommands representing different functional areas. Each of these include multiple nested subcommands. Important top-level flowctl subcommands are described below.
authallows you to authenticate your development session in your local development environment. It's also how you provision Flow roles and users. Learn more about authentication.
catalogallows you to work with your organization's current active catalog. You can investigate the current deployment, or add its specification to a draft, where you can develop it further.
draftallows you to work with draft catalog specifications. You can create, test, develop locally, and then publish, or deploy, them.
You can access full documentation of all flowctl subcommands from the command line by passing the
-h flag, for example:
flowctl --helplists top-level flowctl subcommands
flowctl catalog --helplists subcommands of
When building Flow catalogs,
flowctl uses a build directory
which is typically the root directory of your project, and is controlled by flag
Within this directory,
flowctl creates a number of files and sub-directories.
Except where noted, it's recommended that these outputs be committed within your GitOps project.
flow_generated/: ♻ Directory of generated files, including TypeScript classes and interfaces. See TypeScript code generation.
dist/should be added to your
node_modules/: ♻ Location where
npmwill download your TypeScript dependencies.
node_modules/should be added to your
package-lock.json: ♻ Files used by
package.json, but its
dependenciesstanza will be overwritten by the npmDependencies of your catalog source files.
.eslintrc.js: ⚓ Configures the TypeScript linter that's run as part of the catalog build process. Linting supplements TypeScript compilation to catch additional common mistakes and errors at build time.
.prettierrc.js: ⚓ Configures the formatter that's used to format your TypeScript files.
⚓: Generated only if it does not exist. Never modified or deleted by
flowctl re-generates and overwrites contents.
TypeScript code generation
TypeScript files are used in the Flow catalog both as part of the automatic build process, and to define lambdas functions for derivations, which requires your input.
As part of the catalog build process, Flow translates your
into equivalent TypeScript types on your behalf.
These definitions live within
flow_generated/ in your catalog build directory,
and are frequently over-written by invocations of
Files in this subdirectory are human-readable and stable.
You may want to commit them as part of a GitOps-managed project, but this isn't required.
Whenever you define a derivation that uses a lambda,
you must define the lambda in an accompanying TypeScript module, and reference that module
in the derivation's definition. To facilitate this,
you can generate a stub of the module using
flowctl typescript generate
and simply write the function bodies.
Learn more about this workflow.
If a TypeScript module exists,
flowctl will never overwrite it,
even if you update or expand your catalog sources such that the required interfaces have changed.