importdirective can help you easily handle all of these scenarios while keeping your catalogs well organized. Each catalog spec file may import any number of other files, and each import may refer to either relative or an absolute URL.
importin a catalog spec, you're conceptually bringing the entirety of another catalog — as well as the schemas and typescript files it uses — into your catalog. Imports are also transitive, so when you import another catalog, you're also importing everything that other catalog has imported. This allows you to keep your catalogs organized, and is flexible enough to support collaboration between separate teams and organizations.
flowctl, but is strongly recommended, since it makes your catalogs more readable and maintainable. Each directory contains a
flow.yamlfile that will import all of the catalogs from child directories.
acme/flow.yaml, might look something like this:
flowctl test --source acme/products/flow.yamlto run only the tests for product-related collections. It also allows other imports to be more granular. For example, you might want a derivation under
salesto read from
infohas a separate catalog spec,
acme/products/info/flow.yamlwithout creating a dependency on the
ourMaterializationEndpointthat points to the desired system. The
importblock might be the same for each system, but each file may use a different configuration for the endpoint, which is used by any materializations that reference it.
flowctl test dev.flow.yamland when we push to production we'll likely run
flowctl apply prod.flow.yaml.
customers. If you later try to import a catalog from the outside sales team that also contains a
customerscollection, 💥 there's a collision. A better collection name would be
acme/inside-sales/customers. This allows a catalog to include customer data from separate teams, and also separate organizations.