Working with logs and statistics
are useful for debugging and monitoring catalog tasks.
Access to statistics is still a work in progress. For now, this documentation deals strictly with logs.
You can access logs by materializing them to an external endpoint, or from the command line.
Accessing logs from the command line
flowctl logs subcommand allows you to print logs from the command line.
This method allows more flexibility and is ideal for debugging.
You can retrieve logs for any task that is part of a catalog that is currently deployed.
Printing logs for a specific task
You can print logs for a given deployed task using the flag
--task followed by the task name.
flowctl logs --task acmeCo/anvils/capture-one
Printing all logs for a prefix
You can print all logs for currently deployed catalogs of a given prefix using the flag
flowctl logs --tenant acmeCo
This is the same as printing the entire contents of the collection
Printing logs by task type
Within a given prefix, you can print logs for all deployed tasks of a given type using the flag
--task-type followed by one of
flowctl logs --tenant acmeCo --task-type capture
Accessing logs by materialization
You can materialize your
logs collection to an external system.
This is typically the preferred method if you’d like to continuously work with or monitor logs.
It's easiest to materialize the whole collection, but you can use a partition selector to only materialize specific tasks, as the
logs collection is partitioned on tasks.
Be sure to add a partition selector to exclude the logs of the materialization itself. Otherwise, you could trigger an infinite loop in which the connector materializes its own logs, logs that event, and so on.
# Exclude the logs of this materialization to avoid an infinite loop.