This connector uses DynamoDB streams to continuously capture updates from DynamoDB tables into one or more Flow collections.
It is available for use in the Flow web application. For local development or open-source workflows,
ghcr.io/estuary/source-dynamodb:dev provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions.
To use this connector, you'll need:
One or more DynamoDB tables with DynamoDB streams enabled. To enable DynamoDB streams for a table:
- Select the table in the AWS console
- Go to the Exports and streams tab
- Click Turn on in the DynamoDB stream details section
- Select New and old images for the View type
- Click Turn on stream
An IAM user with the following permissions:
ListTableson all resources
DescribeTableon all resources
DescribeStreamon all resources
Scanon all tables used
GetRecordson all streams used
GetShardIteratoron all streams used
These permissions should be specified with the
dynamodb:prefix in an IAM policy document. For more details and examples, see Using identity-based policies with Amazon DynamoDB in the Amazon docs.
The AWS access key and secret access key for the user. See the AWS blog for help finding these credentials.
You configure connectors either in the Flow web app, or by directly editing the catalog specification file. See connectors to learn more about using connectors. The values and specification sample below provide configuration details specific to the DynamoDB source connector.
|Access Key ID||AWS Access Key ID for capturing from DynamoDB tables.||string||Required|
|Secret Access Key||AWS Secret Access Key for capturing from DynamoDB tables.||string||Required|
|AWS Region||The name of the AWS region where the DynamoDB tables are located.||string||Required|
|Backfill Table Segments||Number of segments to use for backfill table scans. Has no effect if changed after the backfill has started.||integer|
|AWS Endpoint||The AWS endpoint URI to connect to. Use if you're capturing from a compatible API that isn't provided by AWS.||string|
|Scan Limit||Limit the number of items to evaluate for each table backfill scan request.||integer|
|Table Name||The name of the table to be captured.||string||Required|
|RCU Allocation||Read capacity units the capture will attempt to consume during the table backfill. Leave blank to automatically determine based on the provisioned capacity of the table.||integer|
A minimal capture definition will look like the following:
Your capture definition may be more complex, with additional bindings for each DynamoDB table.