This connector captures data from Amazon Kinesis streams.
It is available for use in the Flow web application. For local development or open-source workflows,
ghcr.io/estuary/source-kinesis:dev provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions.
To use this connector, you'll need:
One or more Amazon Kinesis streams. For a given capture, all streams must:
- Contain JSON data only
- Be in the same AWS region
An IAM user with the following permissions:
ListShardson all resources
GetRecordson all streams used
GetShardIteratoron all streams used
DescribeStreamon all streams used
DescribeStreamSummaryon all streams used
These permissions should be specified with the
kinesis:prefix in an IAM policy document. For more details and examples, see Controlling Access to Amazon Kinesis Data in the Amazon docs.
The AWS access key and secret access key for the user. See the AWS blog for help finding these credentials.
You configure connectors either in the Flow web app, or by directly editing the catalog specification file. See connectors to learn more about using connectors. The values and specification sample below provide configuration details specific to the Amazon Kinesis source connector.
|AWS access key ID||Part of the AWS credentials that will be used to connect to Kinesis.||string||Required, |
|AWS secret access key||Part of the AWS credentials that will be used to connect to Kinesis.||string||Required, |
|AWS endpoint||The AWS endpoint URI to connect to, useful if you're capturing from a kinesis-compatible API that isn't provided by AWS.||string|
|AWS region||The name of the AWS region where the Kinesis stream is located.||string||Required, |
A minimal capture definition will look like the following:
Your capture definition will likely be more complex, with additional bindings for each Kinesis stream.