flow
Search…
Captures
How to bind a Flow collection to an external data source and extract data from it
A capture binds a Flow collection to an external data source, such as a cloud storage bucket or a real-time data stream, from which the collection is continuously populated. Captures are defined in the catalog spec.
Although captures are the preferred way to ingest data into Flow and the focus of future development in this area, other methods exist. Learn more.
Captures are objects utilizing the following entities:
1
# A set of captures to include in the catalog.
2
# Optional, type: object
3
captures:
4
5
# The name of the capture.
6
example/name:
7
8
# Bindings define how collections are populated from the data source. A
9
# single capture may populate many collections, each defined as a separate
10
# binding.
11
# Required, type: array
12
bindings:
13
14
- # The target is the name of a collection to populate. This
15
# must be defined somewhere within the catalog spec, but it may be
16
# in a separate file that is imported by, or imports, this file.
17
# Required, type: string
18
target: example/collection/name
19
20
# The resource includes any additional configuration required to
21
# extract data from the endpoint and map it into the collection.
22
# This is freeform configuration based on the endpoint type.
23
# Required, type: object
24
resource: {}
25
26
# Endpoints define how to connect to the source of the capture.
27
# Required, type: object
28
endpoint:
29
30
# An endpoint has a specific connector type.
31
s3:
32
33
# Each type of endpoint has its own set of configuration values specific to
34
# that system.
35
bucket: exampleS3Bucket
36
prefix: filePrefix
37
Copied!
The Endpoint configurations page provides additional detail on supported endpoint types.
Last modified 3mo ago
Copy link