flow
Search…
Materializations
How to bind a Flow collection to an external system and that system keep it up to date
A materialization binds a Flow collection with an external system, such as a database, and a target, such as a SQL table, into which the collection is to be continuously written.
Materializations are objects utilizing the below entities:
1
# A set of materializations to include in the catalog.
2
# Optional, type: object
3
materializations:
4
5
# The name of the materialization.
6
example/name:
7
8
# Bindings define how collections are included in the materialization.
9
# A single materialization may pull from many collections, each
10
# defined as a separate binding.
11
# Required, type: object
12
bindings:
13
14
# The source is the name of a collection to materialize. This
15
# must be defined somewhere within the catalog spec, but it may be
16
# in a separate file that is imported by, or imports, this file.
17
# Required, type: string
18
- source: example/collection
19
20
# The resource is the a freeform set of configuration values used
21
# by the specific endpoint type. Each endpoint type will require its
22
# own set of configuration values.
23
# Required, type: object
24
resource:
25
# In this example, the `sqlite` endpoint type expects a `table` key
26
# to specify the table the data will be materialized into.
27
table: example_table
28
29
# You may optionally materialize only a subset of the partitions in a partitioned
30
# collection by defining a partition selector here. Both the include and exclude fields here
31
# accept objects where the keys are the names of the fields on which the collection is
32
# logically partitioned, and the values are an array of values that name specific partitions.
33
# Optional, type: object
34
partitions:
35
36
# For each key in this object, only include a partition if the field value matches one
37
# of the values in the array. For example, here we would only materialize partitions
38
# where `myPartitionField1` exactly matches one of the given values.
39
include: { "myPartitionField1": [ "value1", "value2" ]}
40
41
# For each key in this object, exclude a partition if the field value matches one of the
42
# values in the array. For example, here we would materialize all the `myPartitionField2`
43
# partitions, except those matching one of the given values.
44
exclude: { "myPartitionField2": [ 1, 5, 7] }
45
46
# Selected projections for this materialization. Flow will select reasonable defaults depending
47
# on the type of system being materialized into. But you may control exactly which fields are
48
# included in a materialization in your fields object.
49
# Optional, type: object
50
fields:
51
52
# Whether or not to include all of the fields that are recommended for the endpoint.
53
# For databases, this means all scalar fields that have a single possible type.
54
# Required, default: true, type: boolean
55
recommended: true
56
57
# Array of projections to exclude.
58
# default: [], type: array
59
exclude: [myField, otherField]
60
61
# Fields to include. This supplements any recommended fields, where enabled.
62
# Values are passed through to the driver, e.x. for customization of the driver's
63
# schema generation or runtime behavior with respect to the field. Flow automatically
64
# adds most fields, so this is an advanced Flow feature.
65
# default: {}, type: object
66
include: {goodField: {}, greatField: {}}
67
68
# Endpoints define how to connect to the destination of the materialization.
69
endpoint:
70
71
# An endpoint has a specific connector type.
72
sqlite:
73
74
# Each type of endpoint has its own set of configuration values specific to
75
# that system.
76
path: db/example.db
77
Copied!
The Endpoint configurations page provides additional detail on supported endpoint types.
Last modified 3mo ago
Copy link