flow
Search…
Derivations
Derivations are the catalog entity used to transform and join data in collections.
In Flow, you can perform transformations and joins on existing collections. This process is known as a derivation, and the results are stored in a derived collection.
A given derivation can contain one or more transformations, where each transformation reads a source collection and re-structures its documents using mapping lambda functions.
Transformations rely on registers to keep track of states and apply functions to a given data point when a particular event occurs. Registers are key-based JSON documents that transformations can read and update.
The process that keys the register is called a data shuffle, and is defined by each transformation using information extracted from its source documents.
Let's look at an example of a derivation with a simple transformation:
1
derivation:
2
transform:
3
transformOnesName:
4
source:
5
name: sourceOneName
6
schema: "remote.schema.yaml#/$defs/withRequired"
7
publish: { lambda: typescript }
Copied!
Subsequent pages in this section will explore important elements of a derivation in-depth using this example.
We created the Stateful transformations and testing tutorial to show registers, lambda functions, and other important derivation concepts in action. If you haven't already, completing this tutorial will greatly enhance your understanding.
Last modified 3mo ago
Copy link