flow
Search…
Lambdas
Lambdas are pure functions that power transformations.
Lambdas functions power transformations in derivations. In MapReduce terms, lambdas are mappers. The Flow runtime performs combine and reduce operations using reduction annotations provided with schemas. While reductions can be applied during captures or materializations, lambdas are only possible in derivations; thus, the full power of map-reduce, joins, and other transformations is only possible in the derivation context.
Lambdas exist as standalone files, which are referenced by derivations as in the following:
1
derivation:
2
transform:
3
transformOnesName:
4
source:
5
name: sourceOneName
6
schema: "remote.schema.yaml#/$defs/withRequired"
7
publish: { lambda: typescript }
Copied!
Lambdas are anonymous pure functions that take documents and return zero, one, or more output documents. The Flow runtime manages the execution and scale-out of lambdas. Under the hood, they're modeled as shards and make use of hot standbys for fast fail-over and scaling.
When you run a catalog containing a reference to a lambda such as the above, Flow automatically creates a stubbed-out file for you which you can simply add to the function body. When you next run the catalog, it will execute the function.
Note that this transform shows a publish lambda. Depending on the transform, you may also use an update lambda. You can learn more on the Transformations page.

TypeScript lambdas

1
import { collections, interfaces, registers } from 'flow/modules';
2
3
export class YourCollectionName implements interfaces.YourCollectionName {
4
yourTransformName(
5
source: collections.YourCollectionName,
6
_register: registers.YourCollectionName,
7
_previous: registers.YourCollectionName,
8
): collections.YourCollectionName[] {
9
// Above here is created by Flow, the below line by the user.
10
return [{ field: source.id}];
11
}
12
}
Copied!
The above is a sample TypeScript lambda. Flow will stub this file out for you and all you have to do is write the function body.
TypeScript is typed JavaScript that compiles to regular JavaScript during catalog builds, which Flow then executes on the NodeJS runtime. JSON Schemas are mapped to TypeScript types with high fidelity, enabling succinct and performant lambdas with rigorous type safety. Lambdas can also take advantage of the NPM package ecosystem.
Flow intends to support a variety of lambda languages in the future, such as Python, SQLIte, and jq.

Remote lambdas

Remote endpoints are URLs that Flow invokes via JSON POST, sending batches of input documents and expecting to receive batches of output documents in return. You can invoke them as follows:
1
publish: { lambda: {"remote": "http://example/api"} }
Copied!
Remote lambdas are a means of integrating other languages and environments into a Flow derivation. Intended uses include APIs implemented in other languages and running as serverless functions (AWS lambdas, or Google Cloud Functions).
Last modified 3mo ago