flow
Search…
set
Using the set reduction strategy
set interprets the document location as an update to a set.
The location must be an object having only “add”, “intersect”, and “remove” properties. Any single “add”, “intersect”, or “remove” is always allowed.
A document with “intersect” and “add” is allowed, and is interpreted as applying the intersection to the LHS set, followed by a union with the additions.
A document with “remove” and “add” is also allowed, and is interpreted as applying the removals to the base set, followed by a union with the additions.
“remove” and “intersect” within the same document are prohibited.
Set additions are deeply merged. This makes sets behave like associative maps, where the “value” of a set member can be updated by adding it to set again, with a reducible update.
Sets may be objects, in which case the object property serves as the set item key:
1
collections:
2
- name: example/reductions/set
3
schema:
4
type: object
5
reduce: { strategy: merge }
6
properties:
7
key: { type: string }
8
value:
9
# Sets are always represented as an object.
10
type: object
11
reduce: { strategy: set }
12
# Schema for "add", "intersect", and "remove" properties
13
# (each a map of keys and their associated sums):
14
additionalProperties:
15
type: object
16
additionalProperties:
17
type: number
18
reduce: { strategy: sum }
19
# Flow requires that all parents of locations with a reduce
20
# annotation also have one themselves.
21
# This strategy therefore must (currently) be here, but is ignored.
22
reduce: { strategy: lastWriteWins }
23
24
required: [key]
25
key: [/key]
26
27
tests:
28
"Expect we can apply set operations to incrementally build associative maps":
29
- ingest:
30
collection: example/reductions/set
31
documents:
32
- { key: "key", value: { "add": { "a": 1, "b": 1, "c": 1 } } }
33
- { key: "key", value: { "remove": { "b": 0 } } }
34
- { key: "key", value: { "add": { "a": 1, "d": 1 } } }
35
- verify:
36
collection: example/reductions/set
37
documents:
38
- { key: "key", value: { "add": { "a": 2, "c": 1, "d": 1 } } }
39
- ingest:
40
collection: example/reductions/set
41
documents:
42
- { key: "key", value: { "intersect": { "a": 0, "d": 0 } } }
43
- { key: "key", value: { "add": { "a": 1, "e": 1 } } }
44
- verify:
45
collection: example/reductions/set
46
documents:
47
- { key: "key", value: { "add": { "a": 3, "d": 1, "e": 1 } } }
Copied!
Sets can also be sorted arrays, which are ordered using a provide key extractor. Keys are given as one or more JSON pointers, each relative to the item. As with merge, arrays must be pre-sorted and de-duplicated by the key, and set reductions always maintain this invariant.
Use a key extractor of [“”] to apply the natural ordering of scalar values.
Whether array or object types are used, the type must always be consistent across the “add” / “intersect” / “remove” terms of both sides of the reduction.
1
collections:
2
- name: example/reductions/set-array
3
schema:
4
type: object
5
reduce: { strategy: merge }
6
properties:
7
key: { type: string }
8
value:
9
# Sets are always represented as an object.
10
type: object
11
reduce:
12
strategy: set
13
key: [/0]
14
# Schema for "add", "intersect", & "remove" properties
15
# (each a sorted array of [key, sum] 2-tuples):
16
additionalProperties:
17
type: array
18
# Flow requires that all parents of locations with a reduce
19
# annotation also have one themselves.
20
# This strategy therefore must (currently) be here, but is ignored.
21
reduce: { strategy: lastWriteWins }
22
# Schema for contained [key, sum] 2-tuples:
23
items:
24
type: array
25
items:
26
- type: string
27
- type: number
28
reduce: { strategy: sum }
29
reduce: { strategy: merge }
30
31
required: [key]
32
key: [/key]
33
34
tests:
35
? "Expect we can apply operations of sorted-array sets to incrementally build associative maps"
36
: - ingest:
37
collection: example/reductions/set-array
38
documents:
39
- { key: "key", value: { "add": [["a", 1], ["b", 1], ["c", 1]] } }
40
- { key: "key", value: { "remove": [["b", 0]] } }
41
- { key: "key", value: { "add": [["a", 1], ["d", 1]] } }
42
- verify:
43
collection: example/reductions/set-array
44
documents:
45
- { key: "key", value: { "add": [["a", 2], ["c", 1], ["d", 1]] } }
46
- ingest:
47
collection: example/reductions/set-array
48
documents:
49
- { key: "key", value: { "intersect": [["a", 0], ["d", 0]] } }
50
- { key: "key", value: { "add": [["a", 1], ["e", 1]] } }
51
- verify:
52
collection: example/reductions/set-array
53
documents:
54
- { key: "key", value: { "add": [["a", 3], ["d", 1], ["e", 1]] } }
Copied!
Last modified 3mo ago
Copy link