flow

Search…

Overview

Getting Started

Concepts

set

Using the set reduction strategy

`set`

interprets the document location as an update to a set.The location must be an object having only “add”, “intersect”, and “remove” properties. Any single “add”, “intersect”, or “remove” is always allowed.

A document with “intersect” and “add” is allowed, and is interpreted as applying the intersection to the LHS set, followed by a union with the additions.

A document with “remove” and “add” is also allowed, and is interpreted as applying the removals to the base set, followed by a union with the additions.

“remove” and “intersect” within the same document are prohibited.

Set additions are deeply merged. This makes sets behave like associative maps, where the “value” of a set member can be updated by adding it to set again, with a reducible update.

Sets may be objects, in which case the object property serves as the set item key:

1

collections:

2

- name: example/reductions/set

3

schema:

4

type: object

5

reduce: { strategy: merge }

6

properties:

7

key: { type: string }

8

value:

9

# Sets are always represented as an object.

10

type: object

11

reduce: { strategy: set }

12

# Schema for "add", "intersect", and "remove" properties

13

# (each a map of keys and their associated sums):

14

additionalProperties:

15

type: object

16

additionalProperties:

17

type: number

18

reduce: { strategy: sum }

19

# Flow requires that all parents of locations with a reduce

20

# annotation also have one themselves.

21

# This strategy therefore must (currently) be here, but is ignored.

22

reduce: { strategy: lastWriteWins }

23

24

required: [key]

25

key: [/key]

26

27

tests:

28

"Expect we can apply set operations to incrementally build associative maps":

29

- ingest:

30

collection: example/reductions/set

31

documents:

32

- { key: "key", value: { "add": { "a": 1, "b": 1, "c": 1 } } }

33

- { key: "key", value: { "remove": { "b": 0 } } }

34

- { key: "key", value: { "add": { "a": 1, "d": 1 } } }

35

- verify:

36

collection: example/reductions/set

37

documents:

38

- { key: "key", value: { "add": { "a": 2, "c": 1, "d": 1 } } }

39

- ingest:

40

collection: example/reductions/set

41

documents:

42

- { key: "key", value: { "intersect": { "a": 0, "d": 0 } } }

43

- { key: "key", value: { "add": { "a": 1, "e": 1 } } }

44

- verify:

45

collection: example/reductions/set

46

documents:

47

- { key: "key", value: { "add": { "a": 3, "d": 1, "e": 1 } } }

Copied!

Sets can also be sorted arrays, which are ordered using a provide

`key`

extractor. Keys are given as one or more JSON pointers, each relative to the item. As with `merge`

, arrays must be pre-sorted and de-duplicated by the key, and set reductions always maintain this invariant.Use a key extractor of

`[“”]`

to apply the natural ordering of scalar values.Whether array or object types are used, the type must always be consistent across the “add” / “intersect” / “remove” terms of both sides of the reduction.

1

collections:

2

- name: example/reductions/set-array

3

schema:

4

type: object

5

reduce: { strategy: merge }

6

properties:

7

key: { type: string }

8

value:

9

# Sets are always represented as an object.

10

type: object

11

reduce:

12

strategy: set

13

key: [/0]

14

# Schema for "add", "intersect", & "remove" properties

15

# (each a sorted array of [key, sum] 2-tuples):

16

additionalProperties:

17

type: array

18

# Flow requires that all parents of locations with a reduce

19

# annotation also have one themselves.

20

# This strategy therefore must (currently) be here, but is ignored.

21

reduce: { strategy: lastWriteWins }

22

# Schema for contained [key, sum] 2-tuples:

23

items:

24

type: array

25

items:

26

- type: string

27

- type: number

28

reduce: { strategy: sum }

29

reduce: { strategy: merge }

30

31

required: [key]

32

key: [/key]

33

34

tests:

35

? "Expect we can apply operations of sorted-array sets to incrementally build associative maps"

36

: - ingest:

37

collection: example/reductions/set-array

38

documents:

39

- { key: "key", value: { "add": [["a", 1], ["b", 1], ["c", 1]] } }

40

- { key: "key", value: { "remove": [["b", 0]] } }

41

- { key: "key", value: { "add": [["a", 1], ["d", 1]] } }

42

- verify:

43

collection: example/reductions/set-array

44

documents:

45

- { key: "key", value: { "add": [["a", 2], ["c", 1], ["d", 1]] } }

46

- ingest:

47

collection: example/reductions/set-array

48

documents:

49

- { key: "key", value: { "intersect": [["a", 0], ["d", 0]] } }

50

- { key: "key", value: { "add": [["a", 1], ["e", 1]] } }

51

- verify:

52

collection: example/reductions/set-array

53

documents:

54

- { key: "key", value: { "add": [["a", 3], ["d", 1], ["e", 1]] } }

Copied!

Last modified 3mo ago

Copy link