Skip to main content

set

set interprets the document location as an update to a set.

The location must be an object having only “add", “intersect", and “remove” properties. Any single “add", “intersect", or “remove” is always allowed.

A document with “intersect” and “add” is allowed, and is interpreted as applying the intersection to the left-hand side set, followed by a union with the additions.

A document with “remove” and “add” is also allowed, and is interpreted as applying the removals to the base set, followed by a union with the additions.

“remove” and “intersect” within the same document are prohibited.

Set additions are deeply merged. This makes sets behave like associative maps, where the “value” of a set member can be updated by adding it to the set again, with a reducible update.

Sets may be objects, in which case the object property serves as the set item key:

collections:
- name: example/reductions/set
schema:
type: object
reduce: { strategy: merge }
properties:
key: { type: string }
value:
# Sets are always represented as an object.
type: object
reduce: { strategy: set }
# Schema for "add", "intersect", and "remove" properties
# (each a map of keys and their associated sums):
additionalProperties:
type: object
additionalProperties:
type: number
reduce: { strategy: sum }
# Flow requires that all parents of locations with a reduce
# annotation also have one themselves.
# This strategy therefore must (currently) be here, but is ignored.
reduce: { strategy: lastWriteWins }

required: [key]
key: [/key]

tests:
"Expect we can apply set operations to incrementally build associative maps":
- ingest:
collection: example/reductions/set
documents:
- { key: "key", value: { "add": { "a": 1, "b": 1, "c": 1 } } }
- { key: "key", value: { "remove": { "b": 0 } } }
- { key: "key", value: { "add": { "a": 1, "d": 1 } } }
- verify:
collection: example/reductions/set
documents:
- { key: "key", value: { "add": { "a": 2, "c": 1, "d": 1 } } }
- ingest:
collection: example/reductions/set
documents:
- { key: "key", value: { "intersect": { "a": 0, "d": 0 } } }
- { key: "key", value: { "add": { "a": 1, "e": 1 } } }
- verify:
collection: example/reductions/set
documents:
- { key: "key", value: { "add": { "a": 3, "d": 1, "e": 1 } } }

Sets can also be sorted arrays, which are ordered using a provide key extractor. Keys are given as one or more JSON pointers, each relative to the item. As with merge, arrays must be pre-sorted and de-duplicated by the key, and set reductions always maintain this invariant.

Use a key extractor of [“”] to apply the natural ordering of scalar values.

Whether array or object types are used, the type must always be consistent across the “add” / “intersect” / “remove” terms of both sides of the reduction.

collections:
- name: example/reductions/set-array
schema:
type: object
reduce: { strategy: merge }
properties:
key: { type: string }
value:
# Sets are always represented as an object.
type: object
reduce:
strategy: set
key: [/0]
# Schema for "add", "intersect", & "remove" properties
# (each a sorted array of [key, sum] 2-tuples):
additionalProperties:
type: array
# Flow requires that all parents of locations with a reduce
# annotation also have one themselves.
# This strategy therefore must (currently) be here, but is ignored.
reduce: { strategy: lastWriteWins }
# Schema for contained [key, sum] 2-tuples:
items:
type: array
items:
- type: string
- type: number
reduce: { strategy: sum }
reduce: { strategy: merge }

required: [key]
key: [/key]

tests:
? "Expect we can apply operations of sorted-array sets to incrementally build associative maps"
: - ingest:
collection: example/reductions/set-array
documents:
- { key: "key", value: { "add": [["a", 1], ["b", 1], ["c", 1]] } }
- { key: "key", value: { "remove": [["b", 0]] } }
- { key: "key", value: { "add": [["a", 1], ["d", 1]] } }
- verify:
collection: example/reductions/set-array
documents:
- { key: "key", value: { "add": [["a", 2], ["c", 1], ["d", 1]] } }
- ingest:
collection: example/reductions/set-array
documents:
- { key: "key", value: { "intersect": [["a", 0], ["d", 0]] } }
- { key: "key", value: { "add": [["a", 1], ["e", 1]] } }
- verify:
collection: example/reductions/set-array
documents:
- { key: "key", value: { "add": [["a", 3], ["d", 1], ["e", 1]] } }