Journals are an advanced concept of Flow. You can use Flow without knowing the details of journals, but this section may help you better understand how Flow works.
Flow collections store their data in one or more journals, resources resembling files. Journals are part of the Gazette project. See Gazette's Journal concepts page for details. The number of journals that comprise a given collection depends on how the collection is partitioned.
Journals are a real-time data lake. Historical journal data is stored as an organized layout of fragment files in cloud storage. Fragment files are regular files that collectively hold the journal's content. Just-written data is held in a replicated buffer, where it is immediately available to readers. From there, buffers are regularly persisted to your bucket for long-term storage.
Journals may be read from any offset. Readers of historical data, such as a new materialization or derivation task, fetch files directly from your bucket for efficiency and throughput. Then, as they reach the present, they automatically switch to streaming new documents within milliseconds of their being written.
All data of a collection is stored as regular JSON files
under a common and unique prefix within your cloud storage bucket.
For example, all fragment files of collection
would live under the storage prefix
Files are directly accessible by other tools. This is an important aspect of Flow's design, and it has some major implications:
- You can use tools including Snowflake, Spark, Hive, Pandas, and many others to read and process the data in your Flow collections.
- You can capture and organize data into Flow collections without knowing how it will be used quite yet. Perform ad-hoc analysis using the collection data lake, and layer in derivations or materializations later, or not at all.
- If you ever decide to stop using Flow, your data is still yours. There's no lock-in or need to migrate data out.
- Removing files from your bucket also removes them from your collection. Apply bucket lifecycle policies or directly delete files to permanently drop their contents from the collection.
Flow collections have one or more logical partitions, and each logical partition has one or more physical partitions. Every physical partition is implemented as a journal, and a journal may have many fragment files.
|Physical Partition / Journal|
Flow collections can control some aspects of how
their contents are mapped into journals through the
# Configuration for journal fragments.
# Required, type: object.
# Codec used to compress fragment files.
# One of ZSTANDARD, SNAPPY, GZIP, or NONE.
# Optional. Default is GZIP.
# Maximum flush delay before in-progress fragment buffers are closed
# and persisted. Default uses no flush interval.
# Optional. Given as a time duration.
# Desired content length of each fragment, in megabytes before compression.
# Default is 512MB.
# Optional, type: integer.
# Duration for which historical files of the collection should be kept.
# Default is forever.
# Optional. Given as a time duration.
Your storage mappings determine which of your cloud storage buckets is used for storage of collection fragment files.
Every logical partition of a Flow collection is created with a single physical partition. Later and as required, new physical partitions are added in order to increase the write throughput of the collection.
Each physical partition is responsible for all new writes
covering a range of keys occurring in collection documents.
Conceptually, if keys range from [A-Z] then one partition
might cover [A-F] while another covers [G-Z].
pivot of a partition reflects the first key
in its covered range.
One physical partition is turned into more partitions
by subdividing its range of key ownership.
For instance, a partition covering [A-F]
is split into partitions [A-C] and [D-F].
Physical partitions are journals. The relationship between the journal and its specific collection and logical partition are encoded within its journal specification.
Journal fragment files each hold a slice of your collection's content, stored as a compressed file of newline-delimited JSON documents in your cloud storage bucket.
Files are flushed to cloud storage periodically, typically after they reach a desired size threshold. They use a content-addressed naming scheme which allows Flow to understand how each file stitches into the overall journal. Consider a fragment file path like:
This path has the following components:
|Storage prefix of physical partition|
|Supplemental time pseudo-partitions|
|Beginning content offset|
|Ending content offset|
|SHA content checksum|
The supplemental time pseudo-partitions are not logical partitions, but are added to each fragment file path to facilitate integration with external tools that understand Hive layouts.
As we've seen,
collection fragment files are written to cloud storage
with path components like
If you've used tools within the Apache Hive ecosystem, this layout should feel familiar. Flow organizes files in this way to make them directly usable by tools that understand Hive partitioning, like Spark and Hive itself. Collections can also be integrated as Hive-compatible external tables in tools like Snowflake and BigQuery for ad-hoc analysis.
SQL queries against these tables can even utilize predicate push-down,
taking query predicates over
and pushing them down into the selection of files that must be read to answer
the query — often offering much faster and more efficient query execution because
far less data must be read.