Skip to main content

Comparisons

Estuary is a hosted data pipeline platform. To understand what makes Estuary unique, it's useful to compare across other similar offerings in the ETL and ELT space.

For detailed feature, cost, and pro/con breakdowns, evaluate solutions in our comparison center. Common comparisons include:

The rest of this document provides technical comparison information to understand how Estuary's underlying technology and assumptions compare to other data frameworks and systems, like Kafka, Apache Beam, and Hadoop. If you are familiar with these systems, this should help you ramp up on Estuary's concepts and terminology quickly.

Platform Overview​

Estuary distinguishes itself from other platforms in a number of ways. Some of Estuary's main features include:

Learn more about who should use Estuary.

Technical Comparisons​

Because Estuary combines many functionalities, it's related to many types of data systems. Choose a familiar system from the list below to jump to an explanation of how it compares with Estuary (or how you can use the two together).

Apache Beam and Google Cloud Dataflow​

Estuary’s most apt comparison is to Apache Beam. You may use a variety of runners (processing engines) for your Beam deployment. One of the most popular, Google Cloud Dataflow, is a more robust redistribution under an additional SDK. Regardless of how you use Beam, there’s a lot of conceptual overlap with Estuary. This makes Beam and Estuary alternatives rather than complementary technologies, but there are key differences.

Like Beam, Estuary’s primary primitive is a collection. You build a processing graph (called a pipeline in Beam and a Data Flow in Estuary) by relating multiple collections together through procedural transformations, or lambdas. As with Beam, Estuary’s runtime performs automatic data shuffles and is designed to allow fully automatic scaling. Also like Beam, collections have associated schemas.

Unlike Beam, Estuary doesn’t distinguish between batch and streaming contexts. Estuary unifies these paradigms under a single collection concept, allowing you to seamlessly work with both.

Where Beam has combine operators, Estuary has reduction strategies. These are built using the declared semantics of the document’s schema, which makes it much more efficient and cost-effective to work with streaming data.

Kafka​

Estuary inhabits a different space than Kafka does by itself. Kafka is an infrastructure that supports streaming applications running elsewhere. Estuary is an opinionated framework for working with real-time data. You might think of Estuary as an analog to an opinionated bundling of several important features from the broader Kafka ecosystem.

Estuary is built on Gazette, a highly-scalable streaming broker similar to log-oriented pub/sub systems. Thus, Kafka is more directly comparable to Gazette. Estuary also uses Gazette’s consumer framework, which has similarities to Kafka consumers. Both manage scale-out execution contexts for consumer tasks, offer durable local task stores, and provide exactly-once semantics.

Journals in Gazette and Estuary are roughly analogous to Kafka partitions. Each journal is a single append-only log. Gazette has no native notion of a topic, but instead supports label-based selection of subsets of journals, which tends to be more flexible. Gazette journals store data in contiguous chunks called fragments, which typically live in cloud storage. Each journal can have its own separate storage configuration, which Estuary leverages to allow users to bring their own cloud storage buckets. Another unique feature of Gazette is its ability to serve reads of historical data by providing clients with pre-signed cloud storage URLs, which enables it to serve many readers very efficiently.

Generally, Estuary users don't need to know or care much about Gazette and its architecture, since Estuary provides a higher-level interface over groups of journals, called collections.

Estuary's collections are somewhat similar to Kafka streams, but with some important differences. Collections always store JSON and must have an associated JSON schema. Collections also support automatic logical and physical partitioning. Each collection is backed by one or more journals, depending on the partitioning.

Estuary's tasks are most similar to Kafka stream processors, but are more opinionated. Tasks fall into one of three categories: captures, derivations, and materializations. Tasks may also have more than one process, which Estuary calls shards, to allow for parallel processing. Tasks and shards are fully managed by Estuary. This includes transactional state management and zero-downtime splitting of shards, which enables turnkey scaling.

See how Estuary compares to popular stream processing platforms that use Kafka:

Or use Estuary alongside existing Kafka infrastructure and Kafka consumers:

Spark​

Spark can be described as a batch engine with stream processing add-ons, where Estuary is fundamentally a streaming system that is able to easily integrate with batch systems.

You can think of an Estuary collection as a set of RDDs with common associated metadata. In Spark, you can save an RDD to a variety of external systems, like cloud storage or a database. Likewise, you can load from a variety of external systems to create an RDD. Finally, you can transform one RDD into another. You use Estuary collections in a similar manner. They represent a logical dataset, which you can materialize to push the data into some external system like cloud storage or a database. You can also create a collection that is derived by applying stateful transformations to one or more source collections.

Unlike Spark RDDs, Estuary collections are backed by one or more unbounded append-only logs. Therefore, you don't create a new collection each time data arrives; you simply append to the existing one. Collections can be partitioned and can support extremely large volumes of data.

Spark's processing primitives, applications, jobs, and tasks, don't translate perfectly to Estuary, but we can make some useful analogies. This is partly because Spark is not very opinionated about what an application does. Your Spark application could read data from cloud storage, then transform it, then write the results out to a database. The closest analog to a Spark application in Estuary is the Data Flow. A Data Flow is a composition of Estuary tasks, which are quite different from tasks in Spark.

In Estuary, a task is a logical unit of work that does one of capture (ingest), derive (transform), or materialize (write results to an external system). What Spark calls a task is actually closer to an Estuary shard. In Estuary, a task is a logical unit of work, and shards represent the potentially numerous processes that actually carry out that work. Shards are the unit of parallelism in Estuary, and you can easily split them for turnkey scaling.

Composing Estuary tasks is also a little different than composing Spark jobs. Estuary tasks always produce and/or consume data in collections, instead of piping data directly from one shard to another. This is because every task in Estuary is transactional and, to the greatest degree possible, fault-tolerant. This design also affords painless backfills of historical data when you want to add new transformations or materializations.

Hadoop, HDFS, and Hive​

There are many different ways to use Hadoop, HDFS, and the ecosystem of related projects, several of which are useful comparisons to Estuary.

To gain an understanding of Estuary's processing model for derivations, see this blog post about MapReduce in Estuary.

HDFS is sometimes used as a system of record for analytics data, typically paired with an orchestration system for analytics jobs. If you do this, you likely export datasets from your source systems into HDFS. Then, you use some other tool to coordinate running various MapReduce jobs, often indirectly through systems like Hive.

For this use case, the best way of describing Estuary is that it completely changes the paradigm. In Estuary, you always append data to existing collections, rather than creating a new one each time a job is run. In fact, Estuary has no notion of a job like there is in Hadoop. Estuary's tasks run continuously and everything stays up to date in real time, so there's never a need for outside orchestration or coordination. Put simply, Estuary's collections are log-like, and files in HDFS typically store table-like data. This blog post explores those differences in greater depth.

To make this more concrete, imagine a hypothetical example of a workflow in the Hadoop world where you export data from a source system, perform some transformations, and then run some Hive queries.

In Estuary, you instead define a capture of data from the source, which runs continuously and keeps a collection up to date with the latest data from the source. Then you transform the data with Estuary derivations, which again apply the transformations incrementally and in real time. While you could actually use tools like Hive to directly query data from Estuary collections — the layout of collection data in cloud storage is intentionally compatible with this — you could also materialize a view of your transformation results to any database, which is also kept up to date in real time.

Fivetran, Airbyte, and other ELT solutions​

Tools like Fivetran and Airbyte are purpose-built to move data from one place to another. These ELT tools typically model sources and destinations, and run regularly scheduled jobs to export from the source directly to the destination. Estuary models things differently. Instead of modeling the world in terms of independent scheduled jobs that copy data from source to destination, Data Flows model a directed graph of captures (reads from sources), derivations (transforms), and materializations (writes to destinations). Collectively, these are called tasks.

Many ELT solutions have a concept of connectors. Where Estuary differs is that these connector tasks in Estuary are only indirectly linked. Captures read data from a source and output to collections. Estuary's collections store all the data in cloud storage, with configurable retention for historical data. You can then materialize each collection to any number of destination systems. Each one will be kept up to date in real time, and new materializations can automatically backfill all your historical data.

Collections in Estuary always have an associated JSON schema, and they use that to ensure the validity of all collection data. Tasks are also transactional and generally guarantee end-to-end exactly-once processing (so long as the endpoint system can accommodate them).

Both Fivetran and Airbyte currently have graphical interfaces that make them simple for non-technical users to configure. Estuary, too, is focused on empowering non-technical users through its dashboard and agent skills. At the same time, Estuary offers declarative YAML for configuration, which works excellently in a GitOps workflow.

dbt​

dbt is a tool that enables data analysts and engineers to transform data in their warehouses more effectively.

In addition to – and perhaps more important than – its transform capability, dbt brought an entirely new workflow for working with data: one that prioritizes version control, testing, local development, documentation, composition, and re-use.

Like dbt, Estuary uses a declarative model and tooling, but the similarities end there. dbt is a tool for defining transformations, which are executed within your analytics warehouse. Estuary is a tool for delivering data to that warehouse, as well as continuous operational transforms that are applied everywhere else.

These two tools can make lots of sense to use together. First, Estuary brings timely, accurate data to the warehouse. Within the warehouse, analysts can use tools like dbt to explore the data. The Data Flow pipeline is then ideally suited to productionize important insights as materialized views or by pushing to another destination.

Put another way, Estuary is a complete ETL platform, but you might choose to perform and manage more complex transformations in a separate, dedicated tool like dbt.

Both tools offer easy integration through your data warehouse. You can also explicitly link your materialization tasks in Estuary with dbt to kick off transformation jobs when new data lands using:

Materialize, ksqlDB, and other real-time databases​

Real-time databases like Materialize and ksqlDB consume streams of data, oftentimes from Kafka brokers, and can keep SQL views up to date in real time.

These real-time databases have some conceptual overlap with Estuary. Estuary is not itself a database; instead, it can materialize this same type of incrementally updated view into any database, regardless of whether that database has real-time capabilities or not.

However, this doesn't mean that Estuary should replace these systems in your stack. In fact, it can be optimal to use Estuary to feed data into them. Estuary adds real-time data capture and materialization options that many real-time databases don't support. Once data has arrived in the database, you have access to real-time SQL analysis and other analytical tools not native to Estuary. For further explanation, read the section below on OLAP databases.

Snowflake, BigQuery, and other OLAP databases​

Estuary differs from OLAP databases mainly in that it's not a database. Estuary has no query interface, and no plans to add one. Instead, Estuary allows you to use the query interfaces of any database by materializing views into it.

Estuary is similar to OLAP databases in that it can be the source of truth for all analytics data (though it's also capable enough to handle operational workloads). Instead of schemas and tables, Estuary defines collections. These collections are conceptually similar to database tables in the sense that they are containers for data with an associated (primary) key. Under the hood, Estuary's collections are each backed by append-only logs, where each document in the log represents a delta update for a given key.

Collections can be easily materialized into a variety of external systems, such as Snowflake or BigQuery. This creates a table in your OLAP database. With Estuary, you can choose whether to continuously keep these tables up to date with the collection or schedule updates on a specific cadence.

Like Snowflake, Estuary uses inexpensive cloud storage for all collection data. You can even bring your own storage bucket, so you retain tight control over your own data.

More data warehouses are beginning to implement capabilities to capture directly from source systems. However, these integration options often remain limited. Their main focus is often bringing data in from OLTP databases, with little support for SaaS systems, leaving data excluded from the central warehouse. And when the data movement is owned by the warehouse, other destination systems generally aren't supported, making it tricky to set up multi-destination workflows or switch to a different warehouse later on.

In contrast, Estuary can directly capture data from numerous source systems, including many SaaS platforms, and continuously and incrementally keep everything up to date. You can also materialize a given collection into multiple destination systems, so you can always use whichever system is best for the type of queries you want to run.

A common pattern is to use Estuary to capture data from multiple different sources and materialize it into a data warehouse. Estuary can also help you avoid expenses associated with queries you frequently pull from a data warehouse by keeping an up-to-date view of them where you want it. Because of Estuary’s exactly-once processing guarantees, these materialized views are always correct, consistent, and fault-tolerant.

See Estuary's warehouse and OLAP integration options.