Skip to main content

Google Cloud Pub/Sub

This connector materializes Flow collections into topics in Google Cloud Pub/Sub.

It is available for use in the Flow web application. For local development or open-source workflows, ghcr.io/estuary/materialize-google-pubsub:dev provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions.

Prerequisites

To use this connector, you'll need:

tip

If you haven't yet captured your data from its external source, start at the beginning of the guide to create a dataflow. You'll be referred back to this connector-specific documentation at the appropriate steps.

OAuth authentication using the Flow web app

OAuth is the simplest authentication method, and is supported in the Flow web app. You'll need:

You'll supply this account's username and password to authenticate.

Manual authentication

Manual authentication is the only method supported when using flowctl to develop locally. You'll need:

See the Google documentation for help creating a new service account and generating its key.

Configuration

To use this connector, begin with data in one or more Flow collections. Use the below properties to configure a Google Cloud Pub/Sub materialization, which will direct one or more of your Flow collections to your desired Pub/Sub topics.

Properties

Endpoint

PropertyTitleDescriptionTypeRequired/Default
/credentialsAuthenticationCredentials used to authenticate with Google.array, boolean, null, number, object, stringRequired
/credentials/auth_typeAuthentication typeSet to Service for manual authentication, or use OAuth in the web app.string
/credentials/credentials_jsonService Account JSONThe JSON key of the service account to use for authorization, if configuring manually.string
/project_idGoogle Cloud Project IDName of the project containing the PubSub topics for this materialization.stringRequired

Bindings

caution

PubSub topics need a default subscription; otherwise, delivered messages will be lost. Leave Create with Default Subscription set to the default, true, unless you have a specific reason not to do so.

PropertyTitleDescriptionTypeRequired/Default
/create_default_subscriptionCreate with Default SubscriptionCreate a default subscription when creating the topic. Will be created as "<topic>-sub". Has no effect if the topic already exists.booleanRequired, true
identifierResource Binding IdentifierOptional identifier for the resource binding if creating a multiplex topic. Included as \"identifier\" attribute in published messages if specified.string
/topicTopic NameName of the topic to publish materialized results to.stringRequired

Sample

This sample reflects the manual authentication method using the CLI.

materializations:
${PREFIX}/${mat_name}:
endpoint:
connector:
config:
credentials:
auth_type: Service
credentials_json: {secret}
project_id: my_google_cloud_project
bindings:
- resource:
create_default_subscription: true
topic: my_new_topic
source: ${PREFIX}/${source_collection}

Multiplex topics

You can materialize multiple Flow collections to the same Pub/Sub topic. This is known as a multiplex topic. You do so by adding the optional identifier field to the binding configuration.

When materializing to a multiplex topic, ensure that:

  • The bindings you want to combine have the same topic name.
  • Each binding pulls from a different Flow collection
  • Each binding has a unique identifier. It can be anything you'd like.

The binding configuration will look similar to:

bindings:
- resource:
identifier: one
topic: multiplex-topic
source: ${PREFIX}/source_collection_one
- resource:
identifier: two
topic: multiplex-topic
source: ${PREFIX}/source_collection_two

Delta updates

Because Google Cloud Pub/Sub is a write-only event-streaming system, this connector uses only delta updates.

Message ordering

Google Cloud Pub/Sub manages message ordering using ordering keys.

This connector sets the ordering key of published messages using the Flow collection key of the documents being being published. Messages are published in order, on a per-key basis.