Elasticsearch
This connector materializes Flow collections into indices in an Elasticsearch cluster.
It is available for use in the Flow web application. For local development or open-source workflows, ghcr.io/estuary/materialize-elasticsearch:v2
provides the latest version of the connector as a Docker image. You can also follow the link in your browser to see past image versions.
Prerequisites
To use this connector, you'll need:
- An Elastic cluster with a known endpoint
- If the cluster is on the Elastic Cloud, you'll need an Elastic user with a role that grants all privileges on indices you plan to materialize to within the cluster. See Elastic's documentation on defining roles and security privileges.
- At least one Flow collection
If you haven't yet captured your data from its external source, start at the beginning of the guide to create a dataflow. You'll be referred back to this connector-specific documentation at the appropriate steps.
Configuration
To use this connector, begin with data in one or more Flow collections. Use the below properties to configure an Elasticsearch materialization, which will direct the contents of these Flow collections into Elasticsearch indices.
Authentication
You can authenticate to Elasticsearch using either a username and password, or using an API key.
The connector will automatically create an Elasticsearch index for each binding of the materialization. It uses the last component of the collection name as the name of the index by default. You can customize the name of the index using the index
property in the resource configuration for each binding. You can also create the indices yourself and just enter the name as the index
. This allows you full control over the index creation properties.
Elasticsearch Mappings
If you let the connector create the indices automatically, it will configure it to use dynamic runtime mappings. This allows you to search based on any fields in your source documents, and avoids any issues that could be caused by the creation of too many mappings. It's recommended that you add explicit mappings as you identify the need, in order to keep your Elasticsearch queries fast.
Properties
Endpoint
Property | Title | Description | Type | Required/Default |
---|---|---|---|---|
**** | Elasticsearch Connection | object | Required | |
/endpoint | Endpoint | Endpoint host or URL. If using Elastic Cloud this follows the format https://CLUSTER_ID.REGION.CLOUD_PLATFORM.DOMAIN:PORT | string | Required |
/credentials | object | |||
/credentials/username | Username | Username to use for authenticating with Elasticsearch | string | |
/credentials/password | Password | Password to use for authenticating with Elasticsearch | string | |
/credentials/apiKey | API Key | API Key to use for authenticating with Elasticsearch | string |
Bindings
Property | Title | Description | Type | Required/Default |
---|---|---|---|---|
/delta_updates | Delta updates | Whether to use standard or delta updates | boolean | false |
/index | index | Name of the ElasticSearch index to store the materialization results. | string | Required |
/number_of_replicas | Number of replicas | The number of replicas in ElasticSearch index. If not set, default to be 0. For single-node clusters, this must be 0. For production systems, a value of 1 or more is recommended | integer | 0 |
/number_of_shards | Number of shards | The number of shards in ElasticSearch index. Must be greater than 0. | integer | 1 |
Sample
materializations:
PREFIX/mat_name:
endpoint:
connector:
# Path to the latest version of the connector, provided as a Docker image
image: ghcr.io/estuary/materialize-elasticsearch:dev
config:
endpoint: https://ec47fc4d2c53414e1307e85726d4b9bb.us-east-1.aws.found.io:9243
credentials:
username: flow_user
password: secret
# If you have multiple collections you need to materialize, add a binding for each one
# to ensure complete data flow-through
bindings:
- resource:
index: my-elasticsearch-index
source: PREFIX/source_collection
Delta updates
This connector supports both standard and delta updates. You must choose an option for each binding.
Learn more about delta updates and the implications of using each update type.