Twilio SendGrid HTTP Ingest (Webhook)

The Twilio SendGrid HTTP Ingest connector allows you to capture data from incoming HTTP requests from Twilio SendGrid Event Webhooks. A common use case is to capture webhook deliveries, turning them into an Estuary collection.

Usage

Configure a Twilio SendGrid Event Webhook

1. To begin, use the dashboard to create a capture. Once published, the connector overview displays a unique URL for your public endpoint.

2. In your SendGrid dashboard, navigate to Settings > Mail Settings. Under Webhook Settings, select Event Webhooks.

3. Click Create new webhook (or click the cog icon to edit an existing one). In the Post URL field, paste the webhook URL generated by your Estuary capture. See the Webhook URLs section below for how to determine your full URL.

4. Select the event types you'd like to track (e.g. Delivered, Opened, Clicked, Bounced, etc.).

5. Under Security features, enable the Enable Signed Event Webhook toggle. Click Save to generate the key pair.

6. Re-open the webhook configuration by clicking the cog icon. The public verification key will now be displayed under the signature toggle. Copy this key.

7. In your Estuary capture configuration, paste the verification key into the signatureConfig.publicKey field with provider set to "twilio". See Webhook signature verification below.

Review Twilio SendGrid's documentation on Event Webhook security features for more information.

Webhook URLs

To determine the full URL, start with the base URL from the Estuary web app (for example https://abc123-8080.us-central1.v1.estuary-data.dev), and then append the path.

The path will be whatever is in the paths endpoint configuration field (/webhook-data by default). For example, your full webhook URL would be https://<your-unique-hostname>/webhook-data. You can add additional paths to paths, and the connector will accept webhook requests on each of them. Each path will correspond to a separate binding. If you're editing the capture via the UI, click the "re-fresh" button after editing the URL paths in the endpoint config to see the resulting collections in the bindings editor. For example, if you set the path to /my-webhook.json, then the full URL for that binding would be https://<your-unique-hostname>/my-webhook.json.

Any URL query parameters that are sent on the request will be captured and serialized under /_meta/query/* in the documents. For example, a webhook request that's sent to /webhook-data?testKey=testValue would result in a document like:

{
  "_meta": {
    "webhookId": "...",
    "query": {
      "testKey": "testValue"
    },
    ...
  }
  ...
}

Authentication

The connector can optionally require each request to present an authentication token as part of an Authorization: Bearer HTTP header. To enable authentication, generate a secret and paste it into the "Require Auth Token" field. We recommend using a password manager to generate these values, but keep in mind that not all systems will be able to send values with certain special characters, so you may want to disable special characters when you generate the secret. If you enable authentication, then each incoming request must have an Authorization header with the value of your token. For example, if you use an auth token value of mySecretToken, then the header on each request must be Authorization: Bearer mySecretToken.

If you don't enable authentication, then anyone who knows the URL will be able to publish data to your collection. We recommend using authentication whenever possible.

Webhook signature verification

This connector supports ECDSA P-256 signature verification. For Twilio SendGrid webhooks, use the streamlined configuration that only requires your verification key:

{
  "signatureConfig": {
    "provider": "twilio",
    "publicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGnYX92sAfrAoZadSDc/qKMHph36YMMhXUbkrle5edS+hTngTe5x3ZziwHv/JE5R7f7YCmrQFlIWM+ghy4Lr1zA=="
  }
}

The verification key can be found in your SendGrid Event Webhook settings under Security features > Signed Event Webhook Requests.

When configured, the connector verifies the X-Twilio-Email-Event-Webhook-Signature and X-Twilio-Email-Event-Webhook-Timestamp headers on each incoming request. Requests missing valid signatures are rejected with a 401 Unauthorized response.

Configuration

Endpoint properties

Property	Title	Description	Type	Required/Default
/requireAuthToken		Optional bearer token to authenticate webhook requests. WARNING: If this is empty or unset, then anyone who knows the URL of the connector will be able to write data to your collections.	null, string	null
/paths	URL Paths	List of URL paths to accept requests at. Discovery will return a separate collection for each given path. Paths must be provided without any percent encoding, and should not include any query parameters or fragment.	null, string	null
/signatureConfig	Signature Verification	Configuration for verifying webhook signatures.	object	{"provider": "none"}

Signature Config: None

Property	Title	Description	Type	Required/Default
/signatureConfig/provider		Provider identifier	string	Required ("none")

Signature Config: Twilio SendGrid

Property	Title	Description	Type	Required/Default
/signatureConfig/provider		Provider identifier	string	Required ("twilio")
/signatureConfig/publicKey	Verification Key	Verification key from Twilio SendGrid Event Webhook settings.	string	Required

Resource properties

Property	Title	Description	Type	Required/Default
/idFromHeader		Set the /_meta/webhookId from the given HTTP header in each request. If not set, then a random id will be generated automatically. If set, then each request will be required to have the header, and the header value will be used as the value of `/_meta/webhookId`.	null, string
/path		The URL path to use for adding documents to this binding. Defaults to the name of the collection.	null, string
/stream		The name of the binding, which is used as a merge key when doing Discovers.	null, string

Sample

captures:
  ${PREFIX}/${CAPTURE_NAME}:
    endpoint:
      connector:
        image: ghcr.io/estuary/source-twilio-ingest:v1
        config:
          paths:
            - /webhook-data
          signatureConfig:
            provider: twilio
            publicKey: verification-key
    bindings:
      - resource:
          path: /webhook-data
          stream: /webhook-data
        target: ${PREFIX}/${COLLECTION_NAME}