Microsoft Azure Fabric Warehouse
This connector materializes Flow collections into tables in Microsoft Azure Fabric Warehouse.
ghcr.io/estuary/azure-fabric-warehouse:dev
provides the latest connector image. You can also follow the link in your
browser to see past image versions.
Prerequisites
To use this connector, you'll need:
- The connection string for a Fabric Warehouse. See instructions for finding the connection string.
- A service principal for connecting to the warehouse. The Client ID and
Client Secret are needed to configure the connector.
- Follow this guide to register a Microsoft Entra app and create a service principal. Use Option 3: Create a new client secret to create the service principal and save its client secret.
- Follow these instructions for enabling service principal access to Fabric APIs.
- Assign the service principal the Contributor role for the workspace as described here.
- A Storage Account Key for a storage account that will be used to store temporary staging files that will be loaded into your warehouse. Follow this guide to create a storage account. You can find your storage account key using these instructions.
- The name of the container within the storage account for storing staging files.
Configuration
Use the below properties to configure the materialization, which will direct one or more of your Flow collections to your tables.
Properties
Endpoint
| Property | Title | Description | Type | Required/Default |
|---|---|---|---|---|
/clientID | Client ID | Client ID for the service principal used to connect to the Azure Fabric Warehouse. services. | string | Required |
/clientSecret | Client Secret | Client Secret for the service principal used to connect to the Azure Fabric Warehouse. services. | string | Required |
/warehouse | Warehouse | Name of the Azure Fabric Warehouse to connect to. to. | string | Required |
/schema | Schema | Schema for bound collection tables (unless overridden within the binding resource configuration) as well as associated materialization metadata tables. | string | Required |
/connectionString | Connection String | SQL connection string for the Azure Fabric Warehouse. | string | Required |
/storageAccountName | Storage Account Name | Name of the storage account that temporary files will be written to. | string | Required |
/storageAccountKey | Storage Account Key | Storage account key for the storage account that temporary files will be written to. | string | Required |
/containerName | Storage Account Container Name | Name of the container in the storage account where temporary files will be written. | string | Required |
/directory | Directory | Optional prefix that will be used for temporary files. | string |
Bindings
| Property | Title | Description | Type | Required/Default |
|---|---|---|---|---|
/table | Table | Table name | string | Required |
/schema | Alternative Schema | Alternative schema for this table | string | |
/delta_updates | Delta updates | Whether to use standard or delta updates | boolean |
Sync Schedule
This connector supports configuring a schedule for sync frequency. You can read about how to configure this here.
Delta updates
This connector supports both standard (merge) and delta updates. The default is to use standard updates.
Enabling delta updates will prevent Flow from querying for documents in your tables, which can reduce latency and costs for large datasets. If you're certain that all events will have unique keys, enabling delta updates is a simple way to improve performance with no effect on the output. However, enabling delta updates is not suitable for all workflows, as the resulting table won't be fully reduced.