Skip to main content

Datometry Documentation

Configure Hyper-Q for use with Azure Blob Storage

To configure Hyper-Q to bulk load data, perform the following steps on each Hyper-Q VM.

  1. Log into each Hyper-Q VM in the Azure workspace using SSH.

  2. Open the file /opt/datometry/config/dtm.ini in a text editor.

  3. In the [gateway] section of the dtm.ini file, configure the following values.

Table 3. Azure Blob Storage Configuration Parameters

Parameters

Description

bulk_load_managed_identity = user | system

Specify either user or system. The user parameter specifies a user-assigned managed identify. The system parameter specifies a system-assigned managed identity.

bulk_load_managed_identity_client_id

The client ID of the user-assigned managed identity.

Note

You cannot use a system-assigned managed identity with this parameter.

blob_store_container

Name of the Blob container you intend to use for storage.

blob_store_url

HTTP address of the Blob storage container.

blob_store_key

Azure Blob storage account key.

Note

If you are using managed identity, you do not need to specify the Azure Blob storage account key using this parameter.

blob_store_sas_token

(Optional) Azure Blob storage SAS token.

This parameter is required if you are authenticating using a Storage Account Key.

bulk_load_utility = ms_copy_api

Specify the ms_copy_api.

The ms_copy_api option uses the Azure Blob storage REST API to efficiently bulk load data into Azure Synapse.

bulk_load_chunk_size = 10MB

(Optional) Lets you specify the bulk load chunk size to optimize performance depending on the workload or instance size. The default and recommended chunk size is 10MB.



Example Azure Blob Storage Configuration

The following example shows the parameters to configure Azure Blob storage in the dtm.ini configuration file. This example uses Managed Identity as the authentication method.

"gateway".bulk_load_managed_identity = user 
"gateway".bulk_load_managed_identity_client_id = "123e4567-e89b-12d3-a456-426614174000" 
"gateway".blob_store_container = "hyperq-container" 
"gateway".blob_store_url = https://storage_account_name.blob.core.windows.net 
"gateway".blob_load_utility = "ms_copy_api"