Prerequisites for Bulk Loading Data for Azure Synapse
Before you begin, ensure that you have the following information available, and that your Microsoft Azure environment meets the following requirements.
Install Hyper-Q for Azure Synapse Analytics
Verify that you have installed Hyper-Q in the Microsoft Azure environment. To learn how to install Hyper-Q, see Install the Hyper-Q Software.
Install the Azure CLI
The Azure CLI is a command-line tool to connect to Azure and execute administrative commands on Azure resources. It allows the execution of commands through a terminal using interactive command-line prompts or scripts.
To learn how to install the Azure CLI, see the Microsoft Azure article Install the Azure CLI.
Create an Azure Storage Account and Blob Container
Azure storage accounts contain all your Azure Storage data objects, such as blobs, files, queues, and tables. The storage account provides a unique namespace for your Azure Storage data. A Blob container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.
The Blob container you create for use with Hyper-Q must have Container level access set to Private. This ensures that there is no anonymous access to the Blob container.
Have available the following information when you create the storage account:
Azure account URL.
Name of the Blob container you intend to use for storage.
Set the access level of the Blob container to Private, which is the default level. This ensures that there is no anonymous access.
Enable access using one of the following authentication methods:
Shared access signature (SAS) tokens.
Account key.
Managed Identity (System and User).
To learn how to create an Azure storage account and Blob container, see the Microsoft Azure articles Create a storage account and Create a container.
Create Managed Identities for Azure Resources
Managed identities for Azure resources provide Azure services with a managed identity in Azure Active Directory. You can create a managed identity and assign it to one or more instances of an Azure service. Azure uses the managed identity to authenticate services that support Azure AD authentication.
To use Azure Storage with Hyper-Q, you must:
Create and assign a managed identify to the Hyper-Q VM.
Create a managed identity for the Azure Synapse instance in use with Hyper-Q.
Assign the Storage Blob Data Contributor role to the managed identities you created for both Hyper-Q VM and SQL Server (Azure Synapse).
To learn about Azure managed identifies, see the Microsoft article What are managed identities for Azure resources?
Create a Managed Identity for the Hyper-Q Virtual Machine
After you create an identity, you assign the identity to the Hyper-Q VM.
To learn how to create an Azure managed identity and assign the identity to the Hyper-Q VM, see the Microsoft Azure articles Create a user-assigned managed identity and User-assigned managed identity.
Important
Azure Synapse only supports system-assigned managed identity with the COPY command used by bulk load processing. To learn more, see System-assigned managed identities for Azure resources authentication in the Microsoft Azure Synapse documentation.
Important
Ensure the user account that will run the bulk load utility has ADMINISTER DATABASE BULK OPERATIONS permissions with which to log into Hyper-Q and Azure Synapse. To learn more, see Bulk load data using the COPY statement in the Microsoft Azure Synapse documentation.
Create a SQL Server Managed Identity
Generate a Managed Identity for the Azure Synapse Analytics instance using the Azure CLI. To learn about the Azure CLI commands, see the Microsoft Azure article az Commands.
Sign into the Azure CLI using an account with the Owner role of the Azure Synapse instance.
The Owner role grants full access to manage all resources, including the ability to assign roles in Azure RBAC.
az login
Set the current active subscription.
az account set -s subscription
Update the name of the resource group and database.
az sql server update -g resource_group -n sql_server -i