While dealing with a requirement to interact with a feed of data from a legacy system I decided to look into how I could leverage the existing ETL workflows in Azure Data Factory (ADF). There is not curenty a connector in ADF for Azure Service Bus (ASB) however ASB does have a simple REST interface. This article is going to focus on sending single messages to a Queue or Topic.
Sending Data from ADF to ASB
To send data from ADF to ASB Create a Web Request and fill in the following properties:
URL : https://{service Bus Namespace}.servicebus.windows.net/{Queue or Topic Name}/messages i.e. https://myasb.servicebus.windows.net/MyTopic/Messages
Method : POST
Headers : Any custom headers you would like
Body : @item This is the data passed in
Authentication : System Assigned Managed Idetity this will use the managed Identity assigned the Data Factory
Resource : https://servicebus.azure.net/ this is the authorisation resource to get the Access token for
Securing the connection
Configuring ASB as above will use Azure Active Directory Authentication, sometimes called Managed Identity, which is a nice managed wrapper over oAuth2. To Grant access from your ADF to the ASB Topic the following IAM role Azure Service Bus Data Sender needs to be granted on the Topic or Queue
When looking for the Managed Identity, don’t forget to change the “Assign Access to” Managed Identity as below
Passing data to the Web Request
To sending data to a topic or queue with one item per message, a Lookup task, and a ForEach iterator can be used
Using the lookup task, Select the data you wish to send to the Topic or Queue
Using the ForEach iterator pass in the output from the previous item
Items @activity('Lookup All new Books').output
When triggered the Pipeline will now place one message per row onto the ASB in JSON format, this will be fine for a small number of rows however can take quite some time for larger ones given the overhead of the Web Request task for each row.