Integrating Logic Apps and Azure Data Factory (v2)

Azure Data Factory (ADF) is a great Orchestration tool for integrating various data platforms within the organization. ADF is designed to create a data specific platform to migrate, transform and load across data storage with the organization. Ideally ADF is a data integration tool.

On the other hand, Azure Logic Apps is more specific for application integration where you can use it to integrate internal & external applications/services to create streamlined business processes. So, it is not focused on the data integration but on integration and connectivity within applications.

But there is no direct connector available in Azure logic apps to connect with Azure Data Factory. So, in this blog we give cover a work around using Azure automation to integrated Logic app and Azure Data Factory.

Here we will use a mail event as a trigger to pipeline in Azure Data Factory(V2). Also, we will pass the parameter file in the mail for ADF pipeline. In order to set this up, we will use

• Azure Data Factory(V2)
• Azure Automation
• Azure Logic apps

Azure Data Factory(V2) setup

We created a pipeline which will call a Data lake Analytics script. Pipeline is configured in a way that it will take parameters from a file stored in blob storage. This parameter file will be loaded by Azure logic app (will be showed at a later point). The steps to create a pipeline can be found at the Microsoft site.

Azure Automation setup

The next steps will be to create Azure Automation resource.

Once the account has been created, import the Azure Data Factory modules into the automation account. In the left pane, scroll to share resources and click modules under it. There we will find the window as below.

Similarly click on credentials under shared resource to safely save you credentials to be used in Azure automation script.

Next step will be to create runbook. In the left pane under process automation select runbook. In the runbook window choose the runbook type to be PowerShell.

Edit the runbook to write a script to invoke the pipeline in ADF as below.

Test the script and then publish the runbook. Now this runbook can be called from Azure logics apps as job.

Logic Apps Setup

Next steps would be calling the runbook on the event of a mail. For this we need Azure Logic apps account. After the account is created, click on Logic app developers in the left pane.

In the developed window, create a step to connect to outlook trigger and select when a new email arrives .So this logic app will get triggered on the event of a new mail . We have several configuration available to customize the events.

We have several configurations available to customize the events like the example given below:

As we have configured, logic app will get triggered when email with subject Automation parameter with attachment is sent to user@example.com.

Next step will be load the attached parameter file to the blob, so that ADF pipeline can use the parameter file. In order for that we need to create a Azure blob action by clicking new step as below:

Here chose create blob action. We need to mention the folder path and name of the blob to be stored. For blob content, we can choose the dynamic content which comes from the mail.

Now to remotely trigger the ADF pipeline , we have to create Azure Automation action and select create job action

In the action, we have call the runbook which has the powershell script to call the ADF pipeline.

With this we have completed the Azure logic app configuration.

Now go ahead send a mail to the mailbox configured. Logic app will automatically capture the event and invoke the ADF pipeline using Azure automation.