How to plan a pipeline in Azure Data Factory

Question

I followed the MS tutorials and created a pipeline to move data from the SQL Tablet messenger to Azure Blob, and then another pipeline to move blob data to the azure sql table. From the provided document, I need to indicate the active period (start time and end time) for the pipeline, and everything works fine.

The question is what can I do if I want the pipeline to activate every 3 hours until I manually stop the operation. Currently, I need to change the start and end times in a json script and publish it every day, and I think there should be a different way to do this.

I am new to azure, any help / comment would be appreciated, thanks.

ps I cannot perform transactional replication, since my local SQL query is on SQL2008.

Answer 1

You can set infinity as the end of the pipeline:

"end": "9999-99-99T12:00:00Z 

and in your activity add

 "scheduler": { "frequency": "Hour", "interval": 3 }, 

to set the scheduler. I do not see the possibility of manually stopping the pipeline (except for restarting).

Answer 2

It’s not entirely clear what you are trying to do from the description, but if the goal is a temporary “non-fulfillment”, you can simply suspend the pipeline itself, namely:

 "properties": { "isPaused": false, } 

Set the value to "true" in your pipeline and redeploy it and it is paused. Then you can set it to false and disable it the next day.

Keep in mind that if you have a slice operation planned for a paused period, it will work while it is paused.