Does the data stream support custom triggers or update trigger delays?

TL: DR; Is it possible to create a custom trigger that works only when a certain flag is set? Is it possible to deploy a task using a trigger with a huge delay when we know that a large data event is occurring, and then deploy the update to the task when the trigger has a normal or no delay after the event ends?

Next: Remove duplicates between triggers / firing in the window

The situation when this is most problematic (millions of duplicate releases) is that we are backfilling old data. Given that we know when this happened, I was wondering if it is possible to implement a special trigger that does not fire when the flag is set. Is that what would be possible? Alternatively, is it possible to deploy a task using a trigger that includes a huge delay when backfill occurs and then issues an update using a regular trigger when it ends?

+4
source share
1 answer

Dataflow , . Update; addDelay() --update .

+1

Source: https://habr.com/ru/post/1648600/


All Articles