Spark StreamingContext awaitTerminationOrTimeout

Question

I'm calling streamingContext.awaitTerminationOrTimeout(timeout), but I want to make it environment dependent timeout.

This means that I want to stop my work if my environment is UAT, but I do not want it to timeout at all, if my environment is production.

I checked the documentation here , but I can’t find any links to see if the transmission went through 0or -1how the timeout will just start the task forever (or until it works).

Are there any possible values timeoutthat could do the trick? Or any other alternatives that do not imply a challenge awaitTerminationfor my production environment and awaitTerminationOrTimeoutfor my UAT environment?

Answer 1

After looking at the library, I see that any negative value will do.

awaitTerminationOrTimeout(timeout)internally calls streamingContext.awaitTerminationOrTimeout(timeout), which then calls contextWriter.waitForStopOrError(timeout).

Inside waitForStopOrErrorthere is a condition that states that if timeoutless than zero, we expect.

TL: DR : -1will never allow a job timeout.