I have a storm topology for processing messages from Kafka, as well as an HTTP call / save in Cassandra based on the task. I process messages as soon as they arrive. More than ever, several messages are not completely processed due to external response forms such as HTTP. I would like to implement an exponential deferral mechanism for retrying in the case where the HTTP server does not respond / does not return an error message, so I can try again after some time. I could come up with several ideas in which I could reach them. I would like to know which one would be the best solution, also if there is another solution that I can use that is fault tolerant. Since this is used to implement exponential delay, each message will have a different delay time.
- Submit another topic to Kafka , which will be used later. My preferred solution . I know that we can use Kafka's compensation to absorb the message in the last step. Be that as it may, I could not find the documentation / sample code to do the same. It will be really helpful if anyone can help me with this.
- Write a Cassandra / Redis message and write a scheduler to receive messages that are not processed and ready for use, and send it to Kafka so that my storm topology can use it. (Existing solution in another legacy project (Non Storm))
- Submit to Beanstalk with a delay (existing solution in another Non Storm project. How would I like to avoid using this solution and use it only if I'm not in the configuration).
Although this is pretty much what I would like to do. I cannot find documentation for implementing delayProcessingUntil as indicated in Kafka Deployment - Delayed Queue using high-level consumer
I did the scheduled work from the data warehouse and delayed the use of Beanstalk in the past, but I would prefer to use Kafka.
source
share