All geek questions in one place

External system requests during Kafka Stream processing

I am trying to create a streaming architecture for streaming analytics. Requirements:

  • Entering RT and NRT streams
  • Stream processors implementing some financial analysis
  • The output stream of RT and NRT analysis.
  • Reference requests for stream processing

I am studying Kafka and Kafka streams for real-time RT / NRT streaming and messaging. My question is: I need to fulfill some requests to external systems (info providers, MongoDB, etc. etc.) during streaming processing. These queries can be either synchronization or asynchronous req-response based on the characteristics of the external system.

I read this post explaining how to join KStream and KTable during processing, and it is very interesting, but in this case KTable is not dependent on the input parameters coming from KStream, it is just a streaming representation of the table.

I need to request an external system to send a KStream message, pass some message fields as request parameters and enrich the streaming message with the query result, and then publish the enriched message in the output subject. Is there a consolidated paradigm for developing this stream processing? Is there any specific technology that I would use better? Remember that requests can be synchronized and asynchronous.

, RPC, Kafka Stream. - /? Akka , , Akka -.

+4
1

:

  • : Kafka Connect Kafka KTable KStream-KTable.
  • UDF. KStream #mapValues(), #map() , #transform() #process(). , .
    • sync lookups: , ( #mapValues(), , )
    • async lookpus: , ( - , ). , , (.. , , ). , - , , ( async, UDF). . , . , Streams, .

: DSL Kafka Streams

+7

Source: https://habr.com/ru/post/1668915/

More articles:


All Articles