Storm structure in Apache Spark

Question

You know how in Apache Storm you can use Spout streaming data for multiple bolts. Is there a way to do something like this in Apache Spark?

Basically, I want to have one program for reading data from the Kafka queue and output it to 2 different programs, which can then process it in their own way, in different ways.

In particular, there will be a reader program that will read data from the Kafka queue and output it to 2 programs x and y. x will process data to calculate indicators of one kind (in my case, it will calculate user actions), while y will calculate indicators of another type (in my case, it will check actions based on different devices).

Can someone help me understand how this is possible in Spark?

Answer 1

Why don't you just create two topologies?

• Both topologies have readings from the Kafka theme (yes, you can have several topologies reading the same topic, it works for me on production systems). Make sure you use different media configurations, otherwise kafka-zookeper will see that both topologies are the same . Check out the documentation here .

Spoutconfig - KafkaConfig, ZooKeeper , KafkaSpout. Zkroot root . .

public SpoutConfig(BrokerHosts hosts, String topic, String zkRoot, String id);

2. x x y y.

   ---

, , IMHO , ( ) , , ; kafka , , .