Spark Streaming: long lines / active parties

Question

can anyone indicate what reason these active parties have been hanging there for many weeks and never processed? Many thanks.

My hunch is not enough for performers, and more workers / performers will solve the problem? Or does Spark assign priority to different parties in its task scheduler?

But the situation is here, very recent parties (end of June) have been successfully processed, but parties in May are still in the queue.

I just checked Spark setup, scheduler policy - FIFO

spark.scheduler.mode    FIFO

Answer 1

Turns out the bottleneck is the master node .

node , , , .

: node EC2

Answer 2

--diver-memory --executor-memory , 10000M