How to run EMR cluster steps at the same time?

Hi I have an EMR cluster. Whenever I send him "steps", he starts them sequentially. Is there a way to run "steps" at the same time?

Or a suitable use case for rotating multiple clusters simultaneously if you want concurrency?

+6
source share
1 answer

Each step is processed simultaneously through the cluster. Therefore, if you have work that can be done at the same time, you might want to make sure that all of this is at the same stage (each step can have 1 or more Hadoop tasks).

Typically, you can use the steps to ensure that all processing that must be performed for the next step is completed before proceeding to the next step. A good example of this is when you are dealing with encrypted data, where there can be one step to decrypt the data, one step to process the data and an additional step to re-encrypt the data before saving.

+2
source

Source: https://habr.com/ru/post/976795/


All Articles