I am working on a project in Spark and have recently switched from using Spark Standalone to Mesos to manage clusters. Now I'm confused about how to allocate resources when submitting jobs on the new system.
In offline mode, I used something like this (following the recommendations of this Cloudera blog post :
/opt/spark/bin/spark-submit
This is a cluster in which each machine has 16 cores and RAM ~ 32 GB.
What was nice about this was that I had good control over the number of executors and resources allocated for each. In the above example, I knew that I was getting 240/8 = 30 artists, each of which has 16 GB of memory and 8 cores. Given the memory on each machine in the cluster, this will be no more than two performers working on each machine. If I wanted more performers, I could do something like
/opt/spark/bin/spark-submit
Now it will give me 240/5 = 47 artists, each of which has 5 cores and 10 GB of memory, and will allow up to 3 artists per machine.
But now that I'm on the Meso, I'm a little confused. Firstly, I work in rough mode to ensure that I can fix and control the allocation of resources (this is in the service of a rather complex model where we want to pre-allocate resources).
Now I can specify --total-executor-cores and --executor-memory , but the documentation tells me that --exeuctor-cores applies only to Spark and YARN independents, which allows me to specify the total number of artists and resources allocated for each complicated. Say I ran this:
/opt/spark/bin/spark-submit --total-executor-cores 240 --executor-memory 16G --conf spark.mesos.coarse=true myscript.py
When I learn this work in the Mesos web interface, everything starts to get confused. So here are my questions:
Terminology. The web interface displays “frames” which, I believe, correspond to “tasks” in the standalone user interface. But when I click on the details for a given structure, it lists the "tasks". But they may not be the actual tasks of Spark, right? As far as I can tell, “task” here should actually mean “executor” in relation to Spark. This will correspond to the user interface, which says that my infrastructure (work) has: 15 active tasks, 240 processors and 264 GB of memory.
264/15 = 17.6, which looks like 16 GB of memory for each artist I specified (plus some overhead, I think). Am I interpreting all this correctly?
Assuming yes, when I look at any of these “tasks” (performers), I see that each of them has 16 cores. Given that we have 16 cores per machine, this seems to indicate that I basically run one performer on each of the 16 machines, and each performer receives all 16 cores, but only 16 GB of RAM. (note that even if I lowered --executor-memory down to something like 4 GB, mesos still just runs one executor per node, with 16 cores and 4 GB of RAM). But what I want to accomplish is something like my first two examples. That is, I want to run several executors per node, each of which shares RAM and the kernels of this node (that is, a moderate number of preliminary kernel executors, 5-8). Given that I cannot specify --executor-cores in Mesos, how do I do this? Or am I somehow leaving the base, even wanting to do it? Will Mesos simply not allow multiple exeuctors per node?