Using qsub (sge) with multi-threaded applications

Question

I wanted to send a multi-threaded job to the cluster network I'm working with - but the qsub man page is not clear how to do this. By default, I assume that it simply sends it as normal work, regardless of multithreading, but this can cause problems, that is, send many multithreaded jobs to the same computer, slowing down the work.

Does anyone know how to do this? Thank you

The batch server system is sge.

Answer 1

In SGE / UGE, the configuration is set by the administrator, so you should check what they called parallel environments

qconf -spl make our_paraq 

find one with $pe_slots in config

 qconf -sp make qconf -sp our_paraq 

qsub with this environment and the number of cores you want to use

 qsub -pe our_paraq 8 -cwd ./myscript 

If you use mpi, you have more options for the configuration distribution rule ( $pe_slots above), for example $round_robin and $fill_up , but this should force you.

Answer 2

If your work is multithreaded, you can take advantage of multithreading even in SGE. A single SGE job can use one or more processors. If you submit a job using one processor, and you have a program that has many threads than one processor can handle, a problem arises. Check how many processors your work uses and how many threads per processor created by your program.

In my case, I have a java program that uses one processor with two threads, it works quite efficiently. I am introducing the same java program to run for many processors with two threads each, to make it parallel, since I do not use MPI.