I am trying to use Spark in a concentrated HPC cluster that has infinite and interconnects. This cluster does not support IPoIB . I saw the Spakr-RDMA project from ohio state university in here . I cannot find anyone else working on this, or if spark apache will support IB in the future. The question is, is there another solution for getting a more updated version of the spark in HPC environments using only IB as a network?
M.Rez source share