All geek questions in one place

Is there a Java DRMAA library that works with Torque / PBS?

Does anyone know of a Java DRMAA-API implementation that is known to work with PBS / Torque cluster software?

At the heart of this: I would like to submit jobs to a new custom linux cluster from Java using the DRMAA compatible API. The cluster is managed by PBS / Torque . Torque includes the PBS DRMAA 1.0 library for Torque / PBS, which contains the DRMA-C binding and provides libdrmaa.so and .a in the files. I know that the Sun grid engine includes drmaa.jar, which provides the Java-DRMAA API. In fact, I decided to use SGE, but it was decided to try PBS first.

The theory underlying this decision was:
"DRMAA is a standard and therefore the Java API only requires drmaa-c bindings compatible with standards." However, I could not find such a "common DRMAA-C-java API" and now I assume that this assumption is incorrect and that the Java libraries are engine specific.

Edit: I was just experimenting with the drmaa.jar package from the solar array and trying to cross-use it with pbs libdrmaa.so. No wonder this failed (JNI unsatisfied link error).

Conclusion: This is not so! After some searching, I see only these few options:

  • Install GridWay ontop from Globus Toolkit. Established ontop by PBS, GridWay claims to provide DRMAA in Java. It looks too complicated for my installation.
  • Scrap DRMAA, introduce the PBS calling system command qsub, qstat, etc. from Java. Simple but not very good.

  • Implement the drmaa call itself. Way too complicated ...

  • Switch to the Grid Engine. GE, in my opinion, is superior to PBS in terms of language bindings.

I prefer option 2. or 4. Any recommendations?

+4
source share
3 answers

After a few more searches, it seems like I have to write something myself. There seems to be no optimal answer yet, but it can serve as a warning to those trying to do the same.

The best place to ask these questions is perhaps the Torque mailing list: www.clusterresources.com/resources/mailing-lists.php

First of all, the reason you cannot just use any DRMAA-Java library and use it with any DRMAA-C implementation: DRMAA describes a resource management interface, not how it is implemented. The provider can use the DRMAA-C implementation and use only these functions, but they do not need it. He can use everything that is in the engine. So, one important message: if you need certain language bindings, make sure that they are available for all necessary languages.

Regarding these options:

  • Using the GridWay / Globus Toolkit: http://www.gridway.org/doku.php?id=start Advantage: Gridway is a meta-scheduler that supports many resource management systems (SGE, PBS, ...). Perhaps the only way to get the DRMAA interface to work with PBS at the moment. Disadvantage: It looks like layer inflation and complexity. Do not have experience with this.

  • Using system commands, qsub, qstat, qdel. Advantage: fast hacking Minuses: dirty hacking, the need for parsers to output, may not notice that something went wrong, send messages from stdin / stdout / stderr, and not transfer.

  • Using JNI, it should be possible to create a binding for each c-function in drmaa.c Advantage: will ensure full implementation of drmaa (hopefully) Disadvanteges: includes compiled code, a lot of manual packaging of C-functions (maybe this can be automated)

  • Switch to another mesh mechanism. Perhaps we should have done this analysis earlier. However, we already have another Torque cluster, and we have experience with this. Workers two would create a more heterogeneous infrastructure.

  • Modifying an existing drmaa library from another vendor. I don’t know if this is possible ... We will also consider this.

+3
source

I also had this problem. This is a project for creating object-oriented C ++ and Java DRMAA bindings for PBS / Torque.

The disadvantage is that you should be able to load a library created from C ++ code, so it is not an implementation of "pure Java", which can only be distributed as a .jar file, expecting the end user to provide "libdrmaa.so "

For what it's worth, here it is: https://github.com/bryan-lunt/PBSJavaDRMAA/

+2
source

Have you ever decided what to do with it? Did you manage to associate Java DRMAA bindings with Torque / PBS? I am looking to get the DRMAA Java code running on the Torque / PBS system, and if you have already done the hard work, I would like to steal it.

However, if you haven’t done this, it should not be too bad to do some Java bindings, and I will do it if nobody does. A few years ago, I successfully modified Java DRMAA bindings for SGE to work with the new DRMAA implementation for Xgrid (now deprecated, but it may be revived soon).

I even wrote a short blog post about my experience (including a link to general instructions):

http://edbaskerville.com/2006/07/11/java-bindings-working/

+1
source

Source: https://habr.com/ru/post/1308043/

More articles:


All Articles