Runtime.exec causes duplicate JVMs to hang indefinitely until it is killed (Solaris 10)

Everything,

We run the J2EE application on a WebLogic 9.2 MP2 server with the 64-bit JVM jrockit (27.3.1) on Solaris 10.

We invoke the use of runtime.exec to invoke the jfmerge executable to create PDFs.

We found that on Solaris, when runtime.exec is called, a duplicate JVM is temporarily created to start the jfmerge process. Although this is inefficient (our JVM is 5 GB, so the duplicated JVM shell is also 5 GB), the main problem is that with a heavy load on this functionality (PDF generation), the JVM never quits in our application sometimes.

When the JVM freezes, the servers create big problems (extreme application delays and terminated user sessions), since the entire duplicated JVM gets all its 5 GB of process size written to disk exchange.

We noted that the following freezing thread is correlated with the JVM freezing process until the process is manually killed:

"[STUCK] ExecuteThread: '17' for the queue: 'weblogic.kernel.Default (self-tuning)'" id = 3463 idx = 0x158 tid = 3460 prio = 1 live, native, daemon in JRockit / IO / FileNativeIO.readBytesPinned ( Ljava / IO / FileDescriptor; [BII) I (Native Method) in jrockit / io / FileNativeIO.readBytes (FileNativeIO.java:30) in java / io / FileInputStream.readBytes ([BII) I (FileInputStream.java) in java / io / FileInputStream.read (FileInputStream.java:194) in Java / languages ​​/ UNIXProcess $ DeferredCloseInputStream.read (UNIXProcess.java:227) in java / io / BufferedInputStream.fill (BufferedInputStream.java:218) in java / io / BufferedInput .read (BufferedInputStream.java:235) ^ - Lock lock: java / io / BufferedInputStream @ 0xfffffffec6510470 [thin lock] in g / v3 / general / formgeneration / SessionBean / FormsBean.getProcessStatus (FormsBean.java: 809) in g / v3 / general / formgeneration / SessionBean / FormsBean.createPDF (FormsBean.java:750) in g / v3 / general / formgeneration / SessionBean / FormsBean.getTemplateDetails (FormsBean.java:43) in g / v3 /general/formgeneration/SessionBean/FormsBean.generateSinglePDF(FormsBean.java:1371) in g / v3 / general / formgeneration / SessionBean / FormsBean.generatePDF (FormsBean.java:263) in g / v3 / Form / sessionBean .endorseDocument (FormsBean.java:2377) in g / v3 / generic / formgeneration / SessionBean / Forms_qaco28_EOImpl.endorseDocument (Forms_qaco28_EOImpl.java:214) in g / v3 / delegates / generic / FormsAndNatejoticeDateDoticeDateDotateDocumentsDotemDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDementesDotemDelegateDotemDelegateDotemDelegate g / v3 / action / general / EndorseDocumentAction.executeRequest (EndorseDocumentAction.java:68) in g / v3 / FWK / controller / struts / action / V3CommonDispatchAction.dispatchToExecuteMethod (V3CommonDispatchAction.javaPoint32) in g / v3 / FWK / controller / struts / action / V3CommonDispatchAction.executeBaseAction (V3CommonDispatchAction.javahaps36) in g / v3 / FWK / controller / struts / action / V3BaseDispatch java: 69) in org / Apache / struts / action / RequestProcessor.processActionPerform (RequestProcessor.java:484) in g / v3 / FWK / controller / struts / requestprocessor / V3TilesRequestProcessor.processActionPerform (V3TilesRequestProcessor.java indicated 84) in org / /spacers/action/RequestProcessor.process(RequestProcessor.java:274) in org / Apache / struts / action / ActionServlet.process (ActionServlet.java:1482) in org / Apache / struts / action / ActionServlet.doGet (ActionServlet.java : 507) in g / v3 / FWK / controller / struts / servlet / V3ControllerServl et.doGet (V3ControllerServlet.java:110) in javax / servlet / http / HttpServlet.service (HttpServlet.java:743) in javax / servlet / http / HttpServlet.service (HttpServlet.java:856) in WebLogic / servlet / internal / StubSecurityHelper $ ServletServiceActionrction .java: 227) in WebLogic / servlet / internal / StubSecurityHelper.invokeServlet (StubSecurityHelper.java:125) in WebLogic / servlet / internal / ServletStubImpl.execute (ServletStubImpl.java:283) in WebLogic / servlet / internal / internal ServletStubImpl.java:175) in WebLogic / servlet / internal / WebAppServletContext $ ServletInvocationAction.run (WebAppServletContext.java:3231) in WebLogic / security / ACL / internal / AuthenticatedSubject.doAs (AuthenticatedSubject.javaog132) /SecurityManager.runAs(SecurityManager.java:121) to WebLogic / servlet / inner / WebAppServletContext.securedExecute (WebAppServletContext.java:2002) to WebLogic / servlet / inner / WebAppServletContext.execute (WebAppServletContext.java:1908 : 1362) in weblogic / work / ExecuteThread.execute (ExecuteThread.java:209) in weblogic / work / ExecuteThread.run (ExecuteThread.java:181) in jrockit / vm / RNI.c2java (JJJJJ) V (native method) - end of track181) in jrockit / vm / RNI.c2java (JJJJJ) V (native method) - end of track181) in jrockit / vm / RNI.c2java (JJJJJ) V (native method) - end of track

:

1.) JVM, - jfmerge, .

2.) , , JVM .

+3
3

, , , Solaris .

, , 10 , "ls", 10 .

Solaris 10 , Linux , " ". .

http://developers.sun.com/solaris/articles/subprocess/subprocess.html

Unix : fork(), exec(). fork() , exec() .

(. Solaris " " , . Unix , . , " " (VM) .)

, fork/exec . , fork.

: fork() - , fork() . , fork() exec(), . , .

, , 64- 6 () VM , ls (1). fork(), , 6 . VM ( ), fork() ENOMEM. , ls (1) 6 , fork() .

, Sun . , dbx Sun RFE ( ): "4748951 dbx posix_spawn() , fork (2)".

RFE 4748951 , dbx script, cut (1) dbx. - , - dbx . , dbx fork/exec cut (1) fork().

Java Solaris (JVM) , Sun RFE: "5049299 posix_spawn, fork, S10, swap".


, 3 .

1.- Runtime.exec .

2. java- Runtime.exec.

3.- JNI C. , .

.

Java.

public class CallOS {
    static {
            System.loadLibrary("CallOS");
    }

    public native int exec(java.lang.String cmd);

    public static void main(String[] args) {
            int returnValue = 0;
            returnValue = new CallOS().exec("ls -la");
            System.out.println("- " + returnValue);
    }
}

C. javah -jni CallOS

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class CallOS */

#ifndef _Included_CallOS
#define _Included_CallOS
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     CallOS
 * Method:    exec
 * Signature: (Ljava/lang/String;)I
 */
JNIEXPORT jint JNICALL Java_CallOS_exec
  (JNIEnv *, jobject, jstring);

#ifdef __cplusplus
}
#endif
#endif

C-.

#include "CallOS.h"
#include <stdlib.h>

JNIEXPORT jint JNICALL Java_CallOS_exec
  (JNIEnv *env, jobject obj, jstring cmd)
{
   jint  retval;
   jbyte *str;

   str = (*env)->GetStringUTFChars(env, cmd, NULL);
   if(str == NULL) return NULL;

   retval = system(str);

   (*env)->ReleaseStringUTFChars(env, cmd, str);
   return retval;
};

, .

+6

stdout/stderr? , . . . , (- stdout/err, ).

, JVM fork/exec. Java (fork), (exec). , ? , , COW (copy-on-write), , , JVM , .

0

, unix fork . exec, . JVM , jfmerge.

, copy-on-write, , exec. , JVM- , , grandfild exec'ing jfmerge , . , JVM, . . , jfmerge , , , jfmerge .

What you could do is get another process to run jfmerge instead of your 5GB JVM. Write a standalone program that only runs jfmerge on demand and will link it to the main process through some form of interprocess communication. For this stand-alone jfmerge server, you don’t need so much memory to work, so the influence of forked child processes will not be so great.

0
source

Source: https://habr.com/ru/post/1713174/


All Articles