We recently updated the technical stack of the JAX-WS web service running on JRE 1.7.0_17 / Tomcat7.0.39 to JRE 1.8.0_66 / Tomcat 8.0.28. The web application runs on Windows Server 2012. The web service uses the Metro implementation for JAX-WS. Clients work on different versions of Windows using JRE 7 and the JAX-WS API built into the JRE. Webservice is used to upload files from client computers to a web service that stores them in a document management system. The implementation worked pretty flawlessly in Java 7 / Tomcat 7, but we had a problem with large payloads (2 MB or more) running under the Java 8 / Tomcat 8 server. Tracing the stack from the client:
12/02/2015 14:12:38.699 [AWT-EventQueue-0] ERROR DocumentImporterMainWindow$SwingAction.importDocument: Unexpected Problem trying to call the CustomerOrderDMService javax.xml.ws.WebServiceException: java.io.IOException: Error writing to server at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(Unknown Source) at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.createResponsePacket(Unknown Source) at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown Source) at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown Source) at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown Source) at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source) at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source) at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source) at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source) at com.sun.xml.internal.ws.client.Stub.process(Unknown Source) at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source) at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source) at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source) at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source) at com.sun.proxy.$Proxy30.importDocument(Unknown Source) at com.mycompany.documentimporter.DocumentImporterMainWindow$SwingAction.importDocument(DocumentImporterMainWindow.java:681) at com.mycompany.documentimporter.DocumentImporterMainWindow$SwingAction.actionPerformed(DocumentImporterMainWindow.java:612) at javax.swing.AbstractButton.fireActionPerformed(Unknown Source) at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source) at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source) at javax.swing.DefaultButtonModel.setPressed(Unknown Source) at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source) at java.awt.AWTEventMulticaster.mouseReleased(Unknown Source) at java.awt.Component.processMouseEvent(Unknown Source) at javax.swing.JComponent.processMouseEvent(Unknown Source) at java.awt.Component.processEvent(Unknown Source) at java.awt.Container.processEvent(Unknown Source) at java.awt.Component.dispatchEventImpl(Unknown Source) at java.awt.Container.dispatchEventImpl(Unknown Source) at java.awt.Component.dispatchEvent(Unknown Source) at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source) at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source) at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source) at java.awt.Container.dispatchEventImpl(Unknown Source) at java.awt.Window.dispatchEventImpl(Unknown Source) at java.awt.Component.dispatchEvent(Unknown Source) at java.awt.EventQueue.dispatchEventImpl(Unknown Source) at java.awt.EventQueue.access$500(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue$4.run(Unknown Source) at java.awt.EventQueue$4.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.run(Unknown Source) Caused by: java.io.IOException: Error writing to server at sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) at java.net.HttpURLConnection.getResponseCode(Unknown Source) ... 54 more
Unfortunately, nothing is logged on the server side in any of the tomcat logs. I spent days to find a solution to the problem without success. I tried to debug the problem using various methods, such as logging the client side of the SOAP request and response using -Dcom.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true (client) and -Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true (server) java, but when an error occurs, only the client-side request is executed:
---[HTTP request - http://localhost:8080/CustomerOrderDM/services/CustomerOrderDMService]--- Accept: text/xml, multipart/related Content-Type: text/xml; charset=utf-8 SOAPAction: "http://www.mycompany.com/CustomerOrderDMService/ImportDocument" User-Agent: JAX-WS RI 2.2.9-b130926.1035 svn-revision#5f6196f2b90e9460065a4c2f4e30e065b245e51e <?xml version="1.0" ?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns3:ImportDocumentRequestDTO xmlns:ns3="http://www.mycompany.com/CustomerOrderDMService/" xmlns:ns2="http://www.mycompany.com/CustomerOrderDMService"><ns2:QuoteNumber>A000049</ns2:QuoteNumber><ns2:SerialNumber>STOCK</ns2:SerialNumber><ns2:DocumentType>Email</ns2:DocumentType><ns2:Description></ns2:Description><ns2:DocumentContents> **some base64 encoded byte[] of the file contents being uploaded** Message has been truncated use com.sun.xml.internal.ws.transport.http.HttpAdapter.dumpTreshold property to increase the amount of printed part of the message --------------------
Since only the request is executed on the client side, we expect that the server does not fully process the request, and it falls into some sort of exception block, but without any protocols to the server log files, we have difficulty resolving the problem.
We tried to use proxies, such as Monitoring, built into Eclipse, but once again I see only a request from the client and no response from the server (when the client sends larger requests that fail, requests or response to requests of small requests both on the client and on the server). Other debugging suggestions would be greatly appreciated.
We also tried different combinations of Java and Tomcat:
- Tomcat 7 / Java 7 = Works
- Tomcat 7 / Java 8 = Works
- Tomcat 8 / Java 7 = not working
- Tomcat 8 / Java 8 = not working
This makes us think that the problem is related to Tomcat 8. Either something has changed in Tomcat 8, and now we need to set some new timeout / payload parameters, and Tomcat 8 has an error related to this specific problem.
We tried to set some Tomcat connector settings, such as maxPostSize="-1" , connectionTimeout="-1" , disableUploadTimeout="true" , connectionUploadTimeout="-1" , keepAliveTimeout="-1" but none of them worked and honestly felt like a shot in the dark, not knowing what was happening on the server side.
We tried updating the server part of Metro jars to the latest version (jaxws-ri-2.2.10), as well as starting the client using Java 8. Unfortunately, none of them worked. Any help would be greatly appreciated.