Connection reset to use the REST service (scala / spray)

Question

I have a problem sending requests for a vacation service at the same time; Messages on the client (Apache JMeter) is "Connection reset" for some requests, depending on the number of requests, for example, I send 100 requests and the server response is 100% successful, but if I send 500 requests, 30% of the responses are errors.

java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.receiveResponseHeader(MeasuringConnectionManager.java:201) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135) at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261) at java.lang.Thread.run(Thread.java:745) 

I changed "application.conf", the contents are as follows:

 spray.can { server { server-header = spray-can/${spray.version} ssl-encryption = off pipelining-limit = 16 idle-timeout = 60 s request-timeout = 30 s timeout-timeout = 2 s timeout-handler = "" reaping-cycle = 250 ms stats-support = on remote-address-header = off raw-request-uri-header = off transparent-head-requests = on chunkless-streaming = off verbose-error-messages = on request-chunk-aggregation-limit = 1m response-header-size-hint = 512 bind-timeout = infinite unbind-timeout = 1s registration-timeout = 1s default-host-header = "" automatic-back-pressure-handling = on back-pressure { noack-rate = 10 reading-low-watermark = infinite } parsing = ${spray.can.parsing} } client { user-agent-header = spray-can/${spray.version} idle-timeout = 60 s request-timeout = 40 s reaping-cycle = 250 ms response-chunk-aggregation-limit = 1m chunkless-streaming = off request-header-size-hint = 256 max-encryption-chunk-size = 1m connecting-timeout = 30s proxy { http = default https = default } ssl-tracing = off parsing = ${spray.can.parsing} } host-connector { max-connections = 80 max-retries = 8 max-redirects = 0 pipelining = enabled idle-timeout = 30 s client = ${spray.can.client} } } 

JVM Settings:

 -Xms1024M -Xmx2048M -Xss1M -XX:MaxPermSize=1024m 

IMPORTANT: since the logic is bussines, it is necessary that the server supports transactions at the same time; 500 individual connections (transactions) in 5 seconds.

Answer 1

The timeout settings look great, and processing 500 requests per second is definitely not a problem.

Most likely, your requests have been processed for too long, i.e. more than request-timeout + timeout-timeout = 32 seconds. You need to check your architecture and see where and why it spends so much time. This would be quite unusual for regular web services, where most of the requests are filled in milliseconds. If you have heavy processing that you have to perform, it takes longer than you can respond with 202 Accepted and do the processing in the background. You can return a URI where the client can check the status of the request or use a callback to the client or any other mechanism to indicate that the request has been completed.

Remember that you do not need to block in the route itself, otherwise you would effectively block all other requests, and you could receive timeout errors. See this answer, for example: Use dispatcher with Spray HttpService . To implement non-blocking request processing, complete the following steps: How is the .HttpService spray request executed? .

Some ideas for troubleshooting: 1) determine how long it takes to process one request, and see how it scales - do you have problems with resources? 2) check that your network and client do not cause a timeout - their timeout should be higher than that of the server.

Answer 2

Thank you for your cooperation and your time, Alexey Izmailov .

Consulting Documentation for spraying , I have to set the spray.can.server timeout-timeout = 500s , with this configuration the server has 500 for accepting the response of the completed request. If it does not have a response, the server will send an error message to complete the request.

The purpose of this is to take a timeout on the server to receive a response, so you have finished waiting for the application (in my case, request-timeout = 30 s ). If this timeout-timeout ( timeout-timeout ) is completed and, of course, no response, the server finally completes the request.

 spray.can { server { ... timeout-timeout = 500 s ... } }