Akka timeout when using a client spray for multiple requests

Question

Using the spray 1.3.2 with Akka 2.3.6. (Acca is used only for spray).
I need to read huge files and make an http request for each line.
I read the files line by line using an iterator, and a request is made for each element. It starts successfully for some lines, but at some point it starts to fail:
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://default/user/IO-HTTP#-35162984]] after [60000 ms] .
At first I thought I was overloading the service, so I set "spray.can.host-connector.max-connections" to 1. It works much slower, but I have the same errors.

Here is the code:

 import spray.http.MediaTypes._ val EdnType = register( MediaType.custom( mainType = "application", subType = "edn", compressible = true, binary = false, fileExtensions = Seq("edn"))) val pipeline = ( addHeader("Accept", "application/json") ~> sendReceive ~> unmarshal[PipelineResponse]) def postData(data: String) = { val request = Post(pipelineUrl).withEntity(HttpEntity.apply(EdnType, data)) val responseFuture: Future[PipelineResponse] = pipeline(request) responseFuture } dataLines.map { d => val f = postData(d) f.onFailure { case e => println("Error - "+e)} // This is where the errors are display f.map { p => someMoreLogic(d, p) } } aggrigateResults(dataLines) 

I do it this way because I don't need all the data, just some clusters.

How can I solve this problem and keep it completely async?

Answer 1

Akka ask timeout is implemented through firstCompletedOf, so the timer starts when the request is initialized.

What you seem to be doing creates a future for each line (during the map) - so that all your calls are made almost simultaneously. Timeouts begin when the futures are initialized, but there are no performing threads left for all the generated actors to do their job. Therefore, it requests time.

Instead of processing all at once, I would suggest a more flexible approach - somewhat similar to using iterations or acca streams: Work Pulling Pattern . ( Github )

You provide an iterator that you already have as Epic . Introduce the actor Worker , who will perform the call and some logic. If you create N workers , then no more than two N lines will be processed (and the processing pipeline may include several steps). Thus, you can make sure that you do not overload the performers, and timeouts should not be executed.