Erlang Binary Leak?

Question

We have an erlang / elixir application (on 18 / erts 7.3.1) that processes large json files.

Here's a typical workflow:

• The listener receives the token from rabbitmq and sends it to gen_server.

• gen_server places the token in the ETS table with a future time (current + n secs). The schedule task in gen_server will display the expired tokens from the ETS and start several short-lived processes with these tokens.

• These short processes load 30-50 thousand useful json data from elasticsearch (using hackney) and process it, then load the results back into elasticsearch, after which the process dies immediately. We track these processes and confirm that they are dying. We process 5-10 requests per second.

Problem: we see an ever-growing binary space and within 48 hours it grows to a couple of GB (visible through an observer and debugging prints). GC leadership is also not affected.

We already added “recon” and ran recon: bin_leak, however this frees up only a few KB and does not affect the ever-growing binary space.

stack: Erlang 18 / erts 7.3.1, elixir 1.3.4, hackney 1.4.4, poison 2.2.0, timex 3.1.13, etc., none of these applications contain memory.

Has anyone encountered a similar problem in the past? I would be grateful for any decisions.

Update 9/15/2017:

Erlang 19/ERTS 8.3, hackney poison libs , . GenServer, , spawn/receive send_after. handle_info ets, - "" , . , {: noreply, state}. ( ), . "" . . : recon.bin_leak (N) : erlang.garbage_collect() .

11: 40: 19.896 [warn] binary 1: 3544.1328125

11: 40: 24.897 [warn] binary 1: 3541.9609375

11: 40: 29.901 [warn] binary 1: 3541.9765625

11: 40: 34.903 [warn] binary 1: 3546.2109375

--- ---

12: 00: 47.307 [warn] binary 1: 7517.515625

--- ---

12: 20: 38.033 [warn] binary 1:15002.1328125

Scala/Akka, 30- . .

Answer 1

, memory_leak , Graylog gen_udp.

defp compress(data) do
    zip = :zlib.open()
    :zlib.deflateInit(zip)
    output = :zlib.deflate(zip, data, :finish)
    :zlib.deflateEnd(zip)
    :zlib.close(zip)  #<--- was missing, hence the slow memory leak.
    output
end

term_to_binary (data, [: compression]) .

Thanks for all the material and comments. Really appreciate!