Go to: memory issues

Question

I need your wisdom.

I have a huge daemon written in Go. Some time ago, the user reported that there might be a memory leak in the code.

I started investigating the problem. When the initial verification of the code did not lead me to any information about the nature of this leak, I tried to focus on how my process works.

My idea was simple: if I was not able to delete links to certain objects, my heap should constantly grow. I wrote the following procedure to track heap:

func PrintHeap() { ticker := time.NewTicker(time.Second * 5) for { <-ticker.C st := &runtime.MemStats{} runtime.ReadMemStats(st) // From Golang docs: HeapObjects increases as objects are allocated // and decreases as the heap is swept and unreachable objects are // freed. fmt.Println("Heap allocs:", st.Mallocs, "Heap frees:", st.Frees, "Heap objects:", st.HeapObjects) } } 

This procedure prints some heap information every 5 seconds, including the number of currently allocated objects.

Now a few words about what the demon does. It processes lines from some UDP input. Each line contains some information about a specific HTTP request and is parsed into a typical Go structure. This structure has several numeric and string fields, including one for the query path. Then many things happen to this structure, but here these things are irrelevant here.

Now I set the input speed to 1,500 lines per second, each line is quite short (you can read it like this: with the standard query path, / ).

After starting the application, I saw that the heap size stabilizes at some point in time:

 Heap allocs: 180301314 Heap frees: 175991675 Heap objects: 4309639 Heap allocs: 180417372 Heap frees: 176071946 Heap objects: 4345426 Heap allocs: 180526254 Heap frees: 176216276 Heap objects: 4309978 Heap allocs: 182406470 Heap frees: 177496675 Heap objects: 4909795 Heap allocs: 183190214 Heap frees: 178248365 Heap objects: 4941849 Heap allocs: 183302680 Heap frees: 178958823 Heap objects: 4343857 Heap allocs: 183412388 Heap frees: 179101276 Heap objects: 4311112 Heap allocs: 183528654 Heap frees: 179181897 Heap objects: 4346757 Heap allocs: 183638282 Heap frees: 179327221 Heap objects: 4311061 Heap allocs: 185609758 Heap frees: 181330408 Heap objects: 4279350 

When this state was reached, memory consumption stopped growing.

Now I changed my input so that each line was longer than 2 thousand characters (with a huge trace of the request /AAAAA... ) and where strange things started.

The heap size has grown sharply, but after a while it became stable:

 Heap allocs: 18353000513 Heap frees: 18335783660 Heap objects: 17216853 Heap allocs: 18353108590 Heap frees: 18335797883 Heap objects: 17310707 Heap allocs: 18355134995 Heap frees: 18336081878 Heap objects: 19053117 Heap allocs: 18356826170 Heap frees: 18336182205 Heap objects: 20643965 Heap allocs: 18366029630 Heap frees: 18336925394 Heap objects: 29104236 Heap allocs: 18366122614 Heap frees: 18336937295 Heap objects: 29185319 Heap allocs: 18367840866 Heap frees: 18337205638 Heap objects: 30635228 Heap allocs: 18368909002 Heap frees: 18337309215 Heap objects: 31599787 Heap allocs: 18369628204 Heap frees: 18337362196 Heap objects: 32266008 Heap allocs: 18373482440 Heap frees: 18358282964 Heap objects: 15199476 Heap allocs: 18374488754 Heap frees: 18358330954 Heap objects: 16157800 

But memory consumption grew and never stopped. My question is: any ideas on what is going on?

I was thinking about memory fragmentation due to the large number of huge objects, but actually I really don't know what to think.

Answer 1

You can try go memory profiling tools.

First you need to change your program to provide a memory profile. There are several ways.

• You can use the net/http/pprof see https://golang.org/pkg/net/http/pprof/ if you are ok with publishing this endpoint.
• You can use the runtime/pprof and make your program dump a memory profile to a known place in response to certain events, such as receiving a signal or something like that.

After that, you can analyze the memory profile using go tool pprof which you can either call as go tool pprof <path/to/executable> <file> if you decide to reset the memory profile to a file or as go tool pprof <path/to/executable> http://<host>:<port>/debug/pprof/heap if you used net/http/pprof and use top5 to get the 5 best functions that allocated most of your memory. You can use the list command for specific functions to see which lines allocate how much memory.

Starting with this, you should be able to reason about the increase in memory you are observing.

You can also read about it at https://blog.golang.org/profiling-go-programs , which also describes how to profile your processor usage. Just find the word memprofile to jump to the relevant parts.