Skip to content

runtime: GC freeing goroutines memory but then taking it again #8832

Closed
@siritinga

Description

@siritinga
I'm seen this in tip version:
go version devel +9472d0e26c23 Sun Sep 28 08:27:05 2014 -0700 linux/amd64

Using this simple program:
http://play.golang.org/p/ircvbhPy3u

There is a related issue here:
https://golang.org/issue/8287

It will immediately use 250 MB of RES memory. After about 10 minutes, the scavenger will
free 205 MB, showing about 45MB of RES. Ignoring that there are still 45 MB used, the
real problem is that after that point, the RES memory will increase slowly until it uses
250MB of RES again!

This is what "ps u" shows every 10 seconds:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user     22626  2.5  3.1 261336 258096 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  1.6  3.1 261336 258164 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  1.0  3.1 261336 258164 pts/4   Sl+  08:34   0:00 ./tmp
[...] 9 minutes later... 
user     22626  0.0  3.2 261336 258792 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  3.2 261336 258792 pts/4   Sl+  08:34   0:00 ./tmp
[Here the scavenger frees the memory]
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.6 261336 48776 pts/4    Sl+  08:34   0:00 ./tmp
[from here, the memory is increased little by little]
user     22626  0.0  0.6 261336 50384 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.8 261336 64736 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  0.9 261336 79096 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  1.1 261336 93472 pts/4    Sl+  08:34   0:00 ./tmp
user     22626  0.0  1.3 261336 107840 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  1.5 261336 122216 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  1.6 261336 136600 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  1.8 261336 151000 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  2.0 261336 165336 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  2.2 261336 179744 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  2.4 261336 194080 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  2.5 261336 208512 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  2.7 261336 222848 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  2.9 261336 237184 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  3.0 261336 249728 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  3.1 261336 255520 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  3.1 261336 255520 pts/4   Sl+  08:34   0:00 ./tmp
user     22626  0.0  3.1 261336 255520 pts/4   Sl+  08:34   0:00 ./tmp
[remains here, it is not freed again and not detected by the GC]

Activity

ianlancetaylor

ianlancetaylor commented on Sep 29, 2014

@ianlancetaylor
Contributor

Comment 2:

Labels changed: added repo-main, release-go1.5.

modified the milestone: Go1.5 on Dec 16, 2014
ricktian1226

ricktian1226 commented on Feb 9, 2015

@ricktian1226

When will this issue be fixed? It happens on our production project too.

bradfitz

bradfitz commented on Feb 9, 2015

@bradfitz
Contributor

It's targeted for Go 1.5. Any updates will occur on this thread.

RLH

RLH commented on Feb 9, 2015

@RLH
Contributor

I'm trying to get a sense of the use case here. You have 100,000 goroutines, could you indicate
which of the following are true, say for 99,000 of the goroutines.

  1. Each go routine gets input from
    a. A channel
    b. A socket
    c. shared immutable memory
    d. shared mutable memory
    e. no input, only output.
  2. Each go routine provides output to
    a. A channel
    b. A socket
    c. shared memory that it mutates
    1. with pointers to freshly allocated data structures
    2. no freshly allocated memory is referenced after the goroutine ends

I ask this since the answers will help dictate the best solution. An optimal solution for a goroutine
that just does a 10 second noop might be to just elide the goroutine, which is probably not what you want.

randall77

randall77 commented on Feb 9, 2015

@randall77
Contributor

I don't see the behavior the OP sees at tip. The RSS drops to 46MB and stays there.
Probably something changed between September and now.

go version devel +f8176f8 Mon Feb 9 18:20:28 2015 +0000 linux/amd64
(one line per minute)
khr 13203 0.2 1.5 260856 255968 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.2 1.5 260856 255972 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.1 1.5 260856 255972 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.1 1.5 260856 255972 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.1 1.5 260856 255972 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.1 1.5 260856 255980 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 1.5 260856 255980 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 1.5 260856 255980 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832
khr 13203 0.0 0.2 260856 46144 pts/1 Sl+ 10:46 0:00 ./issue8832

siritinga

siritinga commented on Feb 9, 2015

@siritinga
Author

It is still there. Slower, but I see it going up to 56 MB before I stopped it, in about half an hour.

randall77

randall77 commented on Feb 10, 2015

@randall77
Contributor

So it is, if you wait long enough (for me, 16 minutes after the free) it starts growing. Very strange. The runtime doesn't allocate a single MSpan during that time, so I'm inclined to think this is a bug in linux/madvise. But I haven't been able to reproduce it outside the Go runtime. I'll keep looking.

32 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @bradfitz@davecheney@minux@RLH@dvyukov

        Issue actions

          runtime: GC freeing goroutines memory but then taking it again · Issue #8832 · golang/go