2x more memory than gometalinter

[dvyukov]

go version go1.12.1 linux/amd64
golangci-lint is on 39f46be (current HEAD)
I am verifying github.com/google/syzkaller on 51fc038380062aec71b4e39af9227c6db0be4fd8.

Currently we use gomatalinter on Travis and it usually passes (with GOGC=50 fits into 10GB).
Running CGO_ENABLED=1 GOMAXPROCS=1 GOGC=50 gometalinter.v2 ./... locally consumes 8-9GB.
But running golangci-lint run ./... crashes with OOM on my laptop that has 14GB of free RAM. I can make it pass locally with GOGC=20 (most of time in GC, but still faster than gometalinter), it consumes all 14GB.
But I can't make it work on Travis. Even with GOGC=20 it still OOM killed:
https://travis-ci.org/google/syzkaller/jobs/523524512
https://travis-ci.org/google/syzkaller/jobs/523524511

I don't know how actionable this is. But maybe there is some kind of leak? Or some way to run linters sequentially and GC in-between? Or some other way to trade time for space?

[dvyukov]

I've noticed that disabling megacheck reduces memory consumption to ~8GB. But as far as I remember it produces lots of useful warnings and it's enabled in gometalinter.

[dvyukov]

I've also noticed that golangci-lint somewhat unexpectedly ignores my GOMAXPROCS=1 setting. I expected that would reduce concurrency to 1, but it didn't. Now I see -j flag for this. Perhaps it would be more reasonable to not mess with GOMAXPROCS if -j is not specified.

But it does not seem to help with the original problem. GOGC=50 -j=1 still crashed on my laptop after consuming 14GB. GOGC=20 -j=1 already consumed 13+GB and I had to abort it after 6 minutes of runtime (much slower than gometalinter).

[jirfag]

Hi,
I'm not sure that comparison with gometalinter is correct because of different staticcheck versions.

[dvyukov]

I can imagine there may be some differences due to somewhat different versions, but was there a change in staticcheck that adds 8 gigs and doubles overall memory consumption? I would assume there is something else in play here.

[daxmc99]

We are observing the same. Within our CI env with only govet, golint,gofmt, and typecheck we are noticing golangci-lint is much slower then gometalinter when only running with 4GB of memory.

[dvyukov]

staticcheck takes more memory, but even vet takes 10G:

syzkaller$ time golangci-lint run -j=8 --disable-all --enable=staticcheck ./...
177.16user 9.50system 0:48.95elapsed 381%CPU (0avgtext+0avgdata 13139696maxresident)k

syzkaller$ time golangci-lint run -j=8 --disable-all --enable=vet ./...
109.44user 9.09system 0:21.64elapsed 547%CPU (0avgtext+0avgdata 10019608maxresident)k

[dvyukov]

Wait, I noticed that golangci-lint does not actually seem to skip the dirs specified in skip-dirs. I have some huge autogenerated files there, and a bunch of such packages are listed in skip-dirs. But if I delete some of the largest files, memory consumption of checking time drops dramatically:

syzkaller$ GOGC=100 time golangci-lint run -j=8 --disable-all --enable=vet ./...
25.71user 2.36system 0:04.62elapsed 606%CPU (0avgtext+0avgdata 2840928maxresident)k

[otto-md]

I ran into this issue now as well. When I tried to upgrade to 1.16 our CircleCI lint job started to get killed due to OOM (I think they might have a 4G limit).

Our project does not yet use go modules.

I use this config:

run:
  skip-dirs:
    - dir1
    - node_modules
    - dir2

linters:
  enable:
    - govet
#    - goimports
#    - gofmt
#    - typecheck
    - staticcheck
    - gosimple
  disable-all: true

linters-settings:
  goimports:
    local-prefixes: ourdomain.com

issues:
  exclude:
    - composite literal uses unkeyed fields

Using 1.16 on macOS:

$ /usr/bin/time -l golangci-lint run --config=config/golangci.yml ./pkg/...
       33.28 real        73.34 user         6.89 sys
4376711168  maximum resident set size

Using 1.15:

$ /usr/bin/time -l golangci-lint run --config=config/golangci.yml ./pkg/...
WARN [runner/megacheck] Bad megacheck checker name "lint" 
       31.43 real        56.12 user         5.56 sys
3629387776  maximum resident set size

Using 1.12.5:

$ /usr/bin/time -l golangci-lint run --config=config/golangci.yml ./pkg/...
       33.18 real        68.97 user         5.33 sys
4361379840  maximum resident set size

Using 1.10:

$ /usr/bin/time -l golangci-lint run --config=config/golangci.yml ./pkg/...
WARN [runner/megacheck.{gosimple,staticcheck}] Can't run megacheck because of compilation errors in packages [runtime/internal/atomic internal/cpu syscall runtime net runtime/internal/sys os internal/syscall/unix]: ../../../../../../usr/local/go/src/runtime/internal/atomic/atomic_wasm.go:14: Load redeclared in this block and 368 more errors: run `golangci-lint run --no-config --disable-all -E typecheck` to see all errors 
       12.88 real        26.63 user         9.75 sys
1312501760  maximum resident set size

I realise that this might be an apple to oranges comparison, but I've seen with other tools that when go module support was introduced, the memory usage increased a lot.

I also tested with a bunch of different packages:

1.16       1.15       1.14
2141724672 1617063936 1731031040
4378001408 3134328832 3000246272
4243808256 3521437696 3591892992
1978617856 1848393728 1959194624
1978662912 1476882432 1536888832
2656006144 1948475392 1937104896
3937079296 2824671232 3017359360
1069805568 1014620160 962297856
3529412608 3735236608 3677290496
3202179072 2258554880 2172289024

In almost all cases memory usage increased going from 1.15 to 1.16.

[alexec]

I also have this issue and it actually crashes Docker on my machine.

[gracenoah]

I saw an OOM as well. gqlgen creates huge auto-generated files and we have a few of those. That might be part of the problem. These files account for 217k lines of code in our repo across 6 files. I hope they are not being processed unnecessarily. It's OOMing for us sometimes on a 16 GB RAM machine.

[dvyukov]

Adding

build-tags:
    - codeanalysis

to the config file and // +build !codeanalysis to the generated files does NOT help.

[dvyukov]

Wait, I put build-tags: into wrong location in the config file. It actually does seem to help.

[dvyukov]

So we now finally switched to golangci-lint! Woohoo!
With some impressive cpu/mem savings. Part of it comes from golangci-lint vs gometalinter. But the major part comes from ignoring auto-generated files for real now (it gave like 7x less time and 20x less mem).

From my perspective it can now be rebranded to updating docs so that people searching for "autogenerated" or "memory consumption" could find the answer.
As far as I understand not loading autogenerated files is not always possible as will break type-checking. But in our case it was possible.

[daxmc99]

@dvyukov Could you share the right location for build-tags in the config file? 😄

[dvyukov]

@daxmc99 you will find it here:
https://github.com/google/syzkaller/blob/master/.golangci.yml

[jirfag]

Hi!

1. I've reduced memory usage a lot in the master branch.
2. There are a lot of duplicating issues, let's continue in Lower or control memory usage #337
3. It makes no sense to compare with gometalinter because our linters set and versions differ a lot.