--new-from-rev may silently fail on large git histories

[schallert]

Thank you for creating the issue!

• Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
• Yes, I've searched similar issues on GitHub and didn't find any. NOTE: the last comment of --new/--new-from-rev flags does not work #948 is similar but that issue seems more related to file name parsing overall.
• Yes, I've included all information below (version, config, etc).

Please include the following information:

Version of golangci-lint

$ golangci-lint --version
golangci-lint has version 1.23.6 built from b9eef79 on 2020-02-10T17:51:52Z

Config file

$ cat .golangci.yml
# None: using command line for minimal test case
$ ~/Downloads/golangci-lint-1.23.6-darwin-amd64/golangci-lint run --disable-all --enable golint

Go environment

$ go version && go env
go version go1.13.8 darwin/amd64
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/REDACTED/Library/Caches/go-build"
GOENV="/Users/REDACTED/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/REDACTED/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13.8/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13.8/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=REDACTED=/tmp/go-build -gno-record-gcc-switches -fno-common"

Verbose output of running

# Large revision history: silent failure
$ ~/Downloads/golangci-lint-1.23.6-darwin-amd64/golangci-lint run --disable-all --enable golint --new-from-rev HEAD~500 -v
INFO [config_reader] Config search paths: [ REDACTED ]
INFO [lintersdb] Active 1 linters: [golint]
INFO [lintersdb] Active 1 linters: [golint]
...
INFO [runner] Issues before processing: N, after processing: 0
INFO [runner] Processors filtering stat (out/in): ... diff: 0/20, ...
...
INFO File cache stats: 0 entries of total size 0B
...

# Shorter revision history: 3 errors
$ ~/Downloads/golangci-lint-1.23.6-darwin-amd64/golangci-lint run --disable-all --enable golint --new-from-rev HEAD~50 -v
INFO [config_reader] Config search paths: [ REDACTED ]
INFO [lintersdb] Active 1 linters: [golint]
INFO [lintersdb] Active 1 linters: [golint]
...
INFO [runner] Issues before processing: N, after processing: 3
INFO [runner] Processors filtering stat (out/in): diff: 3/20, ...
...

3 REDACTED GOLINT ERRORS

INFO File cache stats: 3 entries of total size 64.4KiB
...

When running golangci-lint with --new-from-rev on some large git histories, revgrep silently fails with:

reading standard input: bufio.Scanner: token too long

In the above test cases, --new-from-rev HEAD~500 reports 0 errors but --new-from-rev HEAD~50 reports 3 errors.

When using a forked version of revgrep with golangci/revgrep#4 pulled in, --new-from-rev HEAD~500 reports all expected errors:

$ golint ./... |& revgrep HEAD~500
reading standard input: bufio.Scanner: token too long

$ golint ./... |& revgrep-fork HEAD~50
mycode.go:8:2: a blank import should be only in a main or test package, or have a comment justifying it

I unfortunately don't have a public git repo to reproduce with, but if necessary can try to create a mock one with a lot of large commits in history.

[schallert]

I narrowed down the test case a bit further. It's not necessarily related to the git history being long, but rather a long line in the diff of a single file. So for example, the following command should produce a file that will trigger this behavior if added to a repo:

for I in {0..20000}; do echo -n "A" >> badfile.txt; done

[ernado]

Seems like it is bug in revgrep and can be fixed by increasing bufio.Scanner buffer.

I have no write access to https://github.com/golangci/revgrep so can't fix it.

cc: @jirfag

[andrewmains12]

Hey @jirfag thanks for the help on this! Any chance of reviewing @schallert 's PR fixing this, or pointers to someone else who could review? golangci/revgrep#4

Thanks!

[robskillington]

+1 can we get golangci/revgrep#4 looked at? This has been causing significant pain for a few big repos such as https://github.com/m3db/m3.

[ernado]

I've merged golangci/revgrep#4