Error using --new-from-rev in Google Build Cloud

[valbuena]

Thank you for creating the issue!

• Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).

Please include the following information:

Version of golangci-lint

Already have image: golangci/golangci-lint:v1.26-alpine
golangci-lint has version 1.26.0 built from 6bd10d0 on 2020-05-01T15:33:57Z

Config file

# This file was inspired by the golangci-lint one:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.yml
linters-settings:
  govet:
    check-shadowing: true
  golint:
    min-confidence: 0
  gocyclo:
    min-complexity: 10
  maligned:
    suggest-new: true
  goconst:
    min-len: 2
    min-occurrences: 2
  misspell:
    locale: US
  lll:
    line-length: 160
  gofmt:
    simplify: false
  gocritic:
    enabled-tags:
      - diagnostic
      - experimental
      - opinionated
      - performance
      - style
    disabled-checks:
      - wrapperFunc
      - dupImport # https://github.com/go-critic/go-critic/issues/845
      - ifElseChain
      - octalLiteral
      - hugeParam
  funlen:
    lines: 100
    statements: 50

linters:
  # please, do not use `enable-all`: it's deprecated and will be removed soon.
  # inverted configuration with `enable-all` and `disable` is not scalable during updates of golangci-lint
  disable-all: true
  enable:
    - bodyclose
    - deadcode
    - depguard
    - dogsled
    - dupl
    - errcheck
    - gochecknoglobals
    - gochecknoinits
    - goconst
    - gocritic
    - gocyclo
    - gofmt
    - goimports
    - golint
    - gomnd
    - goprintffuncname
    - gosec
    - gosimple
    - govet
    - ineffassign
    - interfacer
    - lll
    - misspell
    - nakedret
    - rowserrcheck
    - scopelint
    - staticcheck
    - structcheck
    - stylecheck
    - typecheck
    - unconvert
    - unparam
    - unused
    - varcheck
    - whitespace
    - gocognit
    - maligned
    - prealloc

    # don't enable:
    # - funlen
    # - godox


issues:
  # Show only new issues: if there are unstaged changes or untracked files,
  # only those changes are analyzed, else only changes in HEAD~ are analyzed.
  # It's a super-useful option for integration of golangci-lint into existing
  # large codebase. It's not practical to fix all existing issues at the moment
  # of integration: much better don't allow issues in new code.
  # Default is false.
  new: false

Verbose output of running

Status: Downloaded newer image for golangci/golangci-lint:v1.26-alpine
docker.io/golangci/golangci-lint:v1.26-alpine
level=info msg="[config_reader] Used config file .golangci.yml"
level=info msg="[lintersdb] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer lll maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]"
level=info msg="[lintersdb] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer lll maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]"
level=info msg="[loader] Go packages loading at mode 575 (types_sizes|compiled_files|deps|exports_file|files|imports|name) took 59.137298537s"
level=info msg="[runner/enabledLinters] Active 38 linters: [bodyclose deadcode depguard dogsled dupl errcheck gochecknoglobals gochecknoinits gocognit goconst gocritic gocyclo gofmt goimports golint gomnd goprintffuncname gosec gosimple govet ineffassign interfacer lll maligned misspell nakedret prealloc rowserrcheck scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]"
level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 76.533117ms"
level=info msg="[runner/goanalysis_metalinter/goanalysis] analyzers took 6m36.98194291s with top 10 stages: buildssa: 1m35.86414484s, dupl: 18.380915243s, goimports: 13.446273376s, gocritic: 12.763288952s, gosec: 11.04550755s, unconvert: 10.856014233s, golint: 8.033825161s, interfacer: 7.647963528s, gofmt: 7.062299754s, unparam: 7.055950855s"
level=info msg="[runner/unused/goanalysis] analyzers took 19.777256292s with top 10 stages: buildssa: 17.437252531s, U1000: 2.340003761s"
level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"HEAD~\" \"\": exit status 128"

In my local environment this works fine:

golangci-lint run --config .golangci.yml -v --new-from-rev=HEAD~

but in Google Build Cloud I got this error:

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"HEAD~\" \"\": exit status 128"

[jirfag]

Hi,
it looks like no git is installed.
We need to improve the error message.

[valbuena]

Thank you for your answer @jirfag
I thought this so I am using this image: golangci-lint:v1.26-alpine that I thought it has git installed:
https://github.com/golangci/golangci-lint/blob/master/build/Dockerfile.alpine.

[jirfag]

it should have a git installed, you are right.
Is your repository public?

[valbuena]

No sorry, it is no public.

I tried several configurations:

• Using golanci.yml:

  issues:
     new: false
     new-from-rev: HEAD~

• Using command line in google build cloud:

#
# Go Linter
#
- name: "golangci/golangci-lint:v1.26-alpine"
  args: ["golangci-lint","run","--config",".golangci.yml", "-v", "--timeout=20m", "--new-from-rev=HEAD~"]]

• Running with docker

docker run --rm -v .:/app -w /app golangci/golangci-lint:v1.26.0 golangci-lint run -v --timeout=20m --new-from-rev=HEAD~

But I got the same result, it works in my local environment but it doesn't work in google cloud build.

• The error is in this line: https://github.com/golangci/revgrep/blob/276a5c0a103935ee65af49afc254a65335bf1fcf/revgrep.go#L365
  So I suppose if git doesn't work fine I had got an error before, here: https://github.com/golangci/revgrep/blob/276a5c0a103935ee65af49afc254a65335bf1fcf/revgrep.go#L337

• And I found this guy had the same problem with TravisCI: --new/--new-from-rev flags does not work #948 (comment).

Thank you in advance for any help :-)

[valbuena]

@jirfag I got a solucion at the end:

I am using this: https://github.com/golangci/golangci-lint-action

But it works fine for me with this configuration:

name: golangci-lint
on:
  pull_request:
jobs:
  golangci:
    name: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v1
        with:
          # Required:  the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
          version: v1.26

          # Optional: golangci-lint command line arguments.
          # args: --timeout=20m

          # Optional: show only new issues if it's a pull request. The default value is `false`.
          only-new-issues: true

Because If I use for example push option and args to compare with master or HEAD~: I got again the same error with all the issues in the application:

name: golangci-lint
on:
  push:
jobs:
  golangci:
    name: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v1
        with:
          # Required:  the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
          version: v1.26

          # Optional: golangci-lint command line arguments.
          args: --timeout=20m --new-from-rev=master

          # Optional: show only new issues if it's a pull request. The default value is `false`.
          # only-new-issues: true

Error:

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"master\" \"\": exit status 128"

[krhubert]

I'm having the same issue, unfortunately, the repo is not public.

 level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"a32ce..382b\" \"\": exit status 128"

.github/workflows/test.yml

name: test
on: push
env:
  go_version: 1.16.0
jobs:
  backend-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-go@v2
        with:
          go-version: ${{ env.go_version }}

      - uses: actions/checkout@v2

      - name: Run linters
        uses: golangci/golangci-lint-action@v2
        with:
          version: v1.31.0
          skip-go-installation: true
          working-directory: backend/src
          only-new-issues: true

.golangci.yml

output:
  format: github-actions

issues:
  new-from-rev: a32ce..382b

linters:
  disable-all: true
  enable:
    - deadcode
    - errcheck
    - gosimple
    - govet
    - ineffassign
    - staticcheck
    - structcheck
    - typecheck
    - unused
    - varcheck

[ldez]

Some changes has been done in v1.37.0 about that topic, could you try with this version?

[krhubert]

@ldez I've tried v1.37.0 and I've got the same result :/

[yash97]

facing similar issue any update?

[ldez]

Just to keep the issue in the right way.

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff "HEAD~" "": exit status 128"

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff "a32ce..382b" "": exit status 128"

These previous error messages can mean:

• the command git is missing.
• the commits are missing, probably because the git tree is incomplete.

---

If the command git is missing, you have to install it.

---

If the commits are missing, you have to configure your CI to get those commits.

With GitHub Actions

You have to define fetch-depth, ex:

    steps:

      - name: Check out code
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches

With Travis

You have to define git.depth, ex:

git:
  depth: false

https://docs.travis-ci.com/user/customizing-the-build#git-clone-depth

For the other CI or platform, you have to check the documentation of your CI or platform about git fetching configuration.

---

To solve this issue, the error message must be improved if possible to get more context.

[ymh199478]

I modified the configuration of git shallow clone in gitlab to solve this problem.

@ldez is correct. I used to limit the depth of git cloning in order to speed up the clone speed, but this resulted in golangci-lint not being able to obtain enough information (the commits are missing).

Now I will allow the depth of git clone to be 15，However, there may be potential risks in doing so. If you change more than 15 commit, there will be problems.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OlegSchwann]

Since it's on the front page of Google, it's a good place to share my experience with
https://golangci-lint.run/usage/faq/#how-to-integrate-golangci-lint-into-large-project-with-thousands-of-issues .

Sometimes the command

golangci-lint run --new-from-rev=master

crashes inside the Gitlab CI with the error

level=warning msg="[runner] Can't process result by diff processor: can't prepare diff by revgrep: could not read git repo: error executing git diff \"master\" \"\": exit status 128"

The reason is that GitLab only checkout commits without downloading branches. In order to use the master, you need to download it explicitly

git fetch origin master && git branch master remotes/origin/master

[taylorlee]

👆 Oleg's comment was very helpful - builds in TeamCity also crash with the same error. Even using teamcity.git.fetchAllHeads=true, I still needed the git branch master remotes/origin/master

[wijayaerick]

For Google Cloud Build specifically, you'd need to fetch unshallow. This is because cloudbuild by default performs shallow clone.

  - id: gitFetch
    waitFor: ['-']
    name: 'gcr.io/cloud-builders/git'
    entrypoint: git
    args: ['fetch', '--unshallow', '--quiet']

  - id: lint
    waitFor: ['gitFetch']
    name: 'golangci/golangci-lint:${_GOLANGCI_VERSION}'
    entrypoint: golangci-lint
    args: ['run', '--new-from-rev=master']

For GitLab runners, I think configuring the git strategy would work although I cannot verify it as I don't use GitLab https://docs.gitlab.com/ee/ci/pipelines/settings.html#choose-the-default-git-strategy.