Skip to content

dev: add targets to benchmark a linter #4761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 28, 2024

Conversation

ldez
Copy link
Member

@ldez ldez commented May 27, 2024

Adds 2 targets to benchmark a linter.

LINTER=gosec VERSION=v1.59.0 make bench_local
LINTER=gosec VERSION_OLD=v1.58.2 VERSION_NEW=v1.59.0 make bench_version

It uses Hyperfine.

It's not really possible to automate that because it will take too much time to run, but at least we will be able to benchmark manually a PR.

Demo
$ LINTER=misspell VERSION_OLD=v1.57.2 VERSION_NEW=v1.59.0 make bench_version
golangci/golangci-lint info checking GitHub for tag 'v1.57.2'
golangci/golangci-lint info found version: 1.57.2 for v1.57.2/linux/amd64
golangci/golangci-lint info installed ./temp-v1.57.2/golangci-lint
golangci/golangci-lint info checking GitHub for tag 'v1.59.0'
golangci/golangci-lint info found version: 1.59.0 for v1.59.0/linux/amd64
golangci/golangci-lint info installed ./temp-v1.59.0/golangci-lint
Benchmark 1: ./golangci-lint-v1.57.2 run --issues-exit-code 0 --print-issued-lines=false --enable-only misspell
  Time (mean ± σ):     630.3 ms ±  26.9 ms    [User: 2246.2 ms, System: 1483.3 ms]
  Range (min … max):   591.8 ms … 668.5 ms    10 runs
 
Benchmark 2: ./golangci-lint-v1.59.0 run --issues-exit-code 0 --print-issued-lines=false --enable-only misspell
  Time (mean ± σ):     379.5 ms ±  24.7 ms    [User: 1055.4 ms, System: 479.7 ms]
  Range (min … max):   349.4 ms … 430.7 ms    10 runs
 
Summary
  ./golangci-lint-v1.59.0 run --issues-exit-code 0 --print-issued-lines=false --enable-only misspell ran
    1.66 ± 0.13 times faster than ./golangci-lint-v1.57.2 run --issues-exit-code 0 --print-issued-lines=false --enable-only misspell
Clean binaries

Related to #1152

@ldez ldez added enhancement New feature or improvement area: ci PR that update CI labels May 27, 2024
@ldez ldez added this to the next milestone May 27, 2024
@ldez ldez force-pushed the feat/benchmark-script branch 2 times, most recently from b01ae84 to 74751bd Compare May 27, 2024 22:07
@ldez ldez force-pushed the feat/benchmark-script branch from 15e1017 to 34a0e3e Compare May 28, 2024 13:29
@ldez ldez added area: benchmark and removed area: ci PR that update CI labels May 28, 2024
ldez and others added 2 commits May 28, 2024 16:29
@ldez ldez merged commit aea73fb into golangci:master May 28, 2024
13 checks passed
@ldez ldez deleted the feat/benchmark-script branch May 28, 2024 17:39
@ldez ldez modified the milestones: next, v1.59 Jun 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: benchmark enhancement New feature or improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants