Skip to content

perf triage classification of PR 89611 was "probably changed"; why not definite? #1109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pnkfelix opened this issue Nov 24, 2021 · 0 comments

Comments

@pnkfelix
Copy link
Member

Looking at the performance report linked here:

Benchmark & Profile Scenario % Change Significance Factor ?
encoding opt full -0.80%? 2.21x
encoding opt incr-full -0.72%? 2.36x
coercions check incr-patched: println -0.64% 4.82x
helloworld debug incr-full -0.53%? 2.07x
helloworld debug incr-unchanged -0.52% 4.79x
coercions check incr-unchanged -0.51% 7.92x
helloworld debug full -0.45%? 1.83x
coercions opt incr-unchanged -0.43% 5.67x
ripgrep opt incr-unchanged -0.42% 2.73x
cranelift-codegen opt incr-unchanged 0.38%? 1.81x
coercions debug incr-unchanged -0.36% 3.87x
issue-58319 debug incr-unchanged -0.34% 2.09x
token-stream-stress debug full -0.31% 1.74x
token-stream-stress debug incr-unchanged -0.31% 2.25x
coercions doc full -0.25% 2.89x
unicode_normalization check incr-unchanged -0.24% 3.79x
coercions check full -0.24% 3.79x
issue-58319 debug incr-full -0.24% 2.00x
tuple-stress debug incr-unchanged -0.24% 1.94x
helloworld opt incr-patched: println -0.21% 2.55x
helloworld opt incr-unchanged -0.20% 2.19x
coercions check incr-full -0.20% 5.63x

why isn't this classified as a definite improvement in the performance triage script?

Yes, a lot of them have question marks on them (though I'm not sure the performance triage script includes the question marks in its weighting at this point, based on what I saw elsewhere), but even if you remove the question mark cases, you're still left with e.g.

Benchmark & Profile Scenario % Change Significance Factor ?
coercions check incr-patched: println -0.64% 4.82x
helloworld debug incr-unchanged -0.52% 4.79x
coercions check incr-unchanged -0.51% 7.92x

which seems like it should have been enough for being classified as a performance win, not a "probably" ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant