provide benchmarks for plotly.js #1511
Conversation
- use karma-benchmark framework which uses benchmark.js under the hood to run the benchmarks - use karma-benchmark-json-reporter to output benchmark result to JSON file checked in to dist/ (good idea?) - use karma-benchmarkjs-reporter to output log to stdout during runs - add a few suites (more to come)
|
Thoughts on consistency: CI will probably be exceptionally difficult to trust since you really have no idea what else is going on on the physical machine or if you're on the same architecture. A quick check shows a dedicated machine on AWS is $85/mo or so. In lieu of that, one approach is to normalize the benchmarks to find out if a particular test is slower than expected, but this would probably still lead to weird/spurious quirks that might throw the whole thing in to question. One other approach is for each run to check out the previous commit, run the benchmarks, check out master, run the benchmarks again, and compare. Again, the only downside is a lack of coherent history to get the long running trends vs. short regressions. I think the most reliable would be a VM on a physical machine in the office or something so that at least there aren't so many unknowns, both known and unknown. If the machine dies and you can't continue on the same machine, at least you can still check out old tags and rebuild the history. |
|
Plotly will sponsor a cloud machine. Large cloud providers guarantee reliable tenancy performance. I don't think a dedicated machine is necessary. We can ensure the machine has enough resources so that it remains underutilized even during the most grueling benchmark suites. Putting the machine in the cloud will allow a larger circle of developers to have access to the machine. If you are interested let me know and I'll help set that up. |
|
Ping 🚨 @dfcreative and I are currently doing a lot of performance-related work, so it would be nice to finally have some sort of centralised way of doing benchmarks. While setting a cloud machine (like proposed in #1511 (comment)) would be ideal, I'm thinking this PR here might a nice first step that requires minimal effort. I'm proposing the following workflow:
cc @nicolaskruchten who probably has a few ideas about this ... |
|
@etpinard this looks like a great framework, and with all the work you and @dy are doing it's definitely important to get something like this going. My big question, from a practical standpoint, is how we can run matching benchmarks - including new benchmarks just added - against older versions of code, and quickly compare performance? Ideally I don't want to compare to an older run, even one that I did on my own computer - I want to be able to check out some random older version of the code, maybe even 4 or 5 different versions, run the bench suite against each of them in turn, and then see an output table showing the % change vs the first version (or at least the first version that worked, what happens in this system when there's an error?) I guess we could do this by somehow copying the |
|
Thanks for 🖊️ your concerns @alexcjohnson. To make benchmark comparisons easier, I'm thinking of making For example,
|
|
That all sounds great. One possible modification for flexibility would be:
Or I suppose this could all be done with a single command that takes a list of base tags/branches, though that feels dangerous, as it's a whole lot to run in sequence and expect all to go right... but also, with a separate |
|
In order to ensure consistency across runs, disabling Some more tips for accurate benchmarking: https://easyperf.net/blog/2019/08/02/Perf-measurement-environment-on-Linux |
|
This pull request has been sitting for a while, so I would like to close it as part of our effort to tidy up our public repositories. I've assigned it to myself to keep track of it; I'll wait until 2024-06-17 for someone to say it's still relevant and they'll to take it on, and otherwise I will close it then. Thanks - @gvwilson |
|
apologies for letting this fall behind without merging - we'll create a separate ticket for benchmarking the current version of plotly.js. thanks - @gvwilson |
Introducing a
bench/directory for centralised benchmark suites.Goals:
Technology used:
karma-benchmarkkarma framework which usesBenchmark.jsunder the hood in a similar way to howkarma-jasmineexposes the jasmine test specs in a karma-run browser window.Benchmark.jsasync benchmarking (see explanation)karma-benchmark-json-reporterto log the results in JSON file. Suggestions to improve it are welcomed!karma-benchmarkjs-reporterto log 💄 results to the stdoutkarma, the benchmarks should run properly on CircleCI using Chrome and the--ignore-gpu-blacklistpost CircleCI with Ubuntu 14.04, 2x parallelism and WebGL jasmine tests #1455How does this work?
npm run benchbench/suites/andnpm run benchwill automatically bundle and run them (similar to ourtest/jasmine/workflow)dist/benchmarks.jsonwhich I'm thinking we should check in togitRemaining questions:
npm run benchbe run on CI? Or maybe on an old laptop as @rreusser suggested?@rreusser @alexcjohnson @monfera @n-riesco @bpostlethwaite