Skip to content

Commit 449147e

Browse files
targosjasnell
authored andcommitted
doc: document JavaScript tool for benchmark comparison
This makes it simpler for everyone to do the statistical analysis since it doesn't require to install R and its dependencies. Refs: https://github.com/targos/node-benchmark-compare PR-URL: #39835 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Zijian Liu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Michael Dawson <[email protected]>
1 parent d0a8986 commit 449147e

File tree

1 file changed

+21
-11
lines changed

1 file changed

+21
-11
lines changed

doc/guides/writing-and-running-benchmarks.md

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,15 @@ from [nghttp2.org][] or built from source.
6262

6363
### Benchmark analysis requirements
6464

65-
To analyze the results, `R` should be installed. Use one of the available
66-
package managers or download it from <https://www.r-project.org/>.
65+
To analyze the results statistically, you can use either the
66+
[node-benchmark-compare][] tool or the R script `benchmark/compare.R`.
67+
68+
[node-benchmark-compare][] is a Node.js script that can be installed with
69+
`npm install -g node-benchmark-compare`.
70+
71+
To draw comparison plots when analyzing the results, `R` must be installed.
72+
Use one of the available package managers or download it from
73+
<https://www.r-project.org/>.
6774

6875
The R packages `ggplot2` and `plyr` are also used and can be installed using
6976
the R REPL.
@@ -285,10 +292,11 @@ module, you can use the `--filter` option:*
285292
--no-progress don't show benchmark progress indicator
286293
```
287294

288-
For analysing the benchmark results use the `compare.R` tool.
295+
For analysing the benchmark results, use [node-benchmark-compare][] or the R
296+
script `benchmark/compare.R`.
289297

290298
```console
291-
$ cat compare-pr-5134.csv | Rscript benchmark/compare.R
299+
$ node-benchmark-compare compare-pr-5134.csv # or cat compare-pr-5134.csv | Rscript benchmark/compare.R
292300

293301
confidence improvement accuracy (*) (**) (***)
294302
string_decoder/string-decoder.js n=2500000 chunkLen=16 inLen=128 encoding='ascii' *** -3.76 % ±1.36% ±1.82% ±2.40%
@@ -315,17 +323,18 @@ consider at least two stars (`**`) as the threshold, in that case the risk
315323
is 1%. If three stars (`***`) is considered the risk is 0.1%. However this
316324
may require more runs to obtain (can be set with `--runs`).
317325

318-
_For the statistically minded, the R script performs an [independent/unpaired
326+
_For the statistically minded, the script performs an [independent/unpaired
319327
2-group t-test][t-test], with the null hypothesis that the performance is the
320328
same for both versions. The confidence field will show a star if the p-value
321329
is less than `0.05`._
322330

323-
The `compare.R` tool can also produce a box plot by using the `--plot filename`
324-
option. In this case there are 48 different benchmark combinations, and there
325-
may be a need to filter the csv file. This can be done while benchmarking
326-
using the `--set` parameter (e.g. `--set encoding=ascii`) or by filtering
327-
results afterwards using tools such as `sed` or `grep`. In the `sed` case be
328-
sure to keep the first line since that contains the header information.
331+
The `compare.R` tool can additionally produce a box plot by using the
332+
`--plot filename` option. In this case there are 48 different benchmark
333+
combinations, and there may be a need to filter the csv file. This can be done
334+
while benchmarking using the `--set` parameter (e.g. `--set encoding=ascii`) or
335+
by filtering results afterwards using tools such as `sed` or `grep`. In the
336+
`sed` case be sure to keep the first line since that contains the header
337+
information.
329338

330339
```console
331340
$ cat compare-pr-5134.csv | sed '1p;/encoding='"'"ascii"'"'/!d' | Rscript benchmark/compare.R --plot compare-plot.png
@@ -560,5 +569,6 @@ Supported options keys are:
560569
[benchmark-ci]: https://github.com/nodejs/benchmarking/blob/HEAD/docs/core_benchmarks.md
561570
[git-for-windows]: https://git-scm.com/download/win
562571
[nghttp2.org]: https://nghttp2.org
572+
[node-benchmark-compare]: https://github.com/targos/node-benchmark-compare
563573
[t-test]: https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes.2C_unequal_variances
564574
[wrk]: https://github.com/wg/wrk

0 commit comments

Comments
 (0)