@@ -62,8 +62,15 @@ from [nghttp2.org][] or built from source.
62
62
63
63
### Benchmark analysis requirements
64
64
65
- To analyze the results, ` R ` should be installed. Use one of the available
66
- package managers or download it from < https://www.r-project.org/ > .
65
+ To analyze the results statistically, you can use either the
66
+ [ node-benchmark-compare] [ ] tool or the R script ` benchmark/compare.R ` .
67
+
68
+ [ node-benchmark-compare] [ ] is a Node.js script that can be installed with
69
+ ` npm install -g node-benchmark-compare ` .
70
+
71
+ To draw comparison plots when analyzing the results, ` R ` must be installed.
72
+ Use one of the available package managers or download it from
73
+ < https://www.r-project.org/ > .
67
74
68
75
The R packages ` ggplot2 ` and ` plyr ` are also used and can be installed using
69
76
the R REPL.
@@ -285,10 +292,11 @@ module, you can use the `--filter` option:*
285
292
--no-progress don't show benchmark progress indicator
286
293
```
287
294
288
- For analysing the benchmark results use the ` compare.R ` tool.
295
+ For analysing the benchmark results, use [ node-benchmark-compare] [ ] or the R
296
+ script ` benchmark/compare.R ` .
289
297
290
298
``` console
291
- $ cat compare-pr-5134.csv | Rscript benchmark/compare.R
299
+ $ node-benchmark-compare compare-pr-5134.csv # or cat compare-pr-5134.csv | Rscript benchmark/compare.R
292
300
293
301
confidence improvement accuracy (*) (**) (***)
294
302
string_decoder/string-decoder.js n=2500000 chunkLen=16 inLen=128 encoding='ascii' *** -3.76 % ±1.36% ±1.82% ±2.40%
@@ -315,17 +323,18 @@ consider at least two stars (`**`) as the threshold, in that case the risk
315
323
is 1%. If three stars (` *** ` ) is considered the risk is 0.1%. However this
316
324
may require more runs to obtain (can be set with ` --runs ` ).
317
325
318
- _ For the statistically minded, the R script performs an [ independent/unpaired
326
+ _ For the statistically minded, the script performs an [ independent/unpaired
319
327
2-group t-test] [ t-test ] , with the null hypothesis that the performance is the
320
328
same for both versions. The confidence field will show a star if the p-value
321
329
is less than ` 0.05 ` ._
322
330
323
- The ` compare.R ` tool can also produce a box plot by using the ` --plot filename `
324
- option. In this case there are 48 different benchmark combinations, and there
325
- may be a need to filter the csv file. This can be done while benchmarking
326
- using the ` --set ` parameter (e.g. ` --set encoding=ascii ` ) or by filtering
327
- results afterwards using tools such as ` sed ` or ` grep ` . In the ` sed ` case be
328
- sure to keep the first line since that contains the header information.
331
+ The ` compare.R ` tool can additionally produce a box plot by using the
332
+ ` --plot filename ` option. In this case there are 48 different benchmark
333
+ combinations, and there may be a need to filter the csv file. This can be done
334
+ while benchmarking using the ` --set ` parameter (e.g. ` --set encoding=ascii ` ) or
335
+ by filtering results afterwards using tools such as ` sed ` or ` grep ` . In the
336
+ ` sed ` case be sure to keep the first line since that contains the header
337
+ information.
329
338
330
339
``` console
331
340
$ cat compare-pr-5134.csv | sed ' 1p;/encoding=' " '" ascii" '" ' /!d' | Rscript benchmark/compare.R --plot compare-plot.png
@@ -560,5 +569,6 @@ Supported options keys are:
560
569
[ benchmark-ci ] : https://github.com/nodejs/benchmarking/blob/HEAD/docs/core_benchmarks.md
561
570
[ git-for-windows ] : https://git-scm.com/download/win
562
571
[ nghttp2.org ] : https://nghttp2.org
572
+ [ node-benchmark-compare ] : https://github.com/targos/node-benchmark-compare
563
573
[ t-test ] : https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes.2C_unequal_variances
564
574
[ wrk ] : https://github.com/wg/wrk
0 commit comments