Skip to content

Consider unifying messagesg in kotlinx-benchmark executors with JMH's ones #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
CommanderTvis opened this issue Jan 27, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@CommanderTvis
Copy link

CommanderTvis commented Jan 27, 2022

  1. Messages printed by JMH are much more meaningful.
  2. Making messages formats similar improves the library user's experience.
Consider similar benchmark reports. The JMH one:
… space.kscience.kmath.benchmarks.ExpressionsInterpretersBenchmark.mstExpression

Warm-up 1: 2.464 ops/s
Warm-up 2: 2.530 ops/s
Iteration 1: 2.575 ops/s
Iteration 2: 2.561 ops/s
Iteration 3: 2.525 ops/s
Iteration 4: 2.509 ops/s
Iteration 5: 2.517 ops/s
Iteration 6: 2.493 ops/s
Iteration 7: 2.506 ops/s
Iteration 8: 2.477 ops/s
Iteration 9: 2.548 ops/s
Iteration 10: 2.549 ops/s

  Success: 2.526 ±(99.9%) 0.047 ops/s [Average]
  (min, avg, max) = (2.477, 2.526, 2.575), stdev = 0.031
  CI (99.9%): [2.479, 2.573] (assumes normal distribution)

The one by JsExecutor:

… space.kscience.kmath.benchmarks.ExpressionsInterpretersBenchmark.mstExpression
Iteration #0: 0.0569060 ops/sec
Iteration #1: 0.0558908 ops/sec
Iteration #2: 0.0571284 ops/sec
Iteration #3: 0.0568593 ops/sec
Iteration #4: 0.0571386 ops/sec
Iteration #5: 0.0551405 ops/sec
Iteration #6: 0.0564571 ops/sec
Iteration #7: 0.0558157 ops/sec
Iteration #8: 0.0585252 ops/sec
Iteration #9: 0.0602421 ops/sec
  Success:   ~ 0.0570104 ops/sec ±1.6%

It is obvious that JMH's format contains more information: warmup and measurement iterations are separated, and there are basic statistics of results.

@CommanderTvis CommanderTvis changed the title Consider unifying messagesg in kotlinx-benchmark executors with JMH's ones? Consider unifying messagesg in kotlinx-benchmark executors with JMH's ones Jan 27, 2022
@qurbonzoda qurbonzoda added the enhancement New feature or request label Jan 30, 2022
OndrejSliva pushed a commit to OndrejSliva/kotlinx-benchmark that referenced this issue Jan 10, 2024
…ependabot/fetch-metadata-1.3.4

build(deps): bump dependabot/fetch-metadata from 1.3.3 to 1.3.4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants