Add benchmark spec tests #1048
Add benchmark spec tests #1048
Conversation
schmidt-sebastian
force-pushed
the
mrschmidt-perf
branch
2 times, most recently
from
f5c5580 to
67943d9
Compare
schmidt-sebastian
force-pushed
the
mrschmidt-perf
branch
from
67943d9 to
12ce39e
Compare
| @@ -127,6 +132,8 @@ export function specTest( | |||
| runner = it.only; | |||
| } else if (!WEB_SPEC_TEST_FILTER(tags)) { | |||
| runner = it.skip; | |||
| } else if (tags.indexOf(BENCHMARK_TAG) >= 0 && !RUN_BENCHMARK_TESTS) { | |||
There was a problem hiding this comment.
I think this logic should just go in WEB_SPEC_TEST_FILTER... or else we should get rid of WEB_SPEC_TEST_FILTER and inline the NO_WEB_TAG check here. There's no reason to have both NO_WEB_TAG and WEB_SPEC_TEST_FILTER defined as constants if they're used for the same thing.
There was a problem hiding this comment.
Since we special case a bunch of tags in here already, it probably makes more sense to remove web filter. Consider it gone.
| if (tags.indexOf(BENCHMARK_TAG) >= 0) { | ||
| // tslint:disable-next-line:no-console | ||
| console.log(`Runtime: ${end - start} ms.`); | ||
| } |
There was a problem hiding this comment.
FWIW- If you wanted to get fancy you could play with integrating console.time() / console.profile() to get higher-precision timings and to automatically collect profiles. Though they're non-standard. So in theory it could break our tests on some browsers.
There was a problem hiding this comment.
I am thinking that high profile timers run in a headless Chrome instance might give us more of an illusion of precision than actual accuracy.
| import { describeSpec, specTest } from './describe_spec'; | ||
| import { spec } from './spec_builder'; | ||
|
|
||
| const STEP_COUNT = 10; |
| const STEP_COUNT = 10; | ||
|
|
||
| describeSpec('Performance Tests:', ['benchmark'], () => { | ||
| specTest('Insert a new document', [], () => { |
There was a problem hiding this comment.
Could you include ${STEP_COUNT} in the description of tests to accurately reflect what's being benchmarked? With the results in your PR description I was disappointed how slow everything was, but knowing that your doing 10 reps makes me feel a fair amount better.
Similarly, the results (e.g. when copy/pasted into an email or PR description) would be more meaningful if the descriptions were more explicit / precise as to what's being measured. E.g.: Write document and handle server acknowledgement [repeat ${STEP_COUNT} times]
There was a problem hiding this comment.
I added it to the test name via the description in describe.
| return steps; | ||
| }); | ||
|
|
||
| specTest('Insert a new document and wait for snapshot', [], () => { |
There was a problem hiding this comment.
More explicit: Start a listen, write a document, ack write, handle watch snapshot, unlisten [repeat ${STEP_COUNT} times]
There was a problem hiding this comment.
Updated this and most other test titles.
|
|
||
| const docs = []; | ||
|
|
||
| for (let i = 0; i < cachedDocumentCount; ++i) { |
There was a problem hiding this comment.
It is a little tempting to somehow introduce the equivalent of for-loops into the spec test schema to avoid generating overly large / verbose JSON files with stuff like this and the STEP_COUNT stuff... But that's probably a rabbit hole best avoided for now. :-)
There was a problem hiding this comment.
go/changestats
:)
This is a set of benchmarks that I am going to use to compare multi-tab to non-multi tab performance.
Wihtout Multi-Tab:
'Running spec: (Persistence) Insert a new document'
'Runtime: 220 ms.'
'Running spec: (Memory) Insert a new document'
'Runtime: 13 ms.'
'Running spec: (Persistence) Insert a new document and wait for snapshot'
'Runtime: 352 ms.'
'Running spec: (Memory) Insert a new document and wait for snapshot'
'Runtime: 50 ms.'
'Running spec: (Persistence) Watch has cached mutations'
'Runtime: 1444 ms.'
'Running spec: (Memory) Watch has cached mutations'
'Runtime: 188 ms.'
'Running spec: (Persistence) Update a single document'
'Runtime: 187 ms.'
'Running spec: (Memory) Update a single document'
'Runtime: 6 ms.'
'Running spec: (Persistence) Update a single document and wait for snapshot'
'Runtime: 282 ms.'
'Running spec: (Memory) Update a single document and wait for snapshot'
'Runtime: 23 ms.'
'Running spec: (Persistence) Watch sends 100 documents'
'Runtime: 961 ms.'
'Running spec: (Memory) Watch sends 100 documents'
'Runtime: 309 ms.'
'Running spec: (Persistence) Watch has cached results'
'Runtime: 1584 ms.'
'Running spec: (Memory) Watch has cached results'
'Runtime: 681 ms.'
With Multi Tab:
'Running spec: (Persistence) Insert a new document'
'Runtime: 234 ms.' +14
'Running spec: (Memory) Insert a new document'
'Runtime: 12 ms.' - 1
'Running spec: (Persistence) Insert a new document and wait for snapshot'
'Runtime: 482 ms.' +130
'Running spec: (Memory) Insert a new document and wait for snapshot'
'Runtime: 72 ms.' +22
'Running spec: (Persistence) Watch has cached mutations'
'Runtime: 1681 ms.' +233
'Running spec: (Memory) Watch has cached mutations'
'Runtime: 221 ms.' +33
'Running spec: (Persistence) Update a single document'
'Runtime: 242 ms.' +55
'Running spec: (Memory) Update a single document'
'Runtime: 7 ms.' +1
'Running spec: (Persistence) Update a single document and wait for snapshot'
'Runtime: 384 ms.' +104
'Running spec: (Memory) Update a single document and wait for snapshot'
'Runtime: 30 ms.' +7
'Running spec: (Persistence) Watch sends 100 documents'
'Runtime: 979 ms.' +18
'Running spec: (Memory) Watch sends 100 documents'
'Runtime: 354 ms.' +45
'Running spec: (Persistence) Watch has cached results'
'Runtime: 1928 ms.' +344
'Running spec: (Memory) Watch has cached results'
'Runtime: 782 ms.' +101