Skip to content

Fix for b/74749605: Cancel pending backoff operations when closing streams. #564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 14, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions packages/firestore/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Unreleased
- [fixed] Fixed a regression in the Firebase JS release 4.11.0 that could
cause a crash if a user signs out while the client is offline, resulting in
an error of "Attempted to schedule multiple operations with timer id
listen_stream_connection_backoff".

# 0.3.5
- [changed] If the SDK's attempt to connect to the Cloud Firestore backend
neither succeeds nor fails within 10 seconds, the SDK will consider itself
"offline", causing get() calls to resolve with cached results, rather than
Expand Down
12 changes: 9 additions & 3 deletions packages/firestore/src/remote/backoff.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,8 @@ export class ExponentialBackoff {
* already, it will be canceled.
*/
backoffAndRun(op: () => Promise<void>): void {
if (this.timerPromise !== null) {
this.timerPromise.cancel();
}
// Cancel any pending backoff operation.
this.cancel();

// First schedule using the current base (which may be 0 and should be
// honored as such).
Expand Down Expand Up @@ -118,6 +117,13 @@ export class ExponentialBackoff {
}
}

cancel(): void {
if (this.timerPromise !== null) {
this.timerPromise.cancel();
this.timerPromise = null;
}
}

/** Returns a random value in the range [-currentBaseMs/2, currentBaseMs/2] */
private jitterDelayMs(): number {
return (Math.random() - 0.5) * this.currentBaseMs;
Expand Down
11 changes: 7 additions & 4 deletions packages/firestore/src/remote/persistent_stream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,9 @@ export abstract class PersistentStream<

this.cancelIdleCheck();

// Ensure we don't leave a pending backoff operation queued.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment can move above the idle cancel and cover both:

// Ensure we don't leave pending timer-based operations queued.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm. I had trouble creating a unified comment that I didn't feel was overly general or misleading, so instead I just commented both of them separately.

FWIW- the idle timer will only be active if the stream is currently connected when close() is called, and the backoff timer will only be active if the stream is not currently connected when close() is called.

this.backoff.cancel();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little weird that cancelling the idle and backoff timers look different. Is there any way to make these uniform? I suppose this happens because the backoff timer is held within the backoff object but the idle timer is held directly?

I think this may be non-actionable, but mentioning it anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

acknowledged.


if (finalState !== PersistentStreamState.Error) {
// If this is an intentional close ensure we don't delay our next connection attempt.
this.backoff.reset();
Expand Down Expand Up @@ -461,10 +464,10 @@ export abstract class PersistentStream<
this.state = PersistentStreamState.Backoff;

this.backoff.backoffAndRun(async () => {
if (this.state === PersistentStreamState.Stopped) {
// Stream can be stopped while waiting for backoff to complete.
return;
}
assert(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Man am I gun-shy about this assertion. I'm somewhat in favor of just leaving the old behavior in place to keep this consistent with the other callbacks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I am torn between being as strict as possible with our state transitions vs being defensive. I mostly changed it to an assert, since as-is it's basically dead code.

I put it back though and just tweaked the comment.

this.state === PersistentStreamState.Backoff,
'Backoff should have been canceled if we left the Backoff state.'
);

this.state = PersistentStreamState.Initial;
this.start(listener);
Expand Down
34 changes: 33 additions & 1 deletion packages/firestore/test/unit/specs/remote_store_spec.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import { expect } from 'chai';
import { Query } from '../../../src/core/query';
import { Code } from '../../../src/util/error';
import { doc, path } from '../../util/helpers';
import { doc, path, resumeTokenForSnapshot } from '../../util/helpers';

import { describeSpec, specTest } from './describe_spec';
import { spec } from './spec_builder';
Expand Down Expand Up @@ -83,4 +83,36 @@ describeSpec('Remote store:', [], () => {
.expectEvents(query, { added: [doc1] })
);
});

// TODO(b/72313632): This test is web-only because the Android / iOS spec
// tests exclude backoff entirely.
specTest(
'Handles user changes while offline (b/74749605).',
['no-android', 'no-ios'],
() => {
const query = Query.atPath(path('collection'));
const docA = doc('collection/a', 1000, { key: 'a' });
return (
spec()
.userListens(query)

// close the stream (this should trigger retry with backoff; but don't
// run it in an attempt to reproduce b/74749605).
.watchStreamCloses(Code.UNAVAILABLE, { runBackoffTimer: false })

// Because we didn't let the backoff timer run and restart the watch
// stream, there will be no active targets.
.expectActiveTargets()

// Change user (will shut down existing streams and start new ones).
.changeUser('abc')
// Our query should be sent to the new stream.
.expectActiveTargets({ query, resumeToken: '' })

// Close the (newly-created) stream as if it too failed (should trigger
// retry with backoff, potentially reproducing the crash in b/74749605).
.watchStreamCloses(Code.UNAVAILABLE)
);
}
);
});
12 changes: 10 additions & 2 deletions packages/firestore/test/unit/specs/spec_builder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -496,14 +496,22 @@ export class SpecBuilder {
return this;
}

watchStreamCloses(error: Code): SpecBuilder {
watchStreamCloses(
error: Code,
opts?: { runBackoffTimer: boolean }
): SpecBuilder {
if (!opts) {
opts = { runBackoffTimer: true };
}

this.nextStep();
this.currentStep = {
watchStreamClose: {
error: {
code: mapRpcCodeFromCode(error),
message: 'Simulated Backend Error'
}
},
runBackoffTimer: opts.runBackoffTimer
}
};
return this;
Expand Down
3 changes: 2 additions & 1 deletion packages/firestore/test/unit/specs/spec_test_runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -700,7 +700,7 @@ abstract class TestRunner {
)
);
// The watch stream should re-open if we have active listeners.
if (!this.queryListeners.isEmpty()) {
if (spec.runBackoffTimer && !this.queryListeners.isEmpty()) {
await this.queue.runDelayedOperationsEarly(
TimerId.ListenStreamConnectionBackoff
);
Expand Down Expand Up @@ -1167,6 +1167,7 @@ export type SpecSnapshotVersion = TestSnapshotVersion;

export type SpecWatchStreamClose = {
error: SpecError;
runBackoffTimer: boolean;
};

export type SpecWriteAck = {
Expand Down