Skip to content

Don't crash if write cannot be persisted #2938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 21, 2020
2 changes: 2 additions & 0 deletions packages/firestore/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# Unreleased
- [fixed] Firestore now rejects write operations if they cannot be persisted
in IndexedDB. Previously, these errors crashed the client.
- [fixed] Fixed a source of IndexedDB-related crashes for tabs that receive
multi-tab notifications while the file system is locked.

Expand Down
29 changes: 26 additions & 3 deletions packages/firestore/src/core/sync_engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@
*/

import { User } from '../auth/user';
import { ignoreIfPrimaryLeaseLoss, LocalStore } from '../local/local_store';
import {
ignoreIfPrimaryLeaseLoss,
LocalStore,
LocalWriteResult
} from '../local/local_store';
import { LocalViewChanges } from '../local/local_view_changes';
import { ReferenceSet } from '../local/reference_set';
import { TargetData, TargetPurpose } from '../local/target_data';
Expand All @@ -34,7 +38,7 @@ import { RemoteStore } from '../remote/remote_store';
import { RemoteSyncer } from '../remote/remote_syncer';
import { debugAssert, fail, hardAssert } from '../util/assert';
import { Code, FirestoreError } from '../util/error';
import { logDebug } from '../util/log';
import { logDebug, logError } from '../util/log';
import { primitiveComparator } from '../util/misc';
import { ObjectMap } from '../util/obj_map';
import { Deferred } from '../util/promise';
Expand Down Expand Up @@ -373,7 +377,26 @@ export class SyncEngine implements RemoteSyncer, SharedClientStateSyncer {
*/
async write(batch: Mutation[], userCallback: Deferred<void>): Promise<void> {
this.assertSubscribed('write()');
const result = await this.localStore.localWrite(batch);

let result: LocalWriteResult;
try {
result = await this.localStore.localWrite(batch);
} catch (e) {
// If we can't persist the mutation, we reject the user callback and don't
// send the mutation. The user can then retry the write. We currently do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This extra bit about not using enqueueRetryable isn't all that helpful. We're failing user writes because that's the developer experience we want users of the Firestore SDK to have. If we thought it was a better experience not to fail writes we'd make the changes required to make that happen.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it.

// not use `enqueueRetryable()` for writes since this would require us to
// move all view computation logic to `enqueueRetryable()`. Otherwise,
// this write should be surfaced in all subsequent operations.
logError(LOG_TAG, 'Dropping write that cannot be persisted: ' + e);
userCallback.reject(
new FirestoreError(
Code.UNKNOWN,
'Query allocation failed with environment error: ' + e
Copy link
Contributor

@wilhuff wilhuff Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Query allocation" seems like the wrong description of this error. Also, since this is catching everything, this isn't necessarily something to do with the environment. "Failed to enqueue a write in the local database"?

The catch here is also overbroad: this will catch internal assertion errors and the like as well. Taking an approach like the one I suggested in #2919 would make it possible to catch just errors related to IndexedDB.

Also, once we know the error is specifically about IndexedDB, I think the code can probably become DATA_LOSS. Or maybe UNAVAILABLE? DATA_LOSS is probably a bad choice because it is a permanent error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, "Query allocation" was the error message for my next PR.

I updated the PR with the changes in #2919 and only throw this error for IndexedDbTransactionErrors. I am a little uncertain whether we should just re-throw the error here or wrap it. For now, I decided to wrap it in a FirestoreError, as this makes it easier for our users to share error handling logic between the various backend errors and the new IndexedDB error.

)
);
return;
}

this.sharedClientState.addPendingMutation(result.batchId);
this.addMutationCallback(result.batchId, userCallback);
await this.emitNewSnapsAndNotifyLocalStore(result.changes);
Expand Down
62 changes: 59 additions & 3 deletions packages/firestore/test/unit/specs/recovery_spec.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@
*/

import { describeSpec, specTest } from './describe_spec';
import { client } from './spec_builder';
import { client, spec } from './spec_builder';
import { TimerId } from '../../../src/util/async_queue';
import { Query } from '../../../src/core/query';
import { path } from '../../util/helpers';
import { doc, path } from '../../util/helpers';

describeSpec(
'Persistence Recovery',
Expand All @@ -42,7 +42,7 @@ describeSpec(
// Client 1 has received the WebStorage notification that the write
// has been acknowledged, but failed to process the change. Hence,
// we did not get a user callback. We schedule the first retry and
// make sure that it also does not get processed until
// make sure that it also does not get processed until
// `recoverDatabase` is called.
.runTimer(TimerId.AsyncQueueRetry)
.recoverDatabase()
Expand Down Expand Up @@ -75,5 +75,61 @@ describeSpec(
.expectEvents(query, {});
}
);

specTest('Recovers when write cannot be persisted', [], () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these tests need a no-ios and no-android tag or some new kind of tag to indicate that failure recovery is required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: I pushed a commit to this PR to remove the durable-persistence tags as the new recovery tests also work with memory persistence. Most of my changes in my next PR were due to the formatting caused by this change.

return spec()
.userSets('collection/key1', { foo: 'a' })
.expectNumOutstandingWrites(1)
.failDatabase()
.userSets('collection/key2', { bar: 'b' })
.expectUserCallbacks({ rejected: ['collection/key2'] })
.recoverDatabase()
.expectNumOutstandingWrites(1)
.userSets('collection/key3', { baz: 'c' })
.expectNumOutstandingWrites(2)
.writeAcks('collection/key1', 1)
.writeAcks('collection/key3', 2)
.expectNumOutstandingWrites(0);
});

specTest('Does not surface non-persisted writes', [], () => {
const query = Query.atPath(path('collection'));
const doc1Local = doc(
'collection/key1',
0,
{ foo: 'a' },
{ hasLocalMutations: true }
);
const doc1 = doc('collection/key1', 1, { foo: 'a' });
const doc3Local = doc(
'collection/key3',
0,
{ foo: 'c' },
{ hasLocalMutations: true }
);
const doc3 = doc('collection/key3', 2, { foo: 'c' });
return spec()
.userListens(query)
.userSets('collection/key1', { foo: 'a' })
.expectEvents(query, {
added: [doc1Local],
fromCache: true,
hasPendingWrites: true
})
.failDatabase()
.userSets('collection/key2', { foo: 'b' })
.expectUserCallbacks({ rejected: ['collection/key2'] })
.recoverDatabase()
.userSets('collection/key3', { foo: 'c' })
.expectEvents(query, {
added: [doc3Local],
fromCache: true,
hasPendingWrites: true
})
.writeAcks('collection/key1', 1)
.writeAcks('collection/key3', 2)
.watchAcksFull(query, 2, doc1, doc3)
.expectEvents(query, { metadata: [doc1, doc3] });
});
}
);
4 changes: 3 additions & 1 deletion packages/firestore/test/unit/specs/spec_test_runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -687,7 +687,9 @@ abstract class TestRunner {
() => this.rejectedDocs.push(...documentKeys)
);

this.sharedWrites.push(mutations);
if (!this.persistence.injectFailures) {
this.sharedWrites.push(mutations);
}

return this.queue.enqueue(() => {
return this.syncEngine.write(mutations, syncEngineCallback);
Expand Down