Skip to content

[Multi-Tab] Adding Schema Migration #485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Feb 9, 2018
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
eb44929
Adding Schema Migration
schmidt-sebastian Feb 2, 2018
07e44e7
Pseudocode for Schema Migration
schmidt-sebastian Feb 2, 2018
3e1053f
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 2, 2018
5e84782
IndexedDb Schema Migration
schmidt-sebastian Feb 5, 2018
fd50301
Lint cleanup
schmidt-sebastian Feb 6, 2018
be3b463
Removing unused import
schmidt-sebastian Feb 6, 2018
de237c5
Removing user ID from instance row
schmidt-sebastian Feb 6, 2018
53e56b5
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 6, 2018
7b3bedb
Review comments
schmidt-sebastian Feb 7, 2018
7b97c0a
Merge branch 'master' into multitab-schemamigration
schmidt-sebastian Feb 7, 2018
c176eff
Lint fixes
schmidt-sebastian Feb 7, 2018
9154aad
Review
schmidt-sebastian Feb 7, 2018
6b072ff
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 7, 2018
662365f
Fixing the tests
schmidt-sebastian Feb 7, 2018
dd21376
Closing the Database in the Schema tests
schmidt-sebastian Feb 8, 2018
5e54b3a
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 8, 2018
cb81df8
Changing test helper to close the DB
schmidt-sebastian Feb 8, 2018
7991970
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 8, 2018
2c6d6e1
Making v2 the default version
schmidt-sebastian Feb 9, 2018
f16bb4f
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 9, 2018
038c159
Addressing comment
schmidt-sebastian Feb 9, 2018
b70303c
[AUTOMATED]: Prettier Code Styling
schmidt-sebastian Feb 9, 2018
34ab25a
Renamed to ALL_STORES
schmidt-sebastian Feb 9, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions packages/firestore/src/local/indexeddb_migrations.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
/**
* Copyright 2018 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
DbDocumentMutation,
DbInstance,
DbMutationBatch,
DbMutationQueue,
DbOwner,
DbRemoteDocument,
DbTarget,
DbTargetDocument,
DbTargetGlobal
} from './indexeddb_schema';
import { assert, fail } from '../util/assert';

// TODO(mikelehen): Get rid of "as any" if/when TypeScript fixes their types.
// https://github.com/Microsoft/TypeScript/issues/14322
type KeyPath = any; // tslint:disable-line:no-any

function createCache(db: IDBDatabase): void {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, "cache" is non-obvious, so I would at least add a comment.

// Create IndexedDB stores for our query / remote document caches.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed this and split this up to match the names in Android: createQueryCache() and createRemoteDocumentCache()

const targetDocumentsStore = db.createObjectStore(DbTargetDocument.store, {
keyPath: DbTargetDocument.keyPath as KeyPath
});
targetDocumentsStore.createIndex(
DbTargetDocument.documentTargetsIndex,
DbTargetDocument.documentTargetsKeyPath,
{ unique: true }
);

const targetStore = db.createObjectStore(DbTarget.store, {
keyPath: DbTarget.keyPath
});

// NOTE: This is unique only because the TargetId is the suffix.
targetStore.createIndex(
DbTarget.queryTargetsIndexName,
DbTarget.queryTargetsKeyPath,
{ unique: true }
);
db.createObjectStore(DbRemoteDocument.store);
db.createObjectStore(DbTargetGlobal.store);
}

function dropCache(db: IDBDatabase): void {
db.deleteObjectStore(DbTargetDocument.store);
db.deleteObjectStore(DbTarget.keyPath);
db.deleteObjectStore(DbRemoteDocument.store);
db.deleteObjectStore(DbTargetGlobal.store);
}

function createOwnerStore(db: IDBDatabase): void {
db.createObjectStore(DbOwner.store);
}

function createInstanceStore(db: IDBDatabase): void {
db.createObjectStore(DbInstance.store, {
keyPath: DbInstance.keyPath as KeyPath
});
}

function createMutationQueue(db: IDBDatabase): void {
db.createObjectStore(DbMutationQueue.store, {
keyPath: DbMutationQueue.keyPath
});

db.createObjectStore(DbMutationBatch.store, {
keyPath: DbMutationBatch.keyPath as KeyPath
});

// NOTE: keys for these stores are specified explicitly rather than using a
// keyPath.
db.createObjectStore(DbDocumentMutation.store);
}

/**
* Runs any migrations needed to bring the given database up to the current
* schema version.
*/
export function createOrUpgradeDb(
db: IDBDatabase,
oldVersion: number,
newVersion: number
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having newVersion be a parameter is super confusing to me. Are you intending to support downgrades (I think we should try hard to avoid that!)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not at all planning on adding the ability to downgrade. I renamed the arguments to fromVersion and toVersion and added an assert that one must be lower than the other.

): void {
assert(
oldVersion >= 0 || oldVersion <= 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&& ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed (combined the asserts into one)

'Unexpected upgrade from version ' + oldVersion
);
assert(
newVersion >= 1 || newVersion <= 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&& ?

'Unexpected upgrade to version ' + newVersion
);

const createV1 = newVersion >= 1 && oldVersion <= 1;
const dropV1 = oldVersion >= 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be <= ? FWIW, I'm not sure these intermediate variables are helping here. I keep looking back and forth between the if-checks...

const createV2 = newVersion >= 2 && oldVersion <= 2;

if (dropV1) {
dropCache(db);
}

if (createV1) {
createOwnerStore(db);
createMutationQueue(db);
createCache(db);
}

if (createV2) {
createInstanceStore(db);
}
}
3 changes: 2 additions & 1 deletion packages/firestore/src/local/indexeddb_persistence.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,15 @@ import { IndexedDbMutationQueue } from './indexeddb_mutation_queue';
import { IndexedDbQueryCache } from './indexeddb_query_cache';
import { IndexedDbRemoteDocumentCache } from './indexeddb_remote_document_cache';
import { ALL_STORES, DbOwner, DbOwnerKey } from './indexeddb_schema';
import { createOrUpgradeDb, SCHEMA_VERSION } from './indexeddb_schema';
import { SCHEMA_VERSION } from './indexeddb_schema';
import { LocalSerializer } from './local_serializer';
import { MutationQueue } from './mutation_queue';
import { Persistence } from './persistence';
import { PersistencePromise } from './persistence_promise';
import { QueryCache } from './query_cache';
import { RemoteDocumentCache } from './remote_document_cache';
import { SimpleDb, SimpleDbTransaction } from './simple_db';
import { createOrUpgradeDb } from './indexeddb_migrations';

const LOG_TAG = 'IndexedDbPersistence';

Expand Down
77 changes: 28 additions & 49 deletions packages/firestore/src/local/indexeddb_schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,54 +21,10 @@ import { ResourcePath } from '../model/path';
import { assert } from '../util/assert';

import { encode, EncodedResourcePath } from './encoded_resource_path';
import { SnapshotVersion } from '../core/snapshot_version';

export const SCHEMA_VERSION = 1;

/** Performs database creation and (in the future) upgrades between versions. */
export function createOrUpgradeDb(db: IDBDatabase, oldVersion: number): void {
assert(oldVersion === 0, 'Unexpected upgrade from version ' + oldVersion);

db.createObjectStore(DbMutationQueue.store, {
keyPath: DbMutationQueue.keyPath
});

// TODO(mikelehen): Get rid of "as any" if/when TypeScript fixes their
// types. https://github.com/Microsoft/TypeScript/issues/14322
db.createObjectStore(
DbMutationBatch.store,
// tslint:disable-next-line:no-any
{ keyPath: DbMutationBatch.keyPath as any }
);

const targetDocumentsStore = db.createObjectStore(
DbTargetDocument.store,
// tslint:disable-next-line:no-any
{ keyPath: DbTargetDocument.keyPath as any }
);
targetDocumentsStore.createIndex(
DbTargetDocument.documentTargetsIndex,
DbTargetDocument.documentTargetsKeyPath,
{ unique: true }
);

const targetStore = db.createObjectStore(DbTarget.store, {
keyPath: DbTarget.keyPath
});
// NOTE: This is unique only because the TargetId is the suffix.
targetStore.createIndex(
DbTarget.queryTargetsIndexName,
DbTarget.queryTargetsKeyPath,
{ unique: true }
);

// NOTE: keys for these stores are specified explicitly rather than using a
// keyPath.
db.createObjectStore(DbDocumentMutation.store);
db.createObjectStore(DbRemoteDocument.store);
db.createObjectStore(DbOwner.store);
db.createObjectStore(DbTargetGlobal.store);
}

/**
* Wrapper class to store timestamps (seconds and nanos) in IndexedDb objects.
*/
Expand Down Expand Up @@ -131,7 +87,14 @@ export class DbMutationQueue {
* After sending this token, earlier tokens may not be used anymore so
* only a single stream token is retained.
*/
public lastStreamToken: string
public lastStreamToken: string,
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added newlines here and everywhere else where they were missing. I also tried to make this semi-valid JSDoc along with it (I still don't think it's valid as we would probably have to move the JSDoc comments to the function header).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that prettier stripped the newlines again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry. I should have known this was prettier...

Thanks for your efforts to make things JSDoc friendly, though 1) I don't consider this a goal [we're never going to generate docs from our internal implementation files], and 2) I don't think that needed to go in this PR... In this case it's probably fine, but it could be distracting if somebody was revisiting this PR later (to track down a bug or port to another platform or whatever).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if it is not for external documentation, it could show up in code completion when done properly - whereas right not it just shows up as a warning due do the /** JSDoc */ marks.

I agree that this is mostly a non-issue though and happily pressed some buttons to revert this.

* An identifier for the highest numbered batch that has been acknowledged
* by the server. All MutationBatches in this queue with batchIds less
* than or equal to this value are considered to have been acknowledged by
* the server.
*/
public highestPendingBatchId: number
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only exists so that we can leave acknowledged mutations in the queue for the sake of other tabs being able to read them, right? I think it's worth calling that out here. Also, are we going to port this to other platforms? If not, please add a "// PORTING NOTE:" comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the outdated comment. This is to reduce the number of scans we have to do in the mutation batch. Right now, to get the highest batch ID, we do an full scan at startup. I don't want to do this every time we add a mutation from a secondary tab.

) {}
}

Expand Down Expand Up @@ -390,8 +353,8 @@ export class DbTargetDocument {
/** Name of the IndexedDb object store. */
static store = 'targetDocuments';

/** Keys are automatically assigned via the targetId, path properties. */
static keyPath = ['targetId', 'path'];
/** Keys are automatically assigned via the targetId, snapshotVersion, and path properties. */
static keyPath = ['targetId', 'snapshotVersion', 'path'];

/** The index name for the reverse index. */
static documentTargetsIndex = 'documentTargetsIndex';
Expand All @@ -404,6 +367,7 @@ export class DbTargetDocument {
* The targetId identifying a target.
*/
public targetId: TargetId,
public snapshotVersion: number,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment? (especially call out if we are doing any cheating with fake snapshot versions or whatever...)

Also I think we want to do some design review (with Gil at least) before moving forward with this (unless it's living in a non-master branch or something).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I talked to Gil and Greg about this yesterday. Gil favors a separate ChangeLog, as primary key updates are pretty expensive. I updated the Design Doc with this yesterday and added a new table to store this changelog. The nice thing about this is that now all changes in the schema are additive and we don't need to drop the Query & Remote Document Cache anymore.

/**
* The path to the document, as encoded in the key.
*/
Expand Down Expand Up @@ -455,6 +419,20 @@ export class DbTargetGlobal {
) {}
}

export type DbInstanceKey = [string, string];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments?

Also, "Instance" feels too generic. Perhaps "ClientInstance" (and comments should explain the multi-tab scenario)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer used.


export class DbInstance {
static store = 'instances';

static keyPath = ['userId', 'instanceId'];

constructor(
public userId: string,
public instanceId: string,
public updateTimeMs: number
) {}
}

/**
* The list of all IndexedDB stored used by the SDK. This is used when creating
* transactions so that access across all stores is done atomically.
Expand All @@ -467,5 +445,6 @@ export const ALL_STORES = [
DbTarget.store,
DbOwner.store,
DbTargetGlobal.store,
DbTargetDocument.store
DbTargetDocument.store,
DbInstance.store
];
9 changes: 7 additions & 2 deletions packages/firestore/src/local/simple_db.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { debug } from '../util/log';
import { AnyDuringMigration } from '../util/misc';

import { PersistencePromise } from './persistence_promise';
import { SCHEMA_VERSION } from './indexeddb_schema';

const LOG_TAG = 'SimpleDb';

Expand All @@ -34,7 +35,11 @@ export class SimpleDb {
static openOrCreate(
name: string,
version: number,
runUpgrade: (db: IDBDatabase, oldVersion: number) => void
runUpgrade: (
db: IDBDatabase,
oldVersion: number,
newVersion: number
) => void
): Promise<SimpleDb> {
assert(
SimpleDb.isAvailable(),
Expand Down Expand Up @@ -70,7 +75,7 @@ export class SimpleDb {
// cheating and just passing the raw IndexedDB in, since
// createObjectStore(), etc. are synchronous.
const db = (event.target as IDBOpenDBRequest).result;
runUpgrade(db, event.oldVersion);
runUpgrade(db, event.oldVersion, SCHEMA_VERSION);
};
}).toPromise();
}
Expand Down