Skip to content

Implement BundleCache for IDB and memory. #3170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Jun 14, 2020
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions packages/firestore/src/core/bundle.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
/**
* @license
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { Query } from './query';
import { SnapshotVersion } from './snapshot_version';
import { DbBundle, DbNamedQuery } from '../local/indexeddb_schema';
import * as bundleProto from '../protos/firestore_bundle_proto';
import { LocalSerializer } from '../local/local_serializer';

/**
* Represents a Firestore bundle saved by the SDK in its local storage.
*/
export interface Bundle {
readonly id: string;
readonly version: number;
// When the saved bundle is built from the server SDKs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you point out in the documentation that this is to SnapshotVersion.MIN if the above is not true? In its current form, the comment raises the question why the type is not nullable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what do you mean here, sorry!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like:

Set to the snapshot version of the bundle if created by the Server SDKs. Otherwise set to SnapshotVersion.MIN.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

readonly createTime: SnapshotVersion;
}

/**
* Represents a Query saved by the SDK in its local storage.
*/
export interface NamedQuery {
readonly name: string;
readonly query: Query;
// When the results for this query are read to the saved bundle.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/** The time at which the results for this query were read. */

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

readonly readTime: SnapshotVersion;
}

/** Encodes a DbBundle to a Bundle. */
export function fromDbBundle(
serializer: LocalSerializer,
dbBundle: DbBundle
): Bundle {
return {
id: dbBundle.bundleId,
createTime: serializer.fromDbTimestamp(dbBundle.createTime),
version: dbBundle.version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is our versioning strategy? I wonder if we should reject any bundle with a version greater than 1. If so, please file a bug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b/158845787

};
}

/** Encodes a BundleMetadata to a DbBundle. */
export function toDbBundle(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have an actual opinion, but I am a bit surprised that you moved this function out of the serializer file. I think it makes sense that even the free-standing functions are next to the main serializer, but I can also see why you moved them to. Just wanted to know whether you considered this?

No need to move them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved them back. I think putting them here also makes sense, but I don't feel strongly anyways.

serializer: LocalSerializer,
metadata: bundleProto.BundleMetadata
): DbBundle {
return {
bundleId: metadata.id!,
createTime: serializer.toDbTimestamp(
serializer.remoteSerializer.fromVersion(metadata.createTime!)
),
version: metadata.version!
};
}

/** Encodes a DbNamedQuery to a NamedQuery. */
export function fromDbNamedQuery(
serializer: LocalSerializer,
dbNamedQuery: DbNamedQuery
): NamedQuery {
return {
name: dbNamedQuery.name,
query: fromBundledQuery(serializer, dbNamedQuery.bundledQuery),
readTime: serializer.fromDbTimestamp(dbNamedQuery.readTime)
};
}

/** Encodes a NamedQuery from bundle proto to a DbNamedQuery. */
export function toDbNamedQuery(
serializer: LocalSerializer,
query: bundleProto.NamedQuery
): DbNamedQuery {
return {
name: query.name!,
readTime: serializer.toDbTimestamp(
serializer.remoteSerializer.fromVersion(query.readTime!)
),
bundledQuery: query.bundledQuery!
};
}

/**
* Encodes a `BundledQuery` from bundle proto to a Query object.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/from bundle/from a bundle/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

*
* This reconstructs the original query used to build the bundle being loaded,
* including features exists only in SDKs (for example: limit-to-last).
*/
export function fromBundledQuery(
serializer: LocalSerializer,
bundledQuery: bundleProto.BundledQuery
): Query {
const query = serializer.remoteSerializer.convertQueryTargetToQuery({
parent: bundledQuery.parent!,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you file a bug to validate the bundle contents? As part of this we need to validate that the parent matches the Firestore project that we are loading the SDK into.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b/158846981

structuredQuery: bundledQuery.structuredQuery!
});
if (bundledQuery.limitType === 'LAST') {
return query.withLimitToLast(query.limit);
}
return query;
}

/** Encodes a NamedQuery proto object to a NamedQuery model object. */
export function fromProtoNamedQuery(
serializer: LocalSerializer,
namedQuery: bundleProto.NamedQuery
): NamedQuery {
return {
name: namedQuery.name!,
query: fromBundledQuery(serializer, namedQuery.bundledQuery!),
readTime: serializer.remoteSerializer.fromVersion(namedQuery.readTime!)
};
}

/** Encodes a BundleMetadata proto object to a Bundle model object. */
export function fromBundleMetadata(
serializer: LocalSerializer,
metadata: bundleProto.BundleMetadata
): Bundle {
return {
id: metadata.id!,
version: metadata.version!,
createTime: serializer.remoteSerializer.fromVersion(metadata.createTime!)
};
}
3 changes: 2 additions & 1 deletion packages/firestore/src/core/component_provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ export class MemoryComponentProvider implements ComponentProvider {
!cfg.persistenceSettings.durable,
'Can only start memory persistence'
);
return new MemoryPersistence(MemoryEagerDelegate.factory);
const serializer = cfg.platform.newSerializer(cfg.databaseInfo.databaseId);
return new MemoryPersistence(MemoryEagerDelegate.factory, serializer);
}

createRemoteStore(cfg: ComponentConfiguration): RemoteStore {
Expand Down
61 changes: 61 additions & 0 deletions packages/firestore/src/local/bundle_cache.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/**
* @license
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { PersistenceTransaction } from './persistence';
import { PersistencePromise } from './persistence_promise';
import * as bundleProto from '../protos/firestore_bundle_proto';
import { Bundle, NamedQuery } from '../core/bundle';

/**
* Provides interfaces to save and read Firestore bundles.
*/
export interface BundleCache {
/**
* Gets a saved `Bundle` for a given `bundleId`, returns undefined if
* no bundles are found under the given id.
*/
getBundle(
transaction: PersistenceTransaction,
bundleId: string
): PersistencePromise<Bundle | undefined>;

/**
* Saves a `BundleMetadata` from a bundle into local storage, using its id as
* the persistent key.
*/
saveBundleMetadata(
transaction: PersistenceTransaction,
metadata: bundleProto.BundleMetadata
): PersistencePromise<void>;

/**
* Gets a saved `NamedQuery` for the given query name. Returns undefined if
* no queries are found under the given name.
*/
getNamedQuery(
transaction: PersistenceTransaction,
queryName: string
): PersistencePromise<NamedQuery | undefined>;

/**
* Saves a `NamedQuery` from a bundle, using its name as the persistent key.
*/
saveNamedQuery(
transaction: PersistenceTransaction,
query: bundleProto.NamedQuery
): PersistencePromise<void>;
}
112 changes: 112 additions & 0 deletions packages/firestore/src/local/indexeddb_bundle_cache.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
/**
* @license
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { PersistenceTransaction } from './persistence';
import { PersistencePromise } from './persistence_promise';
import * as bundleProto from '../protos/firestore_bundle_proto';
import { BundleCache } from './bundle_cache';
import {
DbBundle,
DbBundlesKey,
DbNamedQuery,
DbNamedQueriesKey
} from './indexeddb_schema';
import { SimpleDbStore } from './simple_db';
import { IndexedDbPersistence } from './indexeddb_persistence';
import { LocalSerializer } from './local_serializer';
import {
Bundle,
fromDbBundle,
fromDbNamedQuery,
NamedQuery,
toDbBundle,
toDbNamedQuery
} from '../core/bundle';

export class IndexedDbBundleCache implements BundleCache {
constructor(private serializer: LocalSerializer) {}

getBundle(
transaction: PersistenceTransaction,
bundleId: string
): PersistencePromise<Bundle | undefined> {
return bundlesStore(transaction)
.get(bundleId)
.next(bundle => {
if (bundle) {
return fromDbBundle(this.serializer, bundle);
}
return undefined;
});
}

saveBundleMetadata(
transaction: PersistenceTransaction,
bundleMetadata: bundleProto.BundleMetadata
): PersistencePromise<void> {
return bundlesStore(transaction).put(
toDbBundle(this.serializer, bundleMetadata)
);
}

getNamedQuery(
transaction: PersistenceTransaction,
queryName: string
): PersistencePromise<NamedQuery | undefined> {
return namedQueriesStore(transaction)
.get(queryName)
.next(query => {
if (query) {
return fromDbNamedQuery(this.serializer, query);
}
return undefined;
});
}

saveNamedQuery(
transaction: PersistenceTransaction,
query: bundleProto.NamedQuery
): PersistencePromise<void> {
return namedQueriesStore(transaction).put(
toDbNamedQuery(this.serializer, query)
);
}
}

/**
* Helper to get a typed SimpleDbStore for the bundles object store.
*/
function bundlesStore(
txn: PersistenceTransaction
): SimpleDbStore<DbBundlesKey, DbBundle> {
return IndexedDbPersistence.getStore<DbBundlesKey, DbBundle>(
txn,
DbBundle.store
);
}

/**
* Helper to get a typed SimpleDbStore for the namedQueries object store.
*/
function namedQueriesStore(
txn: PersistenceTransaction
): SimpleDbStore<DbNamedQueriesKey, DbNamedQuery> {
return IndexedDbPersistence.getStore<DbNamedQueriesKey, DbNamedQuery>(
txn,
DbNamedQuery.store
);
}
11 changes: 11 additions & 0 deletions packages/firestore/src/local/indexeddb_persistence.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import {
EncodedResourcePath,
encodeResourcePath
} from './encoded_resource_path';
import { IndexedDbBundleCache } from './indexeddb_bundle_cache';
import { IndexedDbIndexManager } from './indexeddb_index_manager';
import {
IndexedDbMutationQueue,
Expand Down Expand Up @@ -226,6 +227,7 @@ export class IndexedDbPersistence implements Persistence {
private readonly targetCache: IndexedDbTargetCache;
private readonly indexManager: IndexedDbIndexManager;
private readonly remoteDocumentCache: IndexedDbRemoteDocumentCache;
private readonly bundleCache: IndexedDbBundleCache;
private readonly webStorage: Storage;
readonly referenceDelegate: IndexedDbLruDelegate;

Expand Down Expand Up @@ -259,6 +261,7 @@ export class IndexedDbPersistence implements Persistence {
this.serializer,
this.indexManager
);
this.bundleCache = new IndexedDbBundleCache(this.serializer);
if (platform.window && platform.window.localStorage) {
this.window = platform.window;
this.webStorage = this.window.localStorage;
Expand Down Expand Up @@ -763,6 +766,14 @@ export class IndexedDbPersistence implements Persistence {
return this.indexManager;
}

getBundleCache(): IndexedDbBundleCache {
debugAssert(
this.started,
'Cannot initialize BundleCache before persistence is started.'
);
return this.bundleCache;
}

runTransaction<T>(
action: string,
mode: PersistenceTransactionMode,
Expand Down
Loading