Skip to content

Merge firestore bundles implementation without exposing public APIs #3594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Nov 12, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
f347dea
Add bundle.proto d.ts and introduce IDB object stores for bundles. (…
wu-hui May 21, 2020
b0c8299
Implement a bundle reader for Web (#3097)
wu-hui Jun 5, 2020
2f179bc
Enable bundle reader for Node. (#3167)
wu-hui Jun 10, 2020
d3f0ab4
Implement BundleCache for IDB and memory. (#3170)
wu-hui Jun 14, 2020
6c375b9
Merge branch 'master' into wuandy/Bundles
wu-hui Jun 24, 2020
e8b9ec2
Implement bundle features in local store. (#3200)
wu-hui Jun 25, 2020
2e73801
Merge branch 'master' into wuandy/Bundles
wu-hui Jun 26, 2020
fff6b1b
Merge branch 'master' into wuandy/Bundles
wu-hui Jun 29, 2020
3a78d7c
Fix firestore_bundle_proto rollup issue. (#3316)
wu-hui Jun 29, 2020
0bf9c73
Implement bundle loading. (#3201)
wu-hui Jul 11, 2020
2bfa185
Merge branch 'master' into wuandy/Bundles
wu-hui Jul 11, 2020
9bfb995
Bundles load from secondary tabs (#3393)
wu-hui Jul 23, 2020
a9bf16c
Merge branch 'master' into wuandy/Bundles
wu-hui Jul 24, 2020
8b8a097
Make loadBundle work with exp build (#3488)
wu-hui Jul 28, 2020
5ebf8c3
Bundle's named query resume. (#3395)
wu-hui Aug 6, 2020
8cfb471
Merge branch 'master' into wuandy/Bundles
wu-hui Aug 10, 2020
2bec6d8
Merge remote-tracking branch 'origin/wuandy/Bundles' into wuandy/Bundles
wu-hui Aug 10, 2020
0f697df
Merge with named query change.
wu-hui Aug 10, 2020
dc02c03
Add missing byte_stream_reader.ts for *_lite platforms
wu-hui Aug 10, 2020
15b96a9
Finally builds and passes
wu-hui Aug 11, 2020
b4f9936
Update query-document mapping from bundles. (#3620)
wu-hui Aug 14, 2020
ef202a1
Merge branch 'master' into wuandy/Bundles
wu-hui Aug 17, 2020
c4b0995
Introduce an umbrella target for bundled documents (#3643)
wu-hui Aug 19, 2020
4e09f58
Merge branch 'master' into wuandy/Bundles
wu-hui Oct 8, 2020
775d58a
Restore some files from master
wu-hui Oct 8, 2020
5dd5c9c
Fix build errors
wu-hui Oct 8, 2020
e53cf69
Fix lint errors
wu-hui Oct 8, 2020
4bbcdc0
Merge branch 'master' into wuandy/Bundles
wu-hui Oct 8, 2020
bd6adc6
Hide bundles from public API
wu-hui Oct 9, 2020
259160b
Fix build error.
wu-hui Oct 9, 2020
9a804fc
Add bundle proto to extern, version upgrade and make exp bundle public
wu-hui Oct 13, 2020
72fb466
Merge branch 'master' into wuandy/Bundles
wu-hui Oct 20, 2020
3b208b2
Fix lint error
wu-hui Oct 20, 2020
223bc9a
Move bundle integration tests under api/ and complete exp API (#3962)
wu-hui Oct 27, 2020
b80c880
Merge branch 'master' into wuandy/Bundles
wu-hui Oct 29, 2020
5b0526b
Bundles as free functions (#4006)
wu-hui Nov 9, 2020
089fcdd
Merge branch 'master' into wuandy/Bundles
wu-hui Nov 9, 2020
80dc41d
Address comments
wu-hui Nov 9, 2020
d4ab17f
Remove bundles from public interfaces.
wu-hui Nov 11, 2020
1b3d633
Remove bundles from public interfaces take 2.
wu-hui Nov 12, 2020
205d1c8
Merge branch 'master' into wuandy/Bundles
wu-hui Nov 12, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/lemon-steaks-draw.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a changeset entry to describe the size increase.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

"@firebase/firestore": internal
---

Merge bundle loading implementation without exposing public API
5 changes: 4 additions & 1 deletion packages-exp/auth-exp/index.webworker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,10 @@ registerAuth(ClientPlatform.WORKER);
export function getAuth(app = getApp()): Auth {
// Unlike the other environments, we need to explicitly check if indexedDb is
// available. That means doing the whole rigamarole
const auth = _getProvider(app, _ComponentName.AUTH).getImmediate() as AuthImpl;
const auth = _getProvider(
app,
_ComponentName.AUTH
).getImmediate() as AuthImpl;

// This promise is intended to float; auth initialization happens in the
// background, meanwhile the auth object may be used by the app.
Expand Down
7 changes: 5 additions & 2 deletions packages/database/src/api/Database.ts
Original file line number Diff line number Diff line change
Expand Up @@ -157,14 +157,17 @@ export class Database implements FirebaseService {
validateUrl(apiName, 1, parsedURL);

const repoInfo = parsedURL.repoInfo;
if (!repoInfo.isCustomHost() && repoInfo.host !== this.repo_.repoInfo_.host) {
if (
!repoInfo.isCustomHost() &&
repoInfo.host !== this.repo_.repoInfo_.host
) {
fatal(
apiName +
': Host name does not match the current database: ' +
'(found ' +
repoInfo.host +
' but expected ' +
this.repo_.repoInfo_.host+
this.repo_.repoInfo_.host +
')'
);
}
Expand Down
3 changes: 1 addition & 2 deletions packages/firestore/exp/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,7 @@ export {
waitForPendingWrites,
disableNetwork,
enableNetwork,
terminate,
Settings
terminate
} from './src/api/database';

export {
Expand Down
1 change: 1 addition & 0 deletions packages/firestore/externs.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
"packages/webchannel-wrapper/src/index.d.ts",
"packages/util/dist/src/crypt.d.ts",
"packages/util/dist/src/environment.d.ts",
"packages/firestore/src/protos/firestore_bundle_proto.ts",
"packages/firestore/src/protos/firestore_proto_api.d.ts",
"packages/firestore/src/util/error.ts",
"packages/firestore/src/local/indexeddb_schema.ts",
Expand Down
70 changes: 69 additions & 1 deletion packages/firestore/rollup.shared.js
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,75 @@ const manglePrivatePropertiesOptions = {
},
mangle: {
properties: {
regex: /^__PRIVATE_/
regex: /^__PRIVATE_/,
// All JS Keywords are reserved. Although this should be taken cared of by
// Terser, we have seen issues with `do`, hence the extra caution.
reserved: [
'abstract',
'arguments',
'await',
'boolean',
'break',
'byte',
'case',
'catch',
'char',
'class',
'const',
'continue',
'debugger',
'default',
'delete',
'do',
'double',
'else',
'enum',
'eval',
'export',
'extends',
'false',
'final',
'finally',
'float',
'for',
'function',
'goto',
'if',
'implements',
'import',
'in',
'instanceof',
'int',
'interface',
'let',
'long',
'native',
'new',
'null',
'package',
'private',
'protected',
'public',
'return',
'short',
'static',
'super',
'switch',
'synchronized',
'this',
'throw',
'throws',
'transient',
'true',
'try',
'typeof',
'var',
'void',
'volatile',
'while',
'with',
'yield'
]
}
}
};
Expand Down
140 changes: 140 additions & 0 deletions packages/firestore/src/api/bundle.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
/**
* @license
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { Deferred } from '../util/promise';
import { PartialObserver } from './observer';
import { debugAssert } from '../util/assert';
import { FirestoreError } from '../util/error';

export interface ApiLoadBundleTask {
onProgress(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
next?: (progress: ApiLoadBundleTaskProgress) => any,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
error?: (error: Error) => any,
complete?: () => void
): void;

then<T, R>(
onFulfilled?: (a: ApiLoadBundleTaskProgress) => T | PromiseLike<T>,
onRejected?: (a: Error) => R | PromiseLike<R>
): Promise<T | R>;

catch<R>(
onRejected: (a: Error) => R | PromiseLike<R>
): Promise<R | ApiLoadBundleTaskProgress>;
}

export interface ApiLoadBundleTaskProgress {
documentsLoaded: number;
totalDocuments: number;
bytesLoaded: number;
totalBytes: number;
taskState: TaskState;
}

export type TaskState = 'Error' | 'Running' | 'Success';

export class LoadBundleTask
implements ApiLoadBundleTask, PromiseLike<ApiLoadBundleTaskProgress> {
private _progressObserver: PartialObserver<ApiLoadBundleTaskProgress> = {};
private _taskCompletionResolver = new Deferred<ApiLoadBundleTaskProgress>();

private _lastProgress: ApiLoadBundleTaskProgress = {
taskState: 'Running',
totalBytes: 0,
totalDocuments: 0,
bytesLoaded: 0,
documentsLoaded: 0
};

onProgress(
next?: (progress: ApiLoadBundleTaskProgress) => unknown,
error?: (err: Error) => unknown,
complete?: () => void
): void {
this._progressObserver = {
next,
error,
complete
};
}

catch<R>(
onRejected: (a: Error) => R | PromiseLike<R>
): Promise<R | ApiLoadBundleTaskProgress> {
return this._taskCompletionResolver.promise.catch(onRejected);
}

then<T, R>(
onFulfilled?: (a: ApiLoadBundleTaskProgress) => T | PromiseLike<T>,
onRejected?: (a: Error) => R | PromiseLike<R>
): Promise<T | R> {
return this._taskCompletionResolver.promise.then(onFulfilled, onRejected);
}

/**
* Notifies all observers that bundle loading has completed, with a provided
* `LoadBundleTaskProgress` object.
*/
_completeWith(progress: ApiLoadBundleTaskProgress): void {
debugAssert(
progress.taskState === 'Success',
'Task is not completed with Success.'
);
this._updateProgress(progress);
if (this._progressObserver.complete) {
this._progressObserver.complete();
}

this._taskCompletionResolver.resolve(progress);
}

/**
* Notifies all observers that bundle loading has failed, with a provided
* `Error` as the reason.
*/
_failWith(error: FirestoreError): void {
this._lastProgress.taskState = 'Error';

if (this._progressObserver.next) {
this._progressObserver.next(this._lastProgress);
}

if (this._progressObserver.error) {
this._progressObserver.error(error);
}

this._taskCompletionResolver.reject(error);
}

/**
* Notifies a progress update of loading a bundle.
* @param progress The new progress.
*/
_updateProgress(progress: ApiLoadBundleTaskProgress): void {
debugAssert(
this._lastProgress.taskState === 'Running',
'Cannot update progress on a completed or failed task'
);

this._lastProgress = progress;
if (this._progressObserver.next) {
this._progressObserver.next(progress);
}
}
}
33 changes: 33 additions & 0 deletions packages/firestore/src/api/database.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ import {
FirestoreClient,
firestoreClientGetDocumentsFromLocalCache,
firestoreClientGetDocumentsViaSnapshotListener,
firestoreClientGetNamedQuery,
firestoreClientListen,
firestoreClientLoadBundle,
firestoreClientTransaction,
firestoreClientWrite
} from '../core/firestore_client';
Expand Down Expand Up @@ -157,6 +159,7 @@ import {

import { makeDatabaseInfo } from '../../lite/src/api/database';
import { DEFAULT_HOST } from '../../lite/src/api/components';
import { ApiLoadBundleTask, LoadBundleTask } from './bundle';

/**
* Constant used to indicate the LRU garbage collection should be disabled.
Expand Down Expand Up @@ -440,6 +443,36 @@ export function setLogLevel(level: PublicLogLevel): void {
setClientLogLevel(level);
}

export function loadBundle(
db: Firestore,
bundleData: ArrayBuffer | ReadableStream<Uint8Array> | string
): ApiLoadBundleTask {
const resultTask = new LoadBundleTask();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to figure out where to call "ensureClientConfigured". It might depend on how we expose this function in the end.

// eslint-disable-next-line @typescript-eslint/no-floating-promises
firestoreClientLoadBundle(
ensureFirestoreConfigured(db._delegate),
bundleData,
resultTask
);
return resultTask;
}

export function namedQuery(
db: Firestore,
name: string
): Promise<PublicQuery | null> {
return firestoreClientGetNamedQuery(
ensureFirestoreConfigured(db._delegate),
name
).then(namedQuery => {
if (!namedQuery) {
return null;
}

return new Query(namedQuery.query, db, null);
});
}

/**
* A reference to a transaction.
*/
Expand Down
Loading