Skip to content

Merge firestore bundles implementation without exposing public APIs #3594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Nov 12, 2020

Conversation

wu-hui
Copy link
Contributor

@wu-hui wu-hui commented Aug 10, 2020

No description provided.

wu-hui and others added 18 commits May 21, 2020 13:45
…3073)

* Manual copy proto and create bundle_proto.d.ts

* IndexedDb schema change to introduce bundle object stores.

* Renaming interfaces without leading I

* Reordering imports

* Address comments

* Add totalBytes and totalDocuments
Implement a bundle reader to read from a underlying readable stream lazily, and parse the content to proto objects.
# Conflicts:
#	integration/messaging/test/utils/openNewTab.js
#	packages/firestore/src/core/component_provider.ts
#	packages/firestore/src/local/indexeddb_persistence.ts
#	packages/firestore/src/local/local_serializer.ts
#	packages/firestore/src/platform/platform.ts
#	packages/firestore/src/platform_browser/browser_platform.ts
#	packages/firestore/src/platform_node/node_platform.ts
#	packages/firestore/src/remote/serializer.ts
#	packages/firestore/test/unit/local/persistence_test_helpers.ts
#	packages/firestore/test/unit/specs/spec_test_components.ts
#	packages/firestore/test/util/test_platform.ts
* Renaming interfaces without leading I

* Initial commit of bundle reading - for web only.

* Tests only run when it is not Node.

* Fix redundant imports

* Fix missing textencoder

* Remove generator.

* Support bundle reader for Node

* Fix rebase errors.

* Remote 'only'

* Merge branch 'wuandy/Bundles' into wuandy/BundleReaderNode

# Conflicts:
#	packages/firestore/src/util/bundle_reader.ts
#	packages/firestore/test/unit/util/bundle.test.ts

* Added more comments, and more tests for Node.

* Implement BundleCache.

* Add applyBundleDocuments to local store.

* Add rest of bundle service to localstore
Fix change buffer bug

* Simplify change buffer get read time logic.

* Fix lint errors

* Add comments.

* Change localstore to check for newer bundle directly.

* A little more comments.

* Address comments.

* Address comments 2

* Make it tree-shakeable.

* More comments addressing.

* Another around of comments
* Fix firestore_bundle_proto rollup issue

* Create lemon-steaks-draw.md
Still issues with shim.ts

# Conflicts:
#	packages/firestore/src/core/component_provider.ts
#	packages/firestore/src/core/firestore_client.ts
#	packages/firestore/src/core/sync_engine.ts
#	packages/firestore/src/local/local_store.ts
#	packages/firestore/src/remote/serializer.ts
#	packages/firestore/test/unit/local/persistence_test_helpers.ts
#	packages/firestore/test/util/helpers.ts
* Make loadBundle work with exp build

* Use legacy.LoadBundleTask
# Conflicts:
#	packages/firestore/exp/src/api/database.ts
#	packages/firestore/src/core/component_provider.ts
#	packages/firestore/src/core/firestore_client.ts
#	packages/firestore/src/core/sync_engine.ts
#	packages/firestore/src/platform/serializer.ts
# Conflicts:
#	packages/firestore/exp/src/api/database.ts
@changeset-bot
Copy link

changeset-bot bot commented Aug 10, 2020

🦋 Changeset detected

Latest commit: 205d1c8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages
Name Type
@firebase/firestore
firebase Patch
@firebase/rules-unit-testing Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@wu-hui wu-hui requested a review from egilmorez as a code owner August 10, 2020 20:44
wu-hui and others added 3 commits August 11, 2020 15:05
* Update query-document mapping from bundles.

* Only update mapping when query read time is updated.

* Fix lint errors

* Better code structure.

* Use serializer constructed from firestore client.

* Don't inline await
# Conflicts:
#	packages/firestore/exp/src/api/database.ts
#	packages/firestore/scripts/run-tests.js
#	packages/firestore/src/core/component_provider.ts
#	packages/firestore/src/core/firestore_client.ts
#	packages/firestore/src/core/sync_engine.ts
#	packages/firestore/src/local/local_store.ts
#	packages/firestore/test/unit/local/local_store.test.ts
@wu-hui wu-hui requested a review from zwu52 as a code owner August 17, 2020 14:28
@google-oss-bot
Copy link
Contributor

google-oss-bot commented Aug 17, 2020

Binary Size Report

Affected SDKs

  • @firebase/firestore

    Type Base (878a1bf) Head (7dbf54f) Diff
    browser 241 kB 254 kB +12.8 kB (+5.3%)
    esm2017 190 kB 200 kB +9.15 kB (+4.8%)
    main 475 kB 499 kB +23.9 kB (+5.0%)
    module 241 kB 254 kB +12.8 kB (+5.3%)
    react-native 190 kB 200 kB +9.15 kB (+4.8%)
  • @firebase/firestore/exp

    Type Base (878a1bf) Head (7dbf54f) Diff
    browser 189 kB 198 kB +9.19 kB (+4.9%)
    main 476 kB 500 kB +24.0 kB (+5.0%)
    module 189 kB 198 kB +9.19 kB (+4.9%)
    react-native 189 kB 198 kB +9.19 kB (+4.9%)
  • @firebase/firestore/lite

    Type Base (878a1bf) Head (7dbf54f) Diff
    browser 62.9 kB 63.1 kB +231 B (+0.4%)
    main 139 kB 139 kB +244 B (+0.2%)
    module 62.9 kB 63.1 kB +231 B (+0.4%)
    react-native 63.1 kB 63.3 kB +231 B (+0.4%)
  • @firebase/firestore/memory

    Type Base (878a1bf) Head (7dbf54f) Diff
    browser 177 kB 191 kB +13.2 kB (+7.5%)
    esm2017 139 kB 149 kB +9.90 kB (+7.1%)
    main 344 kB 368 kB +24.1 kB (+7.0%)
    module 177 kB 191 kB +13.2 kB (+7.5%)
    react-native 139 kB 149 kB +9.90 kB (+7.1%)
  • firebase

    Type Base (878a1bf) Head (7dbf54f) Diff
    firebase-firestore.js 280 kB 292 kB +12.3 kB (+4.4%)
    firebase-firestore.memory.js 218 kB 230 kB +12.8 kB (+5.9%)
    firebase.js 822 kB 835 kB +12.3 kB (+1.5%)

Test Logs

@google-cla
Copy link

google-cla bot commented Nov 9, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

# Conflicts:
#	packages/firestore/exp/test/shim.ts
#	packages/firestore/src/api/database.ts
@google-cla
Copy link

google-cla bot commented Nov 9, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

Copy link
Contributor Author

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has changed quite a bit on the api/firestore_client level since last time you reviewed it.

Also, this is no longer release but hide it from public API, it is release bundles as free functions. I will hold it until we got all approvals and ready to launch.

@@ -0,0 +1,3 @@
---
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines 100 to 125
export interface LoadBundleTask {
onProgress(
next?: (progress: LoadBundleTaskProgress) => any,
error?: (error: Error) => any,
complete?: () => void
): void;

then<T, R>(
onFulfilled?: (a: LoadBundleTaskProgress) => T | PromiseLike<T>,
onRejected?: (a: Error) => R | PromiseLike<R>
): Promise<T | R>;

catch<R>(
onRejected: (a: Error) => R | PromiseLike<R>
): Promise<R | LoadBundleTaskProgress>;
}

export interface LoadBundleTaskProgress {
documentsLoaded: number;
totalDocuments: number;
bytesLoaded: number;
totalBytes: number;
taskState: TaskState;
}

export type TaskState = 'Error' | 'Running' | 'Success';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will keep it this way, since this is now a release PR.

}

export type TaskState = 'Error' | 'Running' | 'Success';

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* This ensures that the loaded documents do not get garbage collected
* right away.
*/
export function umbrellaTarget(bundleName: string): Target {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines 597 to 599
return resultTask.catch(e => {
logWarn(LOG_TAG, `Loading bundle failed with ${e}`);
});
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks different now, but should be fixed.

const reader = createBundleReader(bundleData, newSerializer(databaseId));
const syncEngine = await getSyncEngine(firestore);

loadBundleSyncEngine(syncEngine, reader, resultTask);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

loadBundleSyncEngine(syncEngine, reader, resultTask);
return resultTask.catch((e: Error) => {
logWarn(LOG_TAG, `Loading bundle failed with ${e}`);
});
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fixed, although this particular piece of code no longer exists.

}

return null;
// return new Query(namedQuery.query, firestore, null);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


async namedQuery(name: string): Promise<Query | null> {
return null;
// return namedQuery(this._delegate, name);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -129,7 +129,8 @@ const manglePrivatePropertiesOptions = {
},
mangle: {
properties: {
regex: /^__PRIVATE_/
regex: /^__PRIVATE_/,
reserved: ['do']
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@wu-hui wu-hui changed the title Merge firestore bundles into master but hide it from public API Release firestore bundles as free functions Nov 9, 2020
@google-cla
Copy link

google-cla bot commented Nov 9, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla
Copy link

google-cla bot commented Nov 11, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@@ -427,36 +424,6 @@ export function setLogLevel(level: PublicLogLevel): void {
setClientLogLevel(level);
}

export function loadBundle(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can keep these functions as long as they are not referenced.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks there there are still some stray API bits, but otherwise good. Since the size tooling seems to be broken at the moment, can you provide a rough number on the size increase?

"@firebase/firestore": feature
---

Add support for loading Firestore Bundle Files.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

taskState: TaskState;
}

export type TaskState = 'Error' | 'Running' | 'Success';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove all this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines 100 to 125
export interface LoadBundleTask {
onProgress(
next?: (progress: LoadBundleTaskProgress) => any,
error?: (error: Error) => any,
complete?: () => void
): void;

then<T, R>(
onFulfilled?: (a: LoadBundleTaskProgress) => T | PromiseLike<T>,
onRejected?: (a: Error) => R | PromiseLike<R>
): Promise<T | R>;

catch<R>(
onRejected: (a: Error) => R | PromiseLike<R>
): Promise<R | LoadBundleTaskProgress>;
}

export interface LoadBundleTaskProgress {
documentsLoaded: number;
totalDocuments: number;
bytesLoaded: number;
totalBytes: number;
taskState: TaskState;
}

export type TaskState = 'Error' | 'Running' | 'Success';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove as discussed.

}

export type TaskState = 'Error' | 'Running' | 'Success';

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove.

export function namedQuery(
firestore: FirebaseFirestore,
name: string
): Promise<Query<DocumentData> | null>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove these as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

data: ReadableStream<Uint8Array> | ArrayBuffer | string,
resultTask: LoadBundleTask
): Promise<void> {
client.verifyNotTerminated();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other functions in this file don't call verifyNotTerminated (done at callsite). Should we follow this pattern here as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same just below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

});
}

async function loadBundleImpl(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this potentially be inlined? Or did we not want to make syncEngineLoadBundle async?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that was the reason.

@google-cla
Copy link

google-cla bot commented Nov 12, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

Copy link
Contributor Author

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

firebase-firestore.js 283361 -> 286235
firebase-firestore.memory.js 221571 -> 224766

"@firebase/firestore": feature
---

Add support for loading Firestore Bundle Files.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

taskState: TaskState;
}

export type TaskState = 'Error' | 'Running' | 'Success';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines 100 to 125
export interface LoadBundleTask {
onProgress(
next?: (progress: LoadBundleTaskProgress) => any,
error?: (error: Error) => any,
complete?: () => void
): void;

then<T, R>(
onFulfilled?: (a: LoadBundleTaskProgress) => T | PromiseLike<T>,
onRejected?: (a: Error) => R | PromiseLike<R>
): Promise<T | R>;

catch<R>(
onRejected: (a: Error) => R | PromiseLike<R>
): Promise<R | LoadBundleTaskProgress>;
}

export interface LoadBundleTaskProgress {
documentsLoaded: number;
totalDocuments: number;
bytesLoaded: number;
totalBytes: number;
taskState: TaskState;
}

export type TaskState = 'Error' | 'Running' | 'Success';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}

export type TaskState = 'Error' | 'Running' | 'Success';

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

export function namedQuery(
firestore: FirebaseFirestore,
name: string
): Promise<Query<DocumentData> | null>;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -427,36 +424,6 @@ export function setLogLevel(level: PublicLogLevel): void {
setClientLogLevel(level);
}

export function loadBundle(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

data: ReadableStream<Uint8Array> | ArrayBuffer | string,
resultTask: LoadBundleTask
): Promise<void> {
client.verifyNotTerminated();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

});
}

async function loadBundleImpl(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that was the reason.

@wu-hui wu-hui assigned schmidt-sebastian and unassigned wu-hui Nov 12, 2020
@wu-hui wu-hui changed the title Release firestore bundles as free functions Merge firestore bundles implementation without exposing public APIs Nov 12, 2020
Copy link
Contributor

@schmidt-sebastian schmidt-sebastian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wheee!

db: Firestore,
bundleData: ArrayBuffer | ReadableStream<Uint8Array> | string
): ApiLoadBundleTask {
const resultTask = new LoadBundleTask();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to figure out where to call "ensureClientConfigured". It might depend on how we expose this function in the end.

# Conflicts:
#	packages/firestore/src/api/database.ts
#	packages/firestore/test/unit/specs/spec_builder.ts
@google-cla
Copy link

google-cla bot commented Nov 12, 2020

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@hsubox76 hsubox76 merged commit 6dffdf2 into master Nov 12, 2020
@google-oss-bot google-oss-bot mentioned this pull request Nov 12, 2020
@firebase firebase locked and limited conversation to collaborators Dec 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants