-
Notifications
You must be signed in to change notification settings - Fork 937
Merge firestore bundles implementation without exposing public APIs #3594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…3073) * Manual copy proto and create bundle_proto.d.ts * IndexedDb schema change to introduce bundle object stores. * Renaming interfaces without leading I * Reordering imports * Address comments * Add totalBytes and totalDocuments
Implement a bundle reader to read from a underlying readable stream lazily, and parse the content to proto objects.
# Conflicts: # integration/messaging/test/utils/openNewTab.js # packages/firestore/src/core/component_provider.ts # packages/firestore/src/local/indexeddb_persistence.ts # packages/firestore/src/local/local_serializer.ts # packages/firestore/src/platform/platform.ts # packages/firestore/src/platform_browser/browser_platform.ts # packages/firestore/src/platform_node/node_platform.ts # packages/firestore/src/remote/serializer.ts # packages/firestore/test/unit/local/persistence_test_helpers.ts # packages/firestore/test/unit/specs/spec_test_components.ts # packages/firestore/test/util/test_platform.ts
* Renaming interfaces without leading I * Initial commit of bundle reading - for web only. * Tests only run when it is not Node. * Fix redundant imports * Fix missing textencoder * Remove generator. * Support bundle reader for Node * Fix rebase errors. * Remote 'only' * Merge branch 'wuandy/Bundles' into wuandy/BundleReaderNode # Conflicts: # packages/firestore/src/util/bundle_reader.ts # packages/firestore/test/unit/util/bundle.test.ts * Added more comments, and more tests for Node. * Implement BundleCache. * Add applyBundleDocuments to local store. * Add rest of bundle service to localstore Fix change buffer bug * Simplify change buffer get read time logic. * Fix lint errors * Add comments. * Change localstore to check for newer bundle directly. * A little more comments. * Address comments. * Address comments 2 * Make it tree-shakeable. * More comments addressing. * Another around of comments
* Fix firestore_bundle_proto rollup issue * Create lemon-steaks-draw.md
Still issues with shim.ts # Conflicts: # packages/firestore/src/core/component_provider.ts # packages/firestore/src/core/firestore_client.ts # packages/firestore/src/core/sync_engine.ts # packages/firestore/src/local/local_store.ts # packages/firestore/src/remote/serializer.ts # packages/firestore/test/unit/local/persistence_test_helpers.ts # packages/firestore/test/util/helpers.ts
* Make loadBundle work with exp build * Use legacy.LoadBundleTask
# Conflicts: # packages/firestore/exp/src/api/database.ts # packages/firestore/src/core/component_provider.ts # packages/firestore/src/core/firestore_client.ts # packages/firestore/src/core/sync_engine.ts # packages/firestore/src/platform/serializer.ts
# Conflicts: # packages/firestore/exp/src/api/database.ts
🦋 Changeset detectedLatest commit: 205d1c8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
* Update query-document mapping from bundles. * Only update mapping when query read time is updated. * Fix lint errors * Better code structure. * Use serializer constructed from firestore client. * Don't inline await
# Conflicts: # packages/firestore/exp/src/api/database.ts # packages/firestore/scripts/run-tests.js # packages/firestore/src/core/component_provider.ts # packages/firestore/src/core/firestore_client.ts # packages/firestore/src/core/sync_engine.ts # packages/firestore/src/local/local_store.ts # packages/firestore/test/unit/local/local_store.test.ts
Binary Size ReportAffected SDKs
Test Logs |
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
# Conflicts: # packages/firestore/exp/test/shim.ts # packages/firestore/src/api/database.ts
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has changed quite a bit on the api/firestore_client level since last time you reviewed it.
Also, this is no longer release but hide it from public API, it is release bundles as free functions. I will hold it until we got all approvals and ready to launch.
@@ -0,0 +1,3 @@ | |||
--- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firestore-types/index.d.ts
Outdated
export interface LoadBundleTask { | ||
onProgress( | ||
next?: (progress: LoadBundleTaskProgress) => any, | ||
error?: (error: Error) => any, | ||
complete?: () => void | ||
): void; | ||
|
||
then<T, R>( | ||
onFulfilled?: (a: LoadBundleTaskProgress) => T | PromiseLike<T>, | ||
onRejected?: (a: Error) => R | PromiseLike<R> | ||
): Promise<T | R>; | ||
|
||
catch<R>( | ||
onRejected: (a: Error) => R | PromiseLike<R> | ||
): Promise<R | LoadBundleTaskProgress>; | ||
} | ||
|
||
export interface LoadBundleTaskProgress { | ||
documentsLoaded: number; | ||
totalDocuments: number; | ||
bytesLoaded: number; | ||
totalBytes: number; | ||
taskState: TaskState; | ||
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will keep it this way, since this is now a release PR.
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
* This ensures that the loaded documents do not get garbage collected | ||
* right away. | ||
*/ | ||
export function umbrellaTarget(bundleName: string): Target { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
return resultTask.catch(e => { | ||
logWarn(LOG_TAG, `Loading bundle failed with ${e}`); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks different now, but should be fixed.
const reader = createBundleReader(bundleData, newSerializer(databaseId)); | ||
const syncEngine = await getSyncEngine(firestore); | ||
|
||
loadBundleSyncEngine(syncEngine, reader, resultTask); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
loadBundleSyncEngine(syncEngine, reader, resultTask); | ||
return resultTask.catch((e: Error) => { | ||
logWarn(LOG_TAG, `Loading bundle failed with ${e}`); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is fixed, although this particular piece of code no longer exists.
} | ||
|
||
return null; | ||
// return new Query(namedQuery.query, firestore, null); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firestore/exp/test/shim.ts
Outdated
|
||
async namedQuery(name: string): Promise<Query | null> { | ||
return null; | ||
// return namedQuery(this._delegate, name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firestore/rollup.shared.js
Outdated
@@ -129,7 +129,8 @@ const manglePrivatePropertiesOptions = { | |||
}, | |||
mangle: { | |||
properties: { | |||
regex: /^__PRIVATE_/ | |||
regex: /^__PRIVATE_/, | |||
reserved: ['do'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
@@ -427,36 +424,6 @@ export function setLogLevel(level: PublicLogLevel): void { | |||
setClientLogLevel(level); | |||
} | |||
|
|||
export function loadBundle( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can keep these functions as long as they are not referenced.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks there there are still some stray API bits, but otherwise good. Since the size tooling seems to be broken at the moment, can you provide a rough number on the size increase?
.changeset/lemon-steaks-draw.md
Outdated
"@firebase/firestore": feature | ||
--- | ||
|
||
Add support for loading Firestore Bundle Files. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is no longer correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firebase/index.d.ts
Outdated
taskState: TaskState; | ||
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove all this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firestore-types/index.d.ts
Outdated
export interface LoadBundleTask { | ||
onProgress( | ||
next?: (progress: LoadBundleTaskProgress) => any, | ||
error?: (error: Error) => any, | ||
complete?: () => void | ||
): void; | ||
|
||
then<T, R>( | ||
onFulfilled?: (a: LoadBundleTaskProgress) => T | PromiseLike<T>, | ||
onRejected?: (a: Error) => R | PromiseLike<R> | ||
): Promise<T | R>; | ||
|
||
catch<R>( | ||
onRejected: (a: Error) => R | PromiseLike<R> | ||
): Promise<R | LoadBundleTaskProgress>; | ||
} | ||
|
||
export interface LoadBundleTaskProgress { | ||
documentsLoaded: number; | ||
totalDocuments: number; | ||
bytesLoaded: number; | ||
totalBytes: number; | ||
taskState: TaskState; | ||
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove as discussed.
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove.
export function namedQuery( | ||
firestore: FirebaseFirestore, | ||
name: string | ||
): Promise<Query<DocumentData> | null>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove these as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
data: ReadableStream<Uint8Array> | ArrayBuffer | string, | ||
resultTask: LoadBundleTask | ||
): Promise<void> { | ||
client.verifyNotTerminated(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other functions in this file don't call verifyNotTerminated
(done at callsite). Should we follow this pattern here as well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same just below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
}); | ||
} | ||
|
||
async function loadBundleImpl( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this potentially be inlined? Or did we not want to make syncEngineLoadBundle
async?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, that was the reason.
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
firebase-firestore.js 283361 -> 286235
firebase-firestore.memory.js 221571 -> 224766
.changeset/lemon-steaks-draw.md
Outdated
"@firebase/firestore": feature | ||
--- | ||
|
||
Add support for loading Firestore Bundle Files. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firebase/index.d.ts
Outdated
taskState: TaskState; | ||
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
packages/firestore-types/index.d.ts
Outdated
export interface LoadBundleTask { | ||
onProgress( | ||
next?: (progress: LoadBundleTaskProgress) => any, | ||
error?: (error: Error) => any, | ||
complete?: () => void | ||
): void; | ||
|
||
then<T, R>( | ||
onFulfilled?: (a: LoadBundleTaskProgress) => T | PromiseLike<T>, | ||
onRejected?: (a: Error) => R | PromiseLike<R> | ||
): Promise<T | R>; | ||
|
||
catch<R>( | ||
onRejected: (a: Error) => R | PromiseLike<R> | ||
): Promise<R | LoadBundleTaskProgress>; | ||
} | ||
|
||
export interface LoadBundleTaskProgress { | ||
documentsLoaded: number; | ||
totalDocuments: number; | ||
bytesLoaded: number; | ||
totalBytes: number; | ||
taskState: TaskState; | ||
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
} | ||
|
||
export type TaskState = 'Error' | 'Running' | 'Success'; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
export function namedQuery( | ||
firestore: FirebaseFirestore, | ||
name: string | ||
): Promise<Query<DocumentData> | null>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@@ -427,36 +424,6 @@ export function setLogLevel(level: PublicLogLevel): void { | |||
setClientLogLevel(level); | |||
} | |||
|
|||
export function loadBundle( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
data: ReadableStream<Uint8Array> | ArrayBuffer | string, | ||
resultTask: LoadBundleTask | ||
): Promise<void> { | ||
client.verifyNotTerminated(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
}); | ||
} | ||
|
||
async function loadBundleImpl( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, that was the reason.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wheee!
db: Firestore, | ||
bundleData: ArrayBuffer | ReadableStream<Uint8Array> | string | ||
): ApiLoadBundleTask { | ||
const resultTask = new LoadBundleTask(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to figure out where to call "ensureClientConfigured". It might depend on how we expose this function in the end.
# Conflicts: # packages/firestore/src/api/database.ts # packages/firestore/test/unit/specs/spec_builder.ts
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
No description provided.