Skip to content

Implement Firestore IndexBackfiller #6261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Jun 8, 2022
Merged

Conversation

tom-andersen
Copy link
Contributor

@tom-andersen tom-andersen commented May 10, 2022

Implement index backfiller

ehsannas and others added 3 commits April 6, 2022 10:01
* Integrate Document Overlay with the SDK.

* we should call ObjectValue.delete if value is null.

* Remove unnecessary null check from MemoryDocumentOverlayCache.saveOverlay(), like is done in firebase/firebase-android-sdk#3518

* Address comments.

* Port changes from Android SDK PR#3420.

Note that we are not going to do any processing in the background.

* Port overlay recalculation bug (Android SDK PR #3495).

* Fix overlay bug when offline (Port Android SDK PR #3537).

* Address feedback.

* Better null check.

Co-authored-by: Denver Coneybeare <[email protected]>
* Overlay migration

* Remove intermediate state.

* Add test.

* Address feedback.
@changeset-bot
Copy link

changeset-bot bot commented May 10, 2022

⚠️ No Changeset found

Latest commit: 7b8398d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

tom-andersen and others added 11 commits May 10, 2022 13:06
* wip.

* remove console.log.

* remove .only for test.

* lint fixes.
…to tomandersen/index-backfiller

# Conflicts:
#	packages/firestore/src/core/component_provider.ts
#	packages/firestore/src/local/index_backfiller.ts
#	packages/firestore/src/local/persistence_promise.ts
#	packages/firestore/test/unit/specs/spec_test_components.ts
@tom-andersen tom-andersen changed the base branch from master to firestore-overlays May 16, 2022 18:50
@google-oss-bot
Copy link
Contributor

google-oss-bot commented May 16, 2022

Size Report 1

Affected Products

  • @firebase/app

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    browser13.8 kB13.9 kB+84 B (+0.6%)
    esm518.0 kB18.1 kB+84 B (+0.5%)
    main19.0 kB19.1 kB+84 B (+0.4%)
    module13.8 kB13.9 kB+84 B (+0.6%)
  • @firebase/app-check

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    browser25.2 kB25.3 kB+122 B (+0.5%)
    esm529.8 kB29.9 kB+122 B (+0.4%)
    main31.0 kB31.1 kB+122 B (+0.4%)
    module25.2 kB25.3 kB+122 B (+0.5%)
  • @firebase/auth

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    browser?155 kB? (?)
    cordova?183 kB? (?)
    esm5?202 kB? (?)
    main?148 kB? (?)
    module?155 kB? (?)
    react-native?168 kB? (?)
  • @firebase/firestore

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    browser257 kB262 kB+4.81 kB (+1.9%)
    esm5319 kB325 kB+5.87 kB (+1.8%)
    main515 kB521 kB+5.97 kB (+1.2%)
    module257 kB262 kB+4.81 kB (+1.9%)
    react-native257 kB262 kB+4.81 kB (+1.9%)
  • @firebase/messaging

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    browser20.8 kB21.1 kB+289 B (+1.4%)
    esm526.2 kB26.5 kB+289 B (+1.1%)
    main26.9 kB27.2 kB+289 B (+1.1%)
    module20.8 kB21.1 kB+289 B (+1.4%)
  • @firebase/messaging-sw

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    main29.6 kB29.9 kB+289 B (+1.0%)
    module22.9 kB23.2 kB+289 B (+1.3%)
  • @firebase/remote-config

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    browser18.8 kB19.2 kB+331 B (+1.8%)
    esm523.7 kB24.0 kB+331 B (+1.4%)
    main24.9 kB25.2 kB+331 B (+1.3%)
    module18.8 kB19.2 kB+331 B (+1.8%)
  • @firebase/storage

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    main57.3 kB57.4 kB+84 B (+0.1%)
  • bundle

    43 size changes

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    analytics (logEvent)41.5 kB41.6 kB+76 B (+0.2%)
    app-check (CustomProvider)35.2 kB35.4 kB+186 B (+0.5%)
    app-check (ReCaptchaEnterpriseProvider)37.4 kB37.6 kB+186 B (+0.5%)
    app-check (ReCaptchaV3Provider)37.4 kB37.5 kB+186 B (+0.5%)
    auth (Anonymous)66.1 kB66.1 kB+76 B (+0.1%)
    auth (EmailAndPassword)70.2 kB70.2 kB+76 B (+0.1%)
    auth (GoogleFBTwitterGitHubPopup)90.0 kB90.1 kB+76 B (+0.1%)
    auth (GooglePopup)89.7 kB89.8 kB+76 B (+0.1%)
    auth (GoogleRedirect)89.9 kB90.0 kB+76 B (+0.1%)
    auth (Phone)76.1 kB76.2 kB+76 B (+0.1%)
    database (Append to a list of data)145 kB145 kB+76 B (+0.1%)
    database (Filtering data)144 kB144 kB+76 B (+0.1%)
    database (Listen for child events)160 kB160 kB+76 B (+0.0%)
    database (Listen for value events + Detach listeners)160 kB160 kB+76 B (+0.0%)
    database (Listen for value events)160 kB160 kB+76 B (+0.0%)
    database (Read data once)152 kB152 kB+76 B (+0.1%)
    database (Save data as transactions)162 kB162 kB+76 B (+0.0%)
    database (Sort data)146 kB146 kB+76 B (+0.1%)
    database (Write data)144 kB144 kB+76 B (+0.1%)
    firestore (Persistence)269 kB273 kB+4.81 kB (+1.8%)
    firestore (Query Cursors)208 kB210 kB+2.36 kB (+1.1%)
    firestore (Query)209 kB211 kB+2.36 kB (+1.1%)
    firestore (Read data once)197 kB200 kB+2.35 kB (+1.2%)
    firestore (Realtime updates)200 kB202 kB+2.35 kB (+1.2%)
    firestore (Transaction)181 kB183 kB+2.24 kB (+1.2%)
    firestore (Write data)181 kB183 kB+2.48 kB (+1.4%)
    firestore-lite (Query Cursors)67.8 kB210 kB+142 kB (+210.0%)
    firestore-lite (Query)71.0 kB210 kB+139 kB (+195.8%)
    firestore-lite (Read data once)55.4 kB200 kB+144 kB (+260.7%)
    firestore-lite (Transaction)80.0 kB183 kB+104 kB (+129.5%)
    firestore-lite (Write data)65.2 kB183 kB+118 kB (+181.4%)
    functions (call)29.1 kB29.2 kB+76 B (+0.3%)
    messaging (send + receive)44.7 kB45.0 kB+303 B (+0.7%)
    performance (trace)49.5 kB49.6 kB+76 B (+0.2%)
    remote-config (getAndFetch)43.9 kB44.2 kB+375 B (+0.9%)
    storage (getBytes)37.4 kB37.5 kB+76 B (+0.2%)
    storage (getDownloadURL)39.5 kB39.6 kB+76 B (+0.2%)
    storage (getMetadata)39.0 kB39.0 kB+76 B (+0.2%)
    storage (list + listAll)38.4 kB38.4 kB+76 B (+0.2%)
    storage (updateMetadata)39.2 kB39.3 kB+76 B (+0.2%)
    storage (uploadBytes)43.8 kB43.8 kB+76 B (+0.2%)
    storage (uploadBytesResumable)53.2 kB53.3 kB+76 B (+0.1%)
    storage (uploadString)44.0 kB44.0 kB+76 B (+0.2%)

  • firebase

    15 size changes

    TypeBase (db0dd9f)Merge (3b7e0cf)Diff
    firebase-app-check-compat.js22.7 kB22.8 kB+84 B (+0.4%)
    firebase-app-check.js90.0 kB90.2 kB+175 B (+0.2%)
    firebase-app-compat.js27.6 kB27.7 kB+56 B (+0.2%)
    firebase-app.js87.4 kB87.6 kB+120 B (+0.1%)
    firebase-compat.js787 kB793 kB+5.53 kB (+0.7%)
    firebase-firestore-compat.js309 kB314 kB+4.74 kB (+1.5%)
    firebase-firestore-lite.js267 kB845 kB+578 kB (+216.3%)
    firebase-firestore.js837 kB845 kB+8.06 kB (+1.0%)
    firebase-messaging-compat.js37.5 kB37.9 kB+420 B (+1.1%)
    firebase-messaging-sw.js107 kB107 kB+467 B (+0.4%)
    firebase-messaging.js105 kB106 kB+467 B (+0.4%)
    firebase-performance-standalone-compat.es2017.js87.4 kB87.5 kB+76 B (+0.1%)
    firebase-performance-standalone-compat.js65.3 kB65.3 kB+76 B (+0.1%)
    firebase-remote-config-compat.js27.1 kB27.3 kB+227 B (+0.8%)
    firebase-remote-config.js112 kB113 kB+491 B (+0.4%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/sqOl8VbYUU.html

@tom-andersen tom-andersen marked this pull request as ready for review May 16, 2022 19:18
@google-oss-bot
Copy link
Contributor

google-oss-bot commented May 16, 2022

Size Analysis Report 1

This report is too large (848,661 characters) to be displayed here in a GitHub comment. Please use the below link to see the full report on Google Cloud Storage.

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/2aduLLeGJU.html

Copy link
Contributor

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not review the tests yet, want to give you some feedback first.

ehsannas added a commit that referenced this pull request May 26, 2022
This is porting firebase/firebase-android-sdk#3657 to
the Web SDK. The majority of the code was ported as a part of other PRs such as
#6153.

The only thing remaining is to port the tests in SQLiteLocalStoreTest.java which
will be done as a part of #6261.
@wu-hui wu-hui removed their assignment May 27, 2022
ehsannas added a commit that referenced this pull request May 27, 2022
This is porting firebase/firebase-android-sdk#3657 to
the Web SDK. The majority of the code was ported as a part of other PRs such as
#6153.

The only thing remaining is to port the tests in SQLiteLocalStoreTest.java which
will be done as a part of #6261.
@github-actions
Copy link
Contributor

github-actions bot commented May 30, 2022

Changeset File Check ✅

  • No modified packages are missing from the changeset file.
  • No changeset formatting errors detected.

Copy link
Contributor

@ehsannas ehsannas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can run yarn prettier && yarn lint:fix (in packages/firestore directory) before you run git push so you can catch format/lint issues locally.

Copy link
Contributor

@ehsannas ehsannas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my comments. Pass to Andy for thorough review and approval.

@ehsannas ehsannas removed their assignment Jun 2, 2022
@tom-andersen tom-andersen removed their assignment Jun 6, 2022
Copy link
Contributor

@wu-hui wu-hui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

})
);

// Documents before read time should not be fetched.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this line to below

'coll2'
);
});

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are missing:
testBackfillUsesDocumentKeyOffsetForLargeSnapshots
?

@wu-hui wu-hui assigned tom-andersen and unassigned wu-hui Jun 6, 2022
@tom-andersen tom-andersen assigned wu-hui and unassigned tom-andersen Jun 8, 2022
@wu-hui wu-hui assigned tom-andersen and unassigned wu-hui Jun 8, 2022
@tom-andersen tom-andersen merged commit cec9946 into master Jun 8, 2022
@tom-andersen tom-andersen deleted the tomandersen/index-backfiller branch June 8, 2022 20:50
@firebase firebase locked and limited conversation to collaborators Jul 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants