Skip to content

Various improvements to Bloom Filter work #4986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 8, 2023

Conversation

dconeybe
Copy link
Contributor

@dconeybe dconeybe commented May 7, 2023

This PR is a small selection of improvements to the bloom filter work in #4982.

Improvements include

  • Code formatting in QueryTest.java and TestingHooksUtil.java via ./gradlew googleJavaFormat.
  • Make BloomFilter class final.
  • BloomFilter now takes ByteString instead of byte[] to avoid making unnecessary copies of the byte array; it also models a read-only byte array better than byte[].
  • BloomFilter.create() added to more idiomatically provide the "validation" logic of the bloom filter's parameters.
  • BloomFilter toString() and exception tests made less brittle by only validating a substring of the exception's message, rather than the entire message (Googlers see go/java-testing/exceptions).
  • Simplified the local variable names in tests whose complex names were not required.

@google-oss-bot
Copy link
Contributor

1 Warning
⚠️ Did you forget to add a changelog entry? (Add the 'no-changelog' label to the PR to silence this warning.)

Generated by 🚫 Danger

@google-oss-bot
Copy link
Contributor

Coverage Report 1

Affected Products

  • firebase-firestore

    Overall coverage changed from ? (2573baa) to 44.32% (8628b99) by ?.

    330 individual files with coverage change

    FilenameBase (2573baa)Merge (8628b99)Diff
    AbstractStream.java?34.81%?
    ActivityScope.java?0.00%?
    AggregateField.java?31.03%?
    AggregateQuery.java?22.73%?
    AggregateQuerySnapshot.java?59.52%?
    AggregateSource.java?0.00%?
    AggregationResult.java?0.00%?
    AggregationResultOrBuilder.java?0.00%?
    AggregationResultProto.java?0.00%?
    AndroidConnectivityMonitor.java?39.51%?
    ApiUtil.java?12.50%?
    ArrayContainsAnyFilter.java?100.00%?
    ArrayContainsFilter.java?100.00%?
    ArrayTransformOperation.java?86.49%?
    ArrayValue.java?48.60%?
    ArrayValueOrBuilder.java?0.00%?
    Assert.java?60.00%?
    AsyncEventListener.java?0.00%?
    AsyncQueue.java?70.65%?
    AutoValue_FieldIndex.java?58.33%?
    AutoValue_FieldIndex_IndexOffset.java?58.62%?
    AutoValue_FieldIndex_IndexState.java?59.09%?
    AutoValue_FieldIndex_Segment.java?54.17%?
    AutoValue_IndexEntry.java?34.21%?
    AutoValue_Overlay.java?54.55%?
    AutoValue_TestingHooks_ExistenceFilterBloomFilterInfo.java?20.00%?
    AutoValue_TestingHooks_ExistenceFilterMismatchInfo.java?20.00%?
    BackgroundQueue.java?81.25%?
    BasePath.java?86.54%?
    BatchGetDocumentsRequest.java?0.00%?
    BatchGetDocumentsRequestOrBuilder.java?0.00%?
    BatchGetDocumentsResponse.java?0.00%?
    BatchGetDocumentsResponseOrBuilder.java?0.00%?
    BeginTransactionRequest.java?0.00%?
    BeginTransactionRequestOrBuilder.java?0.00%?
    BeginTransactionResponse.java?0.00%?
    BeginTransactionResponseOrBuilder.java?0.00%?
    BitSequence.java?43.48%?
    BitSequenceOrBuilder.java?0.00%?
    Blob.java?76.92%?
    BloomFilter.java?87.72%?
    BloomFilterOrBuilder.java?0.00%?
    BloomFilterProto.java?0.00%?
    Bound.java?39.29%?
    BundleCache.java?0.00%?
    BundleCallback.java?0.00%?
    BundledDocumentMetadata.java?0.00%?
    BundledDocumentMetadataOrBuilder.java?0.00%?
    BundleDocument.java?90.00%?
    BundledQuery.java?34.57%?
    BundledQueryOrBuilder.java?0.00%?
    BundleElement.java?0.00%?
    BundleElementOrBuilder.java?0.00%?
    BundleLoader.java?100.00%?
    BundleMetadata.java?0.00%?
    BundleMetadataOrBuilder.java?0.00%?
    BundleProto.java?0.00%?
    BundleReader.java?95.29%?
    BundleSerializer.java?89.52%?
    ByteBufferInputStream.java?83.33%?
    CollectionReference.java?13.64%?
    CommitRequest.java?0.00%?
    CommitRequestOrBuilder.java?0.00%?
    CommitResponse.java?0.00%?
    CommitResponseOrBuilder.java?0.00%?
    CommonProto.java?0.00%?
    ComponentProvider.java?100.00%?
    CompositeFilter.java?98.57%?
    ConnectivityMonitor.java?0.00%?
    Consumer.java?0.00%?
    CreateDocumentRequest.java?0.00%?
    CreateDocumentRequestOrBuilder.java?0.00%?
    CredentialsProvider.java?100.00%?
    Cursor.java?35.29%?
    CursorOrBuilder.java?0.00%?
    CustomClassMapper.java?84.25%?
    DatabaseId.java?70.37%?
    DatabaseInfo.java?90.91%?
    Datastore.java?24.55%?
    DatastoreTestTrace.java?0.00%?
    DeleteDocumentRequest.java?0.00%?
    DeleteDocumentRequestOrBuilder.java?0.00%?
    DeleteMutation.java?90.48%?
    DirectionalIndexByteEncoder.java?100.00%?
    Document.java?100.00%?
    DocumentChange.java?67.61%?
    DocumentChangeOrBuilder.java?0.00%?
    DocumentCollections.java?83.33%?
    DocumentDelete.java?34.62%?
    DocumentDeleteOrBuilder.java?0.00%?
    DocumentId.java?0.00%?
    DocumentKey.java?97.14%?
    DocumentMask.java?40.66%?
    DocumentMaskOrBuilder.java?0.00%?
    DocumentOrBuilder.java?0.00%?
    DocumentOverlayCache.java?0.00%?
    DocumentProto.java?0.00%?
    DocumentReference.java?13.24%?
    DocumentRemove.java?29.23%?
    DocumentRemoveOrBuilder.java?0.00%?
    DocumentSet.java?83.78%?
    DocumentSnapshot.java?37.50%?
    DocumentTransform.java?29.31%?
    DocumentTransformOrBuilder.java?0.00%?
    DocumentViewChange.java?90.91%?
    DocumentViewChangeSet.java?87.88%?
    EncodedPath.java?93.22%?
    EventListener.java?0.00%?
    EventManager.java?96.15%?
    Exclude.java?0.00%?
    Executors.java?100.00%?
    ExistenceFilter.java?90.00%?
    ExistenceFilterOrBuilder.java?0.00%?
    ExponentialBackoff.java?30.43%?
    FieldFilter.java?93.06%?
    FieldIndex.java?98.18%?
    FieldMask.java?57.89%?
    FieldPath.java?89.66%?
    FieldTransform.java?64.71%?
    FieldValue.java?82.14%?
    FileUtil.java?0.00%?
    Filter.java?0.00%?
    FirebaseAppCheckTokenProvider.java?93.75%?
    FirebaseAuthCredentialsProvider.java?100.00%?
    FirebaseClientGrpcMetadataProvider.java?44.00%?
    FirebaseFirestore.java?38.33%?
    FirebaseFirestoreException.java?83.72%?
    FirebaseFirestoreSettings.java?71.91%?
    FirestoreCallCredentials.java?18.60%?
    FirestoreChannel.java?18.10%?
    FirestoreClient.java?36.11%?
    FirestoreGrpc.java?2.46%?
    FirestoreIndexValueWriter.java?77.11%?
    FirestoreMultiDbComponent.java?100.00%?
    FirestoreProto.java?0.00%?
    FirestoreRegistrar.java?100.00%?
    Function.java?0.00%?
    GeoPoint.java?91.67%?
    GetDocumentRequest.java?0.00%?
    GetDocumentRequestOrBuilder.java?0.00%?
    GrpcCallProvider.java?69.41%?
    GrpcMetadataProvider.java?0.00%?
    IgnoreExtraProperties.java?0.00%?
    IncomingStreamObserver.java?0.00%?
    Index.java?30.02%?
    IndexBackfiller.java?100.00%?
    IndexByteEncoder.java?88.57%?
    IndexEntry.java?100.00%?
    IndexManager.java?100.00%?
    IndexOrBuilder.java?0.00%?
    IndexProto.java?0.00%?
    InFilter.java?100.00%?
    IntMath.java?41.67%?
    KeyFieldFilter.java?66.67%?
    KeyFieldInFilter.java?0.00%?
    KeyFieldNotInFilter.java?0.00%?
    LimboDocumentChange.java?70.59%?
    ListCollectionIdsRequest.java?0.00%?
    ListCollectionIdsRequestOrBuilder.java?0.00%?
    ListCollectionIdsResponse.java?0.00%?
    ListCollectionIdsResponseOrBuilder.java?0.00%?
    ListDocumentsRequest.java?0.00%?
    ListDocumentsRequestOrBuilder.java?0.00%?
    ListDocumentsResponse.java?0.00%?
    ListDocumentsResponseOrBuilder.java?0.00%?
    Listener.java?0.00%?
    ListenerRegistration.java?0.00%?
    ListenerRegistrationImpl.java?0.00%?
    ListenRequest.java?10.66%?
    ListenRequestOrBuilder.java?0.00%?
    ListenResponse.java?32.64%?
    ListenResponseOrBuilder.java?0.00%?
    ListenSequence.java?100.00%?
    LoadBundleTask.java?25.26%?
    LoadBundleTaskProgress.java?60.47%?
    LocalCacheSettings.java?0.00%?
    LocalDocumentsResult.java?100.00%?
    LocalDocumentsView.java?100.00%?
    LocalSerializer.java?97.35%?
    LocalStore.java?100.00%?
    LocalViewChanges.java?100.00%?
    Logger.java?91.67%?
    LogicUtils.java?96.67%?
    LruDelegate.java?0.00%?
    LruGarbageCollector.java?93.64%?
    MapValue.java?53.00%?
    MapValueOrBuilder.java?0.00%?
    MemoryBundleCache.java?100.00%?
    MemoryCacheSettings.java?50.00%?
    MemoryComponentProvider.java?82.93%?
    MemoryDocumentOverlayCache.java?100.00%?
    MemoryEagerGcSettings.java?28.57%?
    MemoryEagerReferenceDelegate.java?100.00%?
    MemoryGarbageCollectorSettings.java?0.00%?
    MemoryIndexManager.java?68.97%?
    MemoryLruGcSettings.java?0.00%?
    MemoryLruReferenceDelegate.java?96.00%?
    MemoryMutationQueue.java?100.00%?
    MemoryOverlayMigrationManager.java?100.00%?
    MemoryPersistence.java?100.00%?
    MemoryRemoteDocumentCache.java?98.28%?
    MemoryTargetCache.java?100.00%?
    MetadataChanges.java?100.00%?
    MutableDocument.java?100.00%?
    Mutation.java?100.00%?
    MutationBatch.java?89.23%?
    MutationBatchResult.java?100.00%?
    MutationQueue.java?0.00%?
    MutationResult.java?100.00%?
    NamedQuery.java?0.00%?
    NamedQueryOrBuilder.java?0.00%?
    NotInFilter.java?85.71%?
    NumberComparisonHelper.java?100.00%?
    NumericIncrementTransformOperation.java?87.50%?
    ObjectValue.java?99.06%?
    OnlineState.java?100.00%?
    OnlineStateTracker.java?98.11%?
    OnProgressListener.java?0.00%?
    OrderBy.java?96.55%?
    OrderedCodeWriter.java?68.52%?
    Overlay.java?100.00%?
    OverlayedDocument.java?100.00%?
    OverlayMigrationManager.java?0.00%?
    PatchMutation.java?100.00%?
    Persistence.java?100.00%?
    PersistentCacheSettings.java?62.50%?
    Precondition.java?37.82%?
    PreconditionOrBuilder.java?0.00%?
    Preconditions.java?57.89%?
    PropertyName.java?0.00%?
    Query.java?3.96%?
    QueryDocumentSnapshot.java?64.71%?
    QueryEngine.java?98.63%?
    QueryListener.java?100.00%?
    QueryProto.java?0.00%?
    QueryPurpose.java?100.00%?
    QueryResult.java?100.00%?
    QuerySnapshot.java?76.36%?
    QueryView.java?100.00%?
    ReferenceDelegate.java?0.00%?
    ReferenceSet.java?92.45%?
    RemoteDocumentCache.java?0.00%?
    RemoteEvent.java?92.31%?
    RemoteSerializer.java?79.45%?
    RemoteStore.java?88.80%?
    ResourcePath.java?94.74%?
    RollbackRequest.java?0.00%?
    RollbackRequestOrBuilder.java?0.00%?
    RunAggregationQueryRequest.java?0.00%?
    RunAggregationQueryRequestOrBuilder.java?0.00%?
    RunAggregationQueryResponse.java?0.00%?
    RunAggregationQueryResponseOrBuilder.java?0.00%?
    RunQueryRequest.java?0.00%?
    RunQueryRequestOrBuilder.java?0.00%?
    RunQueryResponse.java?0.00%?
    RunQueryResponseOrBuilder.java?0.00%?
    Scheduler.java?0.00%?
    ServerTimestamp.java?0.00%?
    ServerTimestampOperation.java?100.00%?
    ServerTimestamps.java?96.00%?
    SetMutation.java?97.22%?
    SetOptions.java?0.00%?
    SnapshotMetadata.java?68.75%?
    SnapshotVersion.java?87.50%?
    Source.java?0.00%?
    SQLiteBundleCache.java?90.00%?
    SQLiteComponentProvider.java?100.00%?
    SQLiteDocumentOverlayCache.java?97.85%?
    SQLiteIndexManager.java?99.49%?
    SQLiteLruReferenceDelegate.java?98.67%?
    SQLiteMutationQueue.java?82.76%?
    SQLiteOverlayMigrationManager.java?90.48%?
    SQLitePersistence.java?86.34%?
    SQLiteRemoteDocumentCache.java?98.21%?
    SQLiteSchema.java?96.63%?
    SQLiteTargetCache.java?98.47%?
    Stream.java?100.00%?
    StructuredAggregationQuery.java?0.00%?
    StructuredAggregationQueryOrBuilder.java?0.00%?
    StructuredQuery.java?32.54%?
    StructuredQueryOrBuilder.java?0.00%?
    Supplier.java?0.00%?
    SyncEngine.java?93.31%?
    Target.java?95.79%?
    TargetCache.java?0.00%?
    TargetChange.java?80.00%?
    TargetChangeOrBuilder.java?0.00%?
    TargetData.java?77.78%?
    TargetIdGenerator.java?100.00%?
    TargetIndexMatcher.java?100.00%?
    TargetOrBuilder.java?0.00%?
    TargetState.java?97.78%?
    TestingHooks.java?64.52%?
    ThrottledForwardingExecutor.java?100.00%?
    ThrowOnExtraProperties.java?0.00%?
    Timestamp.java?80.39%?
    Token.java?0.00%?
    Transaction.java?0.00%?
    TransactionOptions.java?0.00%?
    TransactionOptionsOrBuilder.java?0.00%?
    TransactionRunner.java?0.00%?
    TransformOperation.java?0.00%?
    UpdateDocumentRequest.java?0.00%?
    UpdateDocumentRequestOrBuilder.java?0.00%?
    User.java?78.57%?
    UserData.java?66.00%?
    UserDataReader.java?70.48%?
    UserDataWriter.java?51.02%?
    Util.java?67.63%?
    Value.java?41.22%?
    ValueOrBuilder.java?0.00%?
    Values.java?95.65%?
    VerifyMutation.java?46.15%?
    View.java?96.82%?
    ViewChange.java?100.00%?
    ViewSnapshot.java?89.23%?
    WatchChange.java?70.51%?
    WatchChangeAggregator.java?98.60%?
    WatchStream.java?19.23%?
    Write.java?36.11%?
    WriteBatch.java?0.00%?
    WriteOrBuilder.java?0.00%?
    WriteProto.java?0.00%?
    WriteRequest.java?10.41%?
    WriteRequestOrBuilder.java?0.00%?
    WriteResponse.java?11.70%?
    WriteResponseOrBuilder.java?0.00%?
    WriteResult.java?0.00%?
    WriteResultOrBuilder.java?0.00%?
    WriteStream.java?31.82%?

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/u3STRwfTZ1.html

@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2023

Unit Test Results

   162 files  +     77     162 suites  +77   2m 47s ⏱️ + 1m 31s
1 158 tests +       3  1 142 ✔️ +       3  16 💤 ±  0  0 ±0 
2 316 runs  +1 151  2 284 ✔️ +1 135  32 💤 +16  0 ±0 

Results for commit 8e8d2d4. ± Comparison against base commit 2573baa.

@google-oss-bot
Copy link
Contributor

Size Report 1

Affected Products

  • base

    TypeBase (2573baa)Merge (8628b99)Diff
    apk (aggressive)?8.39 kB? (?)
    apk (release)?8.65 kB? (?)
  • firebase-firestore

    TypeBase (2573baa)Merge (8628b99)Diff
    aar?1.36 MB? (?)
    apk (aggressive)?520 kB? (?)
    apk (release)?3.95 MB? (?)
  • protolite-well-known-types

    TypeBase (2573baa)Merge (8628b99)Diff
    aar?999 kB? (?)
    apk (aggressive)?134 kB? (?)
    apk (release)?666 kB? (?)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/8fcmsrGW6h.html

@google-oss-bot
Copy link
Contributor

Startup Time Report 1

Note: Layout is sometimes suboptimal due to limited formatting support on GitHub. Please check this report on GCS.

Startup time comparison between the CI merge commit (8628b99) and the base commit (2573baa) are not available.

No macrobenchmark data found for the base commit (2573baa). Analysis for the CI merge commit (8628b99) can be found at:

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/0N9rtTvZNb/index.html

@dconeybe dconeybe changed the title Random improvements to Bloom Filter work Various improvements to Bloom Filter work May 7, 2023
@dconeybe dconeybe requested a review from milaGGL May 7, 2023 16:21
if (bitmap == null) {
throw new NullPointerException("Bitmap cannot be null.");
}
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding the docs.

Copy link
Contributor

@milaGGL milaGGL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

@dconeybe dconeybe merged commit d0849f4 into mila/BloomFilter May 8, 2023
@dconeybe dconeybe deleted the dconeybe/BloomFilterFactoryMethod branch May 8, 2023 14:52
@firebase firebase locked and limited conversation to collaborators Jun 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants