Skip to content

Online Count with tentative package private api #3847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 51 commits into from
Aug 29, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
12fc017
Update protos to googleapis HEAD at https://github.com/googleapis/goo…
dconeybe Jun 2, 2022
3f3a823
write.proto: add back string verify = 5
dconeybe Jun 2, 2022
d8dab8e
Update protos to googleapis HEAD of the preview branch at https://git…
dconeybe Jun 2, 2022
3ffc859
write.proto: add back string verify = 5
dconeybe Jun 2, 2022
f5b5ec4
Add API skeleton to aggregate queries and COUNT
dconeybe Jun 3, 2022
3112d79
Datastore.java: runCountQuery() added
dconeybe Jun 3, 2022
728f4cb
wire up count queries
dconeybe Jun 3, 2022
ec2be89
delete group-by stuff, since it's not needed for this experiment
dconeybe Jun 3, 2022
3f2a539
CountTest.java added
dconeybe Jun 4, 2022
dccef12
Revert changes to Firestore.kt, since this prototype only uses java
dconeybe Jun 4, 2022
20b6f61
AggregateField.java: remove everything except COUNT
dconeybe Jun 4, 2022
4eee46c
AggregateSnapshot.java deleted, since it's not used in this prototype
dconeybe Jun 4, 2022
41ce0a8
revert a bunch of other unnecssary changes
dconeybe Jun 4, 2022
931c8a6
Collapse CountQuery.java into AggregateQuery.java for simplicity
dconeybe Jun 4, 2022
57aa89b
/gradlew :firebase-firestore:googleJavaFormat
dconeybe Jun 4, 2022
e95066c
AggregateField.java: remove CountAggregateField.upTo, since it's neve…
dconeybe Jun 4, 2022
20a7e10
Update protos to googleapis HEAD at https://github.com/googleapis/goo…
dconeybe Jun 2, 2022
92c5218
write.proto: add back string verify = 5
dconeybe Jun 2, 2022
5130d16
Update protos to googleapis HEAD of the preview branch at https://git…
dconeybe Jun 2, 2022
9c3a336
write.proto: add back string verify = 5
dconeybe Jun 2, 2022
74e6d14
Add API skeleton to aggregate queries and COUNT
dconeybe Jun 3, 2022
4155ac6
Datastore.java: runCountQuery() added
dconeybe Jun 3, 2022
6b34e44
wire up count queries
dconeybe Jun 3, 2022
8dded2c
delete group-by stuff, since it's not needed for this experiment
dconeybe Jun 3, 2022
12b5cd1
CountTest.java added
dconeybe Jun 4, 2022
77beb18
Revert changes to Firestore.kt, since this prototype only uses java
dconeybe Jun 4, 2022
bcf0bcf
AggregateField.java: remove everything except COUNT
dconeybe Jun 4, 2022
9aef43a
AggregateSnapshot.java deleted, since it's not used in this prototype
dconeybe Jun 4, 2022
01420e4
revert a bunch of other unnecssary changes
dconeybe Jun 4, 2022
965144b
Collapse CountQuery.java into AggregateQuery.java for simplicity
dconeybe Jun 4, 2022
4aa3ceb
/gradlew :firebase-firestore:googleJavaFormat
dconeybe Jun 4, 2022
df05424
AggregateField.java: remove CountAggregateField.upTo, since it's neve…
dconeybe Jun 4, 2022
09685a4
Merge remote-tracking branch 'origin/dconeybe/CountFromDFE' into dcon…
wu-hui Jun 20, 2022
8d640fe
Temp checkin
wu-hui Jun 28, 2022
af28b4c
It runs now
wu-hui Jul 7, 2022
e748b6a
Format and lint
wu-hui Jul 7, 2022
3efce02
Undo change
wu-hui Jul 7, 2022
d4ced5e
Merge branch 'master' into wuandy/CountFromDFE
wu-hui Jul 8, 2022
8d98662
Public doc and changelog
wu-hui Jul 8, 2022
4d7da5a
api.txt
wu-hui Jul 8, 2022
c33a960
Support network change and address feedback
wu-hui Jul 13, 2022
e33fb81
Address more feedback
wu-hui Jul 18, 2022
ca9387d
Merge branch 'master' into wuandy/CountFromDFE
wu-hui Jul 18, 2022
89dc144
More
wu-hui Jul 20, 2022
2dff823
More feedback
wu-hui Jul 27, 2022
d6e7098
Remove retry
wu-hui Aug 15, 2022
419f3fe
Fix CI
wu-hui Aug 15, 2022
ae618b4
Hide this feature for now
wu-hui Aug 22, 2022
3f3cb57
Update API
wu-hui Aug 22, 2022
2dc20b5
Only run tests with emulator
wu-hui Aug 22, 2022
7f090e2
More feedback
wu-hui Aug 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@
import static com.google.firebase.firestore.testutil.IntegrationTestUtil.testCollectionWithDocs;
import static com.google.firebase.firestore.testutil.IntegrationTestUtil.testFirestore;
import static com.google.firebase.firestore.testutil.IntegrationTestUtil.waitFor;
import static com.google.firebase.firestore.testutil.IntegrationTestUtil.waitForException;
import static com.google.firebase.firestore.testutil.TestUtil.map;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;

import androidx.test.ext.junit.runners.AndroidJUnit4;
import com.google.firebase.firestore.testutil.IntegrationTestUtil;
Expand Down Expand Up @@ -108,4 +110,25 @@ public void testCanRunCountOnNonExistentCollection() {
waitFor(collection.whereEqualTo("k", 100).count().get(AggregateSource.SERVER_DIRECT));
assertEquals(Long.valueOf(0), snapshot.getCount());
}

@Test
public void testFailWithoutNetwork() {
CollectionReference collection =
testCollectionWithDocs(
map(
"a", map("k", "a"),
"b", map("k", "b"),
"c", map("k", "c")));
waitFor(collection.getFirestore().disableNetwork());

Exception e = waitForException(collection.count().get(AggregateSource.SERVER_DIRECT));
assertTrue(e instanceof FirebaseFirestoreException);
assertEquals(
FirebaseFirestoreException.Code.UNAVAILABLE, ((FirebaseFirestoreException) e).getCode());

waitFor(collection.getFirestore().enableNetwork());
AggregateQuerySnapshot snapshot =
waitFor(collection.count().get(AggregateSource.SERVER_DIRECT));
assertEquals(Long.valueOf(3), snapshot.getCount());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import com.google.android.gms.tasks.Task;
import com.google.android.gms.tasks.TaskCompletionSource;
import com.google.firebase.firestore.util.Executors;
import java.util.Objects;

/**
* A {@code AggregateQuery} computes some aggregation statistics from the result set of a base
Expand Down Expand Up @@ -79,7 +78,7 @@ public boolean equals(Object o) {

@Override
public int hashCode() {
return Objects.hash(query);
return query.hashCode();
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@
import androidx.annotation.Nullable;
import java.util.Objects;

/**
* A {@code AggregateQuerySnapshot} contains results of a {@link AggregateQuery}.
*
* <p><b>Subclassing Note</b>: Cloud Firestore classes are not meant to be subclassed except for use
* in test mocks. Subclassing is not supported in production code and new SDK releases may break
* code that does so.
*/
public class AggregateQuerySnapshot {

private final long count;
Expand All @@ -25,6 +32,10 @@ public class AggregateQuerySnapshot {
this.count = count;
}

/**
* @return The result of a document count aggregation. Returns null if no count aggregation is
* available in the result.
*/
@Nullable
public Long getCount() {
return count;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@

package com.google.firebase.firestore.core;

import static com.google.firebase.firestore.util.Util.isRetryableBackendError;

import com.google.android.gms.tasks.OnCompleteListener;
import com.google.android.gms.tasks.Task;
import com.google.android.gms.tasks.TaskCompletionSource;
Expand All @@ -31,7 +33,7 @@ public class OnlineQueryRunner {
private TaskCompletionSource<Long> taskSource = new TaskCompletionSource<>();

public OnlineQueryRunner(AsyncQueue asyncQueue, RemoteStore remoteStore) {
backoff = new ExponentialBackoff(asyncQueue, AsyncQueue.TimerId.RETRY_TRANSACTION);
backoff = new ExponentialBackoff(asyncQueue, AsyncQueue.TimerId.RETRY_ONLINE_QUERY);
this.asyncQueue = asyncQueue;
this.remoteStore = remoteStore;
}
Expand Down Expand Up @@ -61,7 +63,7 @@ private void runWithBackoff(Query query) {
}

private void handleError(Task<Long> result, Query query) {
if (attemptsRemaining > 0) {
if (attemptsRemaining > 0 && isRetryableBackendError(result.getException())) {
runWithBackoff(query);
} else {
taskSource.setException(result.getException());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@

package com.google.firebase.firestore.core;

import static com.google.firebase.firestore.util.Util.isRetryableBackendError;

import androidx.annotation.NonNull;
import com.google.android.gms.tasks.Task;
import com.google.android.gms.tasks.TaskCompletionSource;
import com.google.firebase.firestore.FirebaseFirestoreException;
import com.google.firebase.firestore.TransactionOptions;
import com.google.firebase.firestore.remote.Datastore;
import com.google.firebase.firestore.remote.RemoteStore;
import com.google.firebase.firestore.util.AsyncQueue;
import com.google.firebase.firestore.util.AsyncQueue.TimerId;
Expand Down Expand Up @@ -86,22 +86,10 @@ private void runWithBackoff() {
}

private void handleTransactionError(Task task) {
if (attemptsRemaining > 0 && isRetryableTransactionError(task.getException())) {
if (attemptsRemaining > 0 && isRetryableBackendError(task.getException())) {
runWithBackoff();
} else {
taskSource.setException(task.getException());
}
}

private static boolean isRetryableTransactionError(Exception e) {
if (e instanceof FirebaseFirestoreException) {
// In transactions, the backend will fail outdated reads with FAILED_PRECONDITION and
// non-matching document versions with ABORTED. These errors should be retried.
FirebaseFirestoreException.Code code = ((FirebaseFirestoreException) e).getCode();
return code == FirebaseFirestoreException.Code.ABORTED
|| code == FirebaseFirestoreException.Code.FAILED_PRECONDITION
|| !Datastore.isPermanentError(((FirebaseFirestoreException) e).getCode());
}
return false;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@
import androidx.annotation.Nullable;
import androidx.annotation.VisibleForTesting;
import com.google.android.gms.tasks.Task;
import com.google.android.gms.tasks.Tasks;
import com.google.firebase.database.collection.ImmutableSortedSet;
import com.google.firebase.firestore.FirebaseFirestoreException;
import com.google.firebase.firestore.core.OnlineState;
import com.google.firebase.firestore.core.Query;
import com.google.firebase.firestore.core.Transaction;
Expand Down Expand Up @@ -52,7 +54,7 @@
* RemoteStore handles all interaction with the backend through a simple, clean interface. This
* class is not thread safe and should be only called from the worker AsyncQueue.
*/
public final class RemoteStore<TResult> implements WatchChangeAggregator.TargetMetadataProvider {
public final class RemoteStore implements WatchChangeAggregator.TargetMetadataProvider {

/** The maximum number of pending writes to allow. TODO: Negotiate this value with the backend. */
private static final int MAX_PENDING_WRITES = 10;
Expand Down Expand Up @@ -751,6 +753,12 @@ public TargetData getTargetDataForTarget(int targetId) {
}

public Task<Long> runCountQuery(Query query) {
return datastore.runCountQuery(query);
if (canUseNetwork()) {
return datastore.runCountQuery(query);
}

return Tasks.forException(
new FirebaseFirestoreException(
"Failed to get result from server.", FirebaseFirestoreException.Code.UNAVAILABLE));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,11 @@ public enum TimerId {
* multiple of these may be in the queue at a given time.
*/
RETRY_TRANSACTION,
/**
* A timer used to retry queries sent to DFE. Since there can be multiple active queries,
* multiple of these may be in the queue at a given time.
*/
RETRY_ONLINE_QUERY,
/**
* A timer used to monitor when a connection attempt in gRPC is unsuccessful and retry
* accordingly.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.google.firebase.firestore.FieldPath;
import com.google.firebase.firestore.FirebaseFirestoreException;
import com.google.firebase.firestore.FirebaseFirestoreException.Code;
import com.google.firebase.firestore.remote.Datastore;
import com.google.protobuf.ByteString;
import io.grpc.Status;
import io.grpc.StatusException;
Expand Down Expand Up @@ -137,6 +138,18 @@ public static Exception convertThrowableToException(Throwable t) {
}
}

public static boolean isRetryableBackendError(Exception e) {
if (e instanceof FirebaseFirestoreException) {
// In transactions, the backend will fail outdated reads with FAILED_PRECONDITION and
// non-matching document versions with ABORTED. These errors should be retried.
FirebaseFirestoreException.Code code = ((FirebaseFirestoreException) e).getCode();
return code == FirebaseFirestoreException.Code.ABORTED
|| code == FirebaseFirestoreException.Code.FAILED_PRECONDITION
|| !Datastore.isPermanentError(((FirebaseFirestoreException) e).getCode());
}
return false;
}

private static final Continuation<Void, Void> VOID_ERROR_TRANSFORMER =
task -> {
if (task.isSuccessful()) {
Expand Down