Firestore ArraysTransformTests flake on CI

[ashwinraghav]

It seems frequently the case that a couple of integration tests on Firestore flake out on CI. It is the same couple of tests that fail each time (I think?), but I am not sure if that is just symptomatic of a broader issue.

Example

I0906 21:31:45.702] com.google.firebase.firestore.ArrayTransformsTest > removeFromArrayViaSetMerge[test(AVD) - 8.0.0] FAILED 
I0906 21:31:45.702] 	java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out waiting for Task
I0906 21:31:45.702] 	at com.google.firebase.firestore.testutil.IntegrationTestUtil.waitFor(IntegrationTestUtil.java:285)
I0906 21:32:16.601] 
I0906 21:32:16.602] com.google.firebase.firestore.ArrayTransformsTest > createDocumentWithArrayUnion[test(AVD) - 8.0.0] FAILED 
I0906 21:32:16.602] 	java.lang.RuntimeException: java.util.concurrent.TimeoutException: Timed out waiting for Task
I0906 21:32:16.602] 	at com.google.firebase.firestore.testutil.IntegrationTestUtil.waitFor(IntegrationTestUtil.java:285)

@vkryachko If it seems like it is the nature of these tests (as opposed to the nature of CI) that cause these failures, we could consider @schmidt-sebastian 's approach in firebase-database to retry flaky tests. wdyt?

[bjornick]

I think the issue might be load related on the backend. I suspect we can fix this quickly by just retrying the test a few times like the database tests do.

[ashwinraghav]

Keeping the issue open (for a month?), until we know it does not resurface

[mikelehen]

FYI- I just saw a CI failure (https://k8s-gubernator.appspot.com/build/android-ci/pr-logs/pull/firebase_firebase-android-sdk/19/connected-check/1041755699446550530/) although it was an adb timeout instead of an actual test timeout...?

I0917 18:36:18.748] com.google.firebase.firestore.ArrayTransformsTest > removeFromArrayViaSetMerge[test(AVD) - 8.0.0] FAILED
I0917 18:36:18.748] Test failed to run to completion. Reason: 'Failed to receive adb shell test output within 60000 ms. Test may have timed out, or adb connection to device became unresponsive'. Check device logcat for details
I0917 18:36:18.748] Tests on test(AVD) - 8.0.0 failed: Failed to receive adb shell test output within 60000 ms. Test may have timed out, or adb connection to device became unresponsive

Also, FWIW- We believe we understand the root problem here. There's a "cold start" issue in the backend where if a project hasn't been used in an hour, then the first request could take ~35 seconds to complete. There is an improvement rolling out soon (perhaps Wednesday).

[bjornick]

Is this fixed?

[mikelehen]

#50 should fix it once and for all, but I can't submit it because CI is broken. 😛

[mikelehen]

#50 was submitted. Please holler at me if anybody is still seeing this.