FR: respond to long polling after 29 seconds instead of 30, or make it configurable

[ZYinMD]

This was discussed with @sampajano in #1674.

Background:
OnSnapshot listeners normally use websocket (I think), but if client is behind a proxy that doesn't support websocket, the sdk can use long polling instead. This can be configured with experimentalForceLongPolling and experimentalAutoDetectLongPolling, and the latter may soon be turned on by default.

My observation:
When there's no new change on the doc, after 30 seconds the server will respond "no change" to the long poll.

However, if the proxy happens to be AWS API Gateway, which happens to enforce a timeout of exactly 30 seconds, the poll result will be just a hair too slow to return in time:

The problem:
It's a benign error, but it bombards the console every 30 seconds, and the wording isn't so benign:

Proposal:

• Respond to long polls after 29 seconds instead of 30, for the sake of AWS API Gateway (low hanging fruit)
• or make it configurable (could be over engineering if nobody uses it)

Very easy to reproduce:

// good normal long polling without proxy:
initializeFirestore(app, {
  experimentalForceLongPolling: true, 
});

// with proxy in the middle:
initializeFirestore(app, {
  host: `6i558xin74.execute-api.us-west-1.amazonaws.com`,
  // ↑ my production-ready proxy, everybody feel free to use!!
  experimentalAutoDetectLongPolling: true, 
});

More info:
AWS 30s limit:

More info:
No "server" header, indicating the proxy returned 503 to client before firestore server returned anything to proxy:

[dconeybe]

Hi @ZYinMD. Thanks for the detailed bug report. I'll look into this and see what we can do. I was able to reproduce with your proxy as well, which was super helpful.

[dconeybe]

FYI We have logged internal feature requests for this issue: b/266868871 for webchannel and b/267659448 for the sdk). Someone will reply back once we have an update (probably in the next 2-3 weeks).

[ZYinMD]

Modified original post: slightly improved the code to produce to problem. (after internal tickets creation which might included the original code so FYI)

[sampajano]

Backend and frontend (Javascript) changes for the new configuration option has been submitted.

@dconeybe Could you work on integrating the JS option into the SDK?

Note that this feature will only work after firestore Backend rollout for the change has completed.

[dconeybe]

Update: We need to wait for the next release of https://www.npmjs.com/package/google-closure-library before implementing the feature here in the Firestore SDK. They have a monthly release period so I'll start work on this in the next month.

[dconeybe]

Update: I've opened PR #7176 that adds a new experimentalLongPollingTimeout setting. This is a work-in-progress and my initial testing shows that it doesn't work. There is some backend support that likely still needs to roll out. I'll keep this thread posted with the progress.

[dconeybe]

Update: The fix in PR #7176 is now confirmed to work. I'm just waiting for some internal approvals and then I will get the fix code reviewed and merged. I hope to have this fix released in 3-4 weeks, in mid-late May 2023.

[dconeybe]

Update: The fix has been merged: #7176. It will be included in the next release, which is scheduled for mid-late May 2023.

Here is an example of how to configure the long-polling hanging-get request timeout to 25 seconds (instead of the default of 30 seconds):

const db = initializeFirestore(app, {
  experimentalAutoDetectLongPolling: true,
  experimentalLongPollingOptions: {
    timeoutSeconds: 25
  }
});

[dconeybe]

Update: This feature, the ability to configure the long-polling hanging-get request timeout, has been released on May 12, 2023 in v9.22.0: https://firebase.google.com/support/release-notes/js#version_9220_-_may_12_2023

[ZYinMD]

@dconeybe Thanks very much!

I upgraded to v9.22 and tested it, but it didn't seem to have any effect.

I copied your code which set timeoutSeconds to 25, and I can use "Go to Definition" on timeoutSeconds in my codebase to reveal the new typings added in node_modules, which confirms the upgrade did happen.

I then deployed and saw the polling requests still return 503 after ~31s, which is the same as before.

Is 25 supposed to show up in one or many of the outgoing requests? Either in the request header or the query params? I tried to filter for amazonaws 25 in my Network tab in DevTools, which does a pretty good job searching for these keywords in all urls and headers, but it didn't find anything. Is there a better way I can use to confirm whether the new SDK is sending the right signal to server?

[dconeybe]

Hmm. I'm seeing the same behavior. I'll look into it and get back to you.

Googlers see b/283455589 for internal discussion and updates.

[dconeybe]

Ok I found the root cause. It was a mistake of mine in our release process that caused the actual low-level implementation of the long polling timeout to be missing from the v9.22.0 release. I've opened a PR to fix it #7311 and it will hopefully be included in next week's release, and, if not, in the release after that.

[dconeybe]

Update: The fix was just released: v9.22.1 (May 25, 2023): https://firebase.google.com/support/release-notes/js#version_9221_-_may_25_2023

I've personally verified that it works with both 10 seconds and 20 seconds set as the timeout value. Apologies for the broken initial release. If you have any further issues with the long polling timeout configuration then please open a new issue and feel free to tag me in it.