Get Docs currently hanging using pagination cursors #7669

Rterrell25 · 2023-09-30T00:23:02Z

Operating System

Mac OS Ventura 13.5.2

Browser Version

Google Chrome Version 117.0.5938.132

Firebase SDK Version

10.4.0

Firebase SDK Product:

Firestore

Describe your project's tooling

React app using Vite's react template, React version 18.2.0, Firebase web version 9 sdk version 10.4.0

Describe the problem

getDocs() currently hangs when trying to paginate data in firestore using the following query cursors startAfter, endAt, limit, and limitToLast. I am noticing the getDocs() queries will randomly hang and the promises never get resolved. The lastDoc and firstDoc references are always correct when running the queries, it just seems the getDocs() promises hang and never resolve at random. I am currently experiencing this heavily using emulators, but I can only assume this affects production as well.

Steps and code to reproduce issue

Set up a react app template using vite. The below code examples are being used to query a collection in firebase.

Initial fetch always works

  useEffect(() => {
    console.log('this is running')
    const fetchData = async () => {
      let q = query(
        collection(db, 'appointments.requests'),
        orderBy('created'),
        limit(rowsPerPage + 1)
      )

      const querySnapshot = await getDocs(q)

      const initialData = querySnapshot.docs.map((doc) => ({
        id: doc.id,
        ...doc.data()
      }))

      setFirstVisibleDoc(querySnapshot.docs[0])

      if (querySnapshot.docs.length <= rowsPerPage) {
        setHasMore(false)
        setLastVisibleDoc(querySnapshot.docs[querySnapshot.docs.length - 1])
      } else {
        setHasMore(true)
        setLastVisibleDoc(querySnapshot.docs[querySnapshot.docs.length - 2])
      }

      setRows(initialData.slice(0, rowsPerPage))
      setLoading(false)
    }

    fetchData()
  }, [rowsPerPage])

Querying next page and previous page will randomly hang at the line where getDocs(q) is being called as I go to nextPage or previousPage.

  const getNextPage = async () => {
    const q = query(
      collection(db, 'appointments.requests'),
      orderBy('created'),
      startAfter(lastVisibleDoc),
      limit(rowsPerPage + 1)
    )
    console.log(q)
    console.log(lastVisibleDoc)
    let querySnapshot
    try {
      console.log('in try catch next')
      querySnapshot = await getDocs(q)
    } catch (error) {
      console.log('error next', error)
    }
    console.log('getNextSnapshot', querySnapshot)

    const allData = querySnapshot.docs.map((doc) => ({
      id: doc.id,
      ...doc.data()
    }))

    setFirstVisibleDoc(querySnapshot.docs[0])
    if (allData.length <= rowsPerPage) {
      setHasMore(false)
      setLastVisibleDoc(querySnapshot.docs[querySnapshot.docs.length - 1])
    } else {
      setHasMore(true)
      setLastVisibleDoc(querySnapshot.docs[querySnapshot.docs.length - 2])
    }

    setRows(allData.slice(0, rowsPerPage))
    setPage((prevState) => prevState + 1)
  }

  const getPreviousPage = async () => {
    const q = query(
      collection(db, 'appointments.requests'),
      orderBy('created'),
      endAt(firstVisibleDoc),
      limitToLast(rowsPerPage + 1)
    )
    console.log(q)
    console.log(firstVisibleDoc)

    let querySnapshot
    try {
      console.log('in try catch prev')
      querySnapshot = await getDocs(q)
    } catch (error) {
      console.log('error prev', error)
    }

    console.log('getPreviousSnapshot', querySnapshot)
    const allData = querySnapshot.docs.map((doc) => ({
      id: doc.id,
      ...doc.data()
    }))

    setFirstVisibleDoc(querySnapshot.docs[0])

    setLastVisibleDoc(querySnapshot.docs[querySnapshot.docs.length - 2])

    setRows(allData.slice(0, rowsPerPage))
    setPage((prevState) => prevState - 1)
    setHasMore(true)
  }

Very hard to diagnose as the try catches do absolutely nothing in terms of error handling.

The text was updated successfully, but these errors were encountered:

rob-yeshearing · 2023-10-06T15:34:06Z

Is there any resolution here....

MarkDuckworth · 2023-10-10T12:07:46Z

I'm working on a reproduction. Thanks for your patience on this.

RobertJewell · 2023-10-11T18:26:07Z

Confirming I'm seeing the same behaviour on the v10 sdk.

I can reliably recreate it by throttling the browser connection speed and calling getDocs with a "where ==" clause and an orderby.

I'm seeing it in the initial request.

It happens using the local emulator and cloud firestore.

Edit - I've only tested this using NextJS 13

MarkDuckworth · 2023-10-11T19:25:56Z

Is anyone reproducing this on a non-React app? I was testing today using v10, but without React, and I couldn't reproduce.

rob-yeshearing · 2023-10-11T21:12:35Z

Confirming I'm seeing the same behaviour on both the v9 and v10 sdk.

I can reliably recreate it by throttling the browser connection speed and calling getDocs with a "where ==" clause and an orderby.

I'm seeing it in the initial request.

It happens using the local emulator and cloud firestore.

Have begun to notice this will happen with initial request as well.

RobertJewell · 2023-10-12T10:33:08Z

Some more details now i've tested with a new project on a live database

@Rterrell25 @rob-yeshearing

TLDR - Downgrading to sdk version 9.1.0. seems to fix the issue.

Operating System
Mac OS Monterey 12.6.5

Browser Version
Arc Version Version 1.11.0

Firebase SDK Version
10.4.0

Firebase SDK Product:
Firestore

Describe your project's tooling
NextJS 13.5 (pages and app directory), React version 18.2.0, Firebase web version 9 sdk version 10.4.0.

The QuerySnapshot returned by getDocs on a slow connection often never resolves.
It does sometimes work, but the cases of it not resolving increase as the connection slows.

I've tested by setting up a new project with one page that runs getDocs on page load and logs the returned data, it's about as simple as I could make it.

In both this test project and the more complex production app - downgrading to sdk version 9.1.0. seems to fix the issue.
There are likely versions in between 9.1.0 and 10.4.0 that also work, I haven't tested that.

Currently everyone reporting this issue seems to be called Robert - unlikely that's related though.

MarkDuckworth · 2023-10-12T19:20:57Z

@RobertJewell, can you test with 10.2.x? There was a change made to the transport in 10.3, which could be affecting you.

RobertJewell · 2023-10-12T23:08:51Z

@MarkDuckworth im away from my machine for a few days so I won't be much help for a bit.

Before I left I tested with 10.1.0 and that worked correctly.

rob-yeshearing · 2023-10-16T21:17:55Z

@MarkDuckworth any luck with a solution going forward on this? Do we still need someone to test 10.2.x and see if that's affected?

MarkDuckworth · 2023-10-16T23:10:43Z

@rob-yeshearing, I've had a few other higher priority items on my plate, but I'm freeing up from that to re-visit this.

I have a hunch that this is related to a change in v10.3, but I haven't proven it. If you do have a chance to test with v10.2, that may give guidance to other readers.

MarkDuckworth · 2023-10-19T23:26:37Z

I built a test react app based on the code from @Rterrell25, however I'm still not able to reproduce the issue locally. We have had some other similar reported issues, #7652 and #7641. For #7652 I could only repro with the customer code and data, not with my own data. Could be the same case here.

If anyone reporting in this thread can provide Firestore logs by setting setLogLevel('debug') and providing us with the output, that will help determine if this is the same or a different issue. Thanks.

rob-yeshearing · 2023-10-20T03:37:42Z

@MarkDuckworth When I have a chance I will report the logs that are returned when setting the log level to debug in firestore. I will also test 10.2.x and see if the bug is still present in 10.2.x

rob-yeshearing · 2023-10-20T14:05:41Z

@MarkDuckworth here are the logs

started here
logger.ts:115 [2023-10-20T13:59:19.896Z]  @firebase/firestore: Firestore (10.4.0): MemoryPersistence Starting transaction: Allocate target
logger.ts:115 [2023-10-20T13:59:19.897Z]  @firebase/firestore: Firestore (10.4.0): MemoryPersistence Starting transaction: Execute query
logger.ts:115 [2023-10-20T13:59:19.897Z]  @firebase/firestore: Firestore (10.4.0): QueryEngine Using full collection scan to execute query: Query(target=Target(appointments.requests, limit: 6, orderBy: [created (asc), __name__ (asc)], startAt: a:time(1697807969,881000000),appointments.requests/TBrmWAptImwcDrQJhyib); limitType=F)
logger.ts:115 [2023-10-20T13:59:19.897Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab sending: {"database":"projects/ys-portal-8e373/databases/(default)","addTarget":{"query":{"structuredQuery":{"from":[{"collectionId":"appointments.requests"}],"orderBy":[{"field":{"fieldPath":"created"},"direction":"ASCENDING"},{"field":{"fieldPath":"__name__"},"direction":"ASCENDING"}],"limit":6,"startAt":{"before":false,"values":[{"timestampValue":"2023-10-20T13:19:29.881Z"},{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/TBrmWAptImwcDrQJhyib"}]}},"parent":"projects/ys-portal-8e373/databases/(default)/documents"},"targetId":16}}
logger.ts:115 [2023-10-20T13:59:19.907Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"targetChange":{"targetChangeType":"ADD","targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.908Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"targetChange":{"targetChangeType":"ADD","targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.908Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"documentChange":{"document":{"name":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/OdC2L9KcVWjn3vrv6KAf","fields":{"owner":{"mapValue":{"fields":{"firstName":{"stringValue":"Roger"},"lastName":{"stringValue":"Pucci"}}}},"createdBy":{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/users/vZTWsxVXGsLA0Dg1aXKTj5UvucIx"},"patient":{"mapValue":{"fields":{"firstName":{"stringValue":"Roger"},"lastName":{"stringValue":"Pucci"},"emailAddress":{"stringValue":"[email protected]"},"phoneNumber":{"stringValue":"(609) 650-8297"}}}},"lastPriorityOrder":{"integerValue":"0"},"created":{"timestampValue":"2023-10-20T13:19:29.888Z"},"salesforceId":{"stringValue":"SFAKA1FMWPE3TQK"},"status":{"stringValue":"CANCELLED"}},"createTime":"2023-10-20T13:19:29.893907Z","updateTime":"2023-10-20T13:19:29.893907Z"},"targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.909Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"documentChange":{"document":{"name":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/jJpokU5fqqpZLCnfaVsT","fields":{"owner":{"mapValue":{"fields":{"firstName":{"stringValue":"Marie"},"lastName":{"stringValue":"Naldini"}}}},"createdBy":{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/users/rviiS8zbxX3kDN9q7IUPhb0znP9b"},"patient":{"mapValue":{"fields":{"firstName":{"stringValue":"Marie"},"lastName":{"stringValue":"Naldini"},"emailAddress":{"stringValue":"[email protected]"},"phoneNumber":{"stringValue":"(567) 576-4269"}}}},"lastPriorityOrder":{"integerValue":"0"},"created":{"timestampValue":"2023-10-20T13:19:29.896Z"},"salesforceId":{"stringValue":"SFVSGADY9A2OYL9"},"status":{"stringValue":"SCHEDULED"}},"createTime":"2023-10-20T13:19:29.900762Z","updateTime":"2023-10-20T13:19:29.900762Z"},"targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.909Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"documentChange":{"document":{"name":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/5X0lhELub6IdrzM324JZ","fields":{"owner":{"mapValue":{"fields":{"firstName":{"stringValue":"Ryan"},"lastName":{"stringValue":"Charpentier"}}}},"createdBy":{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/users/O4lKWSvSFyATWi0yNP6mQUpFcgVc"},"patient":{"mapValue":{"fields":{"firstName":{"stringValue":"Ryan"},"lastName":{"stringValue":"Charpentier"},"emailAddress":{"stringValue":"[email protected]"},"phoneNumber":{"stringValue":"(543) 993-9917"}}}},"lastPriorityOrder":{"integerValue":"0"},"created":{"timestampValue":"2023-10-20T13:19:29.903Z"},"salesforceId":{"stringValue":"SF080WP6NFYMCMX"},"status":{"stringValue":"NEW"}},"createTime":"2023-10-20T13:19:29.907537Z","updateTime":"2023-10-20T13:19:29.907537Z"},"targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.909Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"documentChange":{"document":{"name":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/lfpd9B4eMuQDQYvCAtqu","fields":{"owner":{"mapValue":{"fields":{"firstName":{"stringValue":"Matthew"},"lastName":{"stringValue":"Pearce"}}}},"createdBy":{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/users/K9CLCZVPnxOHKiUfLxDPq1rJUfp3"},"patient":{"mapValue":{"fields":{"firstName":{"stringValue":"Matthew"},"lastName":{"stringValue":"Pearce"},"emailAddress":{"stringValue":"[email protected]"},"phoneNumber":{"stringValue":"(707) 885-8972"}}}},"lastPriorityOrder":{"integerValue":"0"},"created":{"timestampValue":"2023-10-20T13:19:29.910Z"},"salesforceId":{"stringValue":"SF2IP3MP1EXKNMX"},"status":{"stringValue":"SCHEDULED"}},"createTime":"2023-10-20T13:19:29.915593Z","updateTime":"2023-10-20T13:19:29.915593Z"},"targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.909Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"documentChange":{"document":{"name":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/wfJeOvr7ua3Px1SGhqza","fields":{"owner":{"mapValue":{"fields":{"firstName":{"stringValue":"Lester"},"lastName":{"stringValue":"Santoni"}}}},"createdBy":{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/users/LxvUPs92UeIhFZxoKiX7Ox2GgCrV"},"patient":{"mapValue":{"fields":{"firstName":{"stringValue":"Lester"},"lastName":{"stringValue":"Santoni"},"emailAddress":{"stringValue":"[email protected]"},"phoneNumber":{"stringValue":"(638) 527-2183"}}}},"lastPriorityOrder":{"integerValue":"0"},"created":{"timestampValue":"2023-10-20T13:19:29.918Z"},"salesforceId":{"stringValue":"SF5XDWAO80ZYSM0"},"status":{"stringValue":"NEW"}},"createTime":"2023-10-20T13:19:29.923229Z","updateTime":"2023-10-20T13:19:29.923229Z"},"targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.909Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"documentChange":{"document":{"name":"projects/ys-portal-8e373/databases/(default)/documents/appointments.requests/gq6rNjDGlxUTuHRiYruB","fields":{"owner":{"mapValue":{"fields":{"firstName":{"stringValue":"Eugene"},"lastName":{"stringValue":"Webb"}}}},"createdBy":{"referenceValue":"projects/ys-portal-8e373/databases/(default)/documents/users/zUb3Foj5HpT5bY6N6InkNCrBKqgk"},"patient":{"mapValue":{"fields":{"firstName":{"stringValue":"Eugene"},"lastName":{"stringValue":"Webb"},"emailAddress":{"stringValue":"[email protected]"},"phoneNumber":{"stringValue":"(363) 391-5831"}}}},"lastPriorityOrder":{"integerValue":"0"},"created":{"timestampValue":"2023-10-20T13:19:29.925Z"},"salesforceId":{"stringValue":"SFXLOGURZSNIRW1"},"status":{"stringValue":"NEW"}},"createTime":"2023-10-20T13:19:29.929783Z","updateTime":"2023-10-20T13:19:29.929783Z"},"targetIds":[16]}}
logger.ts:115 [2023-10-20T13:59:19.909Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"targetChange":{"targetChangeType":"CURRENT","targetIds":[16],"resumeToken":"CgkIxOKXu+SEggM=","readTime":"2023-10-20T13:59:19.906628Z"}}
logger.ts:115 [2023-10-20T13:59:19.910Z]  @firebase/firestore: Firestore (10.4.0): WebChannelConnection RPC 'Listen' stream 0x721469ab received: {"targetChange":{"resumeToken":"CgkIp+OXu+SEggM=","readTime":"2023-10-20T13:59:19.906727Z"}}
logger.ts:115 [2023-10-20T13:59:19.910Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.910Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.910Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.910Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.910Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.911Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.911Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.911Z]  @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target 16
logger.ts:115 [2023-10-20T13:59:19.911Z]  @firebase/firestore: Firestore (10.4.0): MemoryPersistence Starting transaction: Get last remote snapshot version
logger.ts:115 [2023-10-20T13:59:19.911Z]  @firebase/firestore: Firestore (10.4.0): MemoryPersistence Starting transaction: Apply remote event
logger.ts:115 [2023-10-20T13:59:19.911Z]  @firebase/firestore: Firestore (10.4.0): MemoryPersistence Starting transaction: notifyLocalViewChanges

I'm noticing similar to the other issues you've cited, [2023-10-20T13:59:19.910Z] @firebase/firestore: Firestore (10.4.0): WatchChangeAggregator Detected inactive target appears when the query fails to get more docs.

MarkDuckworth · 2023-10-27T22:36:09Z

@Rterrell25 and all, we released a fix for hanging queries so you can now use v10.5.2. Thanks for your help identifying this problem!

rob-yeshearing · 2023-10-30T14:43:25Z

@MarkDuckworth Thank you!

Rterrell25 added new A new issue that hasn't be categoirzed as question, bug or feature request question labels Sep 30, 2023

google-oss-bot added the api: firestore label Sep 30, 2023

Rterrell25 changed the title ~~Title for the bug~~ Get Docs currently hanging using pagination cursors Sep 30, 2023

jbalidiong added needs-attention and removed new A new issue that hasn't be categoirzed as question, bug or feature request labels Oct 2, 2023

MarkDuckworth self-assigned this Oct 3, 2023

MarkDuckworth closed this as completed Oct 27, 2023

firebase locked and limited conversation to collaborators Nov 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get Docs currently hanging using pagination cursors #7669

Get Docs currently hanging using pagination cursors #7669

Rterrell25 commented Sep 30, 2023 •

edited

Loading

rob-yeshearing commented Oct 6, 2023

MarkDuckworth commented Oct 10, 2023

RobertJewell commented Oct 11, 2023 •

edited

Loading

MarkDuckworth commented Oct 11, 2023

rob-yeshearing commented Oct 11, 2023

RobertJewell commented Oct 12, 2023 •

edited

Loading

MarkDuckworth commented Oct 12, 2023

RobertJewell commented Oct 12, 2023

rob-yeshearing commented Oct 16, 2023

MarkDuckworth commented Oct 16, 2023

MarkDuckworth commented Oct 19, 2023

rob-yeshearing commented Oct 20, 2023 •

edited

Loading

rob-yeshearing commented Oct 20, 2023

MarkDuckworth commented Oct 27, 2023

rob-yeshearing commented Oct 30, 2023

Get Docs currently hanging using pagination cursors #7669

Get Docs currently hanging using pagination cursors #7669

Comments

Rterrell25 commented Sep 30, 2023 • edited Loading

Operating System

Browser Version

Firebase SDK Version

Firebase SDK Product:

Describe your project's tooling

Describe the problem

Steps and code to reproduce issue

rob-yeshearing commented Oct 6, 2023

MarkDuckworth commented Oct 10, 2023

RobertJewell commented Oct 11, 2023 • edited Loading

MarkDuckworth commented Oct 11, 2023

rob-yeshearing commented Oct 11, 2023

RobertJewell commented Oct 12, 2023 • edited Loading

TLDR - Downgrading to sdk version 9.1.0. seems to fix the issue.

MarkDuckworth commented Oct 12, 2023

RobertJewell commented Oct 12, 2023

rob-yeshearing commented Oct 16, 2023

MarkDuckworth commented Oct 16, 2023

MarkDuckworth commented Oct 19, 2023

rob-yeshearing commented Oct 20, 2023 • edited Loading

rob-yeshearing commented Oct 20, 2023

MarkDuckworth commented Oct 27, 2023

rob-yeshearing commented Oct 30, 2023

Rterrell25 commented Sep 30, 2023 •

edited

Loading

RobertJewell commented Oct 11, 2023 •

edited

Loading

RobertJewell commented Oct 12, 2023 •

edited

Loading

rob-yeshearing commented Oct 20, 2023 •

edited

Loading