FR: ability to *only* subscribe to doc/collection changes

[jorroll]

At the moment, there doesn't appear to be a way to only subscribe to the changes of a collection or document without also immediately downloading the entire current collection / the document. E.g. if you already have the latest version of a document downloaded, you might want to just subscribe to changes to that document without re-downloading the current version of the document.

At the moment, methods like QuerySnapshot#docChanges retrieve the entire query, without any apparent way of just getting changes (i.e. on initial subscription, nothing should be returned).

This is a highly desirable feature request from a performance / data consumption standpoint. My goal is to only load documents once, and subsequently load them from the cache while ensuring changes are watched.

Usage might be the following:

• Component calls a personService to retrieve all 'people' where person.age > 18.
• personService checks to see if this query has been run before. If it has not, it uses the firebase-js-sdk to fetch all documents from the 'people' collection where person.age > 18 and it stores these documents in an ngrx/redux store. It returns these documents to the caller.
• On subsequent invocations, personService sees that the query was run before, and just runs the query against the local ngrx/redux store.
• In order to ensure that already downloaded documents / queries are up to date, on first invocation the personService also seperately subscribes to all changes to the 'people' collection (no where filter). This subscription is now maintained for the lifetime of the application. As these changes come in, the personService updates, adds, or removes items from the store as appropriate. This ensures that the client only need to issue a particular query to the backend once, and subsequently can just query data locally. This has major performance implications.

Unfortunately, there does not appear to be any way to accomplish this scenario at the moment because when you subscribe to all changes to the 'people' collection (no where filter), the entire people collection will be returned, which may have thousands of documents.

[REQUIRED] Describe your environment

• Operating System version: Mac OS 10.14
• Browser version: Google Chrome 70
• Firebase SDK version: 5.5.5
• Firebase Product: Firebase

[jorroll]

As pointed out in #1542, the ability to only subscribe to changes since a specific timestamp would be very useful.

Example usage: if you subscribed to "changes starting 5 minutes ago" the query would immediately return a snapshot of the changes that had occurred starting 5 minutes ago up until the present, and then return additional changes as they come in.

Separately, this implementation would allow for "changes polling": maybe it isn't important for your app to always have the most recent version of a collection. Rather than continuously subscribing to all changes to this collection, you simply fetch all documents once (no subscription) and then every 5 minutes request a snapshot of the changes which had occurred since the last request.

This would be an important feature for a rapidly changing collection where it doesn't make sense to subscribe a user to continuous updates.

This may (?) be a separate feature request. I'll include it here for now as it's highly related.

[rsgowman]

Unfortunately, subscribing only to changes isn't currently supported. That said, depending on your use case, the following workaround may be of use.

Example usage: if you subscribed to "changes starting 5 minutes ago" the query would immediately return a snapshot of the changes that had occurred starting 5 minutes ago up until the present, and then return additional changes as they come in.

This can be accomplished with the sdks as they are today. For instance, in your collection, add a 'last_updated_time' field to all your documents. Then you can query for all documents in the collection where the last_updated_time is greater than 5 minutes ago, which would immediately give you exactly those documents, plus provide additional changes as they arrive. If instead of listening on that query, you instead issue a "get", you could implement your "changes polling" by rerunning that every 5 minutes or so. (But this wouldn't result in much/any bandwidth savings, since you'd end up reading all the new/modified documents anyways. Though it might show savings if those documents are modified multiple times within the 5 minute interval.) I think this same approach could be applied to your personService example too.

Would this approach help?

[jorroll]

@rsgowman thanks for the thoughtful response. That's a good suggestion for a workaround! I'll need to explore how far I can get with your suggestion. Off the top of my head, I definitely think that this solution will work in many (most?) scenerios. This being said, given that the firestore only supports querying on a single field with an inequality (>=,<, etc) filter, I can foresee this solution having limitations (as it would eat up the one available inequality filter). I can't actually think of a scenerio where this use case would need two inequality filters though, so maybe it's not an issue.

I'm going to try integrating this idea into my app. I'll update this post if I run into an unforeseen issue but again, very helpful suggestion. Thank you!

[rsgowman]

Your welcome! For now, I'll close this issue. Feel free to re-open if you run into any trouble.

[jorroll]

I've been able to implement @rsgowman's suggestion in my app and I can report that it works well. There are some additional limitations anyone else looking at this solution should keep in mind though:

1. When you subscribe to collection('people').where('updatedAt', '>', now), if a person document is deleted, the client will never see that (technically, the client may see the change if the deleted document was previously updated in the same session, but this will usually not be the case and the client can't count on this).

   • The workaround I came up with for this is to create a deletedDocs collection of the form {path: string, deletedAt: Timestamp}. Clients can subscribe to collection('deletedDocs').where('deletedAt', '>', now) and use that information to remove stale information from their local cache.
   • Unfortunately, you'll need to handle cleanup of the deletedDocs collection. Maybe you can let clients do the cleanup, maybe you'll need some kind of chron job to handle it.

2. When you subscribe to collection('people').where('updatedAt', '>', now), all of the initial changes which come down are of type 'added'. I don't think this is actually a problem for my use case, but it might be for someone else.

If I run into any more issues I'll post them here.