[DISCUSSION] Deep JSON key fields filter

[u8sand]

Description

A working hack of this feature to illuminate what needs to be done. I'm open to suggestions for things I should improve but wanted to put this here for anyone interested in the feature. This is not meant to be merged in its current state.

Discussion

Supporting filtering by json fields should be possible with the postgresql connector. That-is, a find that looks like so:

{
  "where": {
    "address.city": "Hill Valley",
  },
  "fields": ["address.street.number"],
}

Should result in

{
  "address": {
    "street": {
      "number": 56
    }
  }
}

A number of issues make this difficult, not least of all because strongloop/loopback-datasource-juggler pre-filters unrecognizable fields in the query. A workaround is to alter the strict setting on the model definition; see https://github.com/strongloop/loopback-connector-postgresql/blob/e58ffccc58232d50e01620aa54951f398754c317/test/postgresql.test.js#L692-L702 for full discussion.

Once we get the fields passed the juggler, we can deal with the filters in the buildColumnNames override. The same code used in #304 to create the where clause can be used for the select clause, but unfortunately the postgresql output becomes no longer identifiable.

 ?column? 
----------
 56

To mitigate this, we can use an as on nested fields, and because PostgreSQL doesn't like . for names, we can replace those with __ among other things.

 address__street__number 
-------------------------
 56

To capture this we need to override fromRow; and construct a deep dictionary using the key/value.

{
  "address": {
    "street": {
      "number": 56
    }
  }
}

Finally, the juggler tries to do our job once again--unfortunately this time, there doesn't seem to be a fix beyond patching upstream. It filters the fields again after they've been assembled, this removes the entire address since it doesn't appear in fields. Just adding it wouldn't be good else we'd get the entire deep object... I've patched it upstream here: u8sand/loopback-datasource-juggler@81ed9df

With the patch applied, it works, except that we can't identify sub-field types and end up with a string instead of an integer.

I hope we can somehow mitigate some duplication happening in the loopback-datasource-juggler before this becomes a viable feature.

I also need to add more tests, deal with the other filter formatting, and carefully review things to ensure we're not introducing any vulnerabilities (more field escaping may be in order).

Any suggestions welcome to make this feature more of a reality.

Related issues

• Finishes off what was started by Add basic json query support #304 by adding fields
• Nested object filtering on a JSON field #266
• ... definitely others

Checklist

• Received feedback
• Patch strongloop/loopback-datasource-juggler
• New tests added or existing tests modified to cover all changes
• Code conforms with the style guide

[slnode]

Can one of the admins verify this patch? To accept patch and trigger a build add comment ".ok\W+to\W+test."

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.