Custom Mongo queries inexplicably return zero results after upgrading to 3.0.8.RELEASE or later (spring-boot dep mgmt: 2.3.10.RELEASE or later)

[oliverlockwood]

I've created a repo to demonstrate the issue, at https://github.com/oliverlockwood/mongodb-issue-demonstration, based on following the guidelines at https://spring.io/guides/gs/accessing-data-mongodb/.

That repo has a clear README but I'll highlight the main points here as well.

• In terms of environment, I have a local Docker container running MongoDB version 4.2.14, mapping ports 27017 and 28017.
• The Customer model class contains a map field, like: public Map<String, DataValue> additionalData;
• There are custom read/write converters configured; we persist the map as an array of (name, value) tuples, viz:
  
• We have a custom repository implementation which allows dynamic queries to be specified (in our real use-case, via REST query payloads; in the example repo, via hard-coding in the run() method)
• The custom Mongo query we generate in the application (regardless of spring-data-mongodb version) is logged out as:
  Query: { "$and" : [{ "additionalData.name" : "a"}, { "additionalData.value" : "b"}]}, Fields: {}, Sort: {}
• With spring-boot version 2.3.9.RELEASE (therefore spring-data-mongodb version 3.0.7.RELEASE) an entry is found as expected by this query
• With spring-boot version 2.3.10.RELEASE or higher (therefore spring-data-mongodb version 3.0.8.RELEASE or higher) no results are found
• Directly querying Mongo via the console, i.e.
  db.getCollection('customer').find({ "$and" : [{ "additionalData.name" : "a"}, { "additionalData.value" : "b"}]})
  finds an entry as expected

I therefore conclude this is a regression in spring-data-mongodb, but I can't for the life of me figure out why it stops returning results.

I hope this in combination with the repo is enough context for you to investigate; please advise if you need me to provide any more information.

Many thanks!

[christophstrobl]

thanks for reporting and the sample - we'll have a look.

[mp911de]

Looks like it is related to #3659.

[oliverlockwood]

@mp911de that's interesting, and slightly surprises me, given the different affected versions (the ticket you refer is noted as a regression in versions after 3.1.9, while this ticket is a regression in versions after 3.0.7).

Still, I look forward to hearing more and will track both issues.

[mp911de]

We had a change where we updated mapping of property paths pointing into collection and map-like structures that maps errornously additionalData.name into additionalData.name.name.

There's quite a bit of a mismatch in the data model. additionalData is defined as Map<String, DataValue> while it actually is List<DataValue>. Changing the data type to additionalData fixes the problem.

We map queries against the domain model so trying to compare additionalData.name=foo against Map<String, DataValue>describes an attempt to query whether the map (subdocument)additionalDatacontains a fieldnamewith the valuefoo while DataValue isn't compatible with that representation.

In any case, trying to query into a map should render the proper field name into the query.

[oliverlockwood]

Thanks for your comment on 18th June, @mp911de. Any news on when this might be fixed? I'm a bit worried about getting stuck on an old version of spring-data-mongodb while we continue to keep updating our platform (next step is to move to Spring Boot 2.5.x).,

[mp911de]

You can fix the issue immediately within your code by aligning the data model to the actual representation in your MongoDB documents. Other than that, we need to revisit the entire map and array position translation that isn't planned short term.