ReadPreference not applied consistently in concurrent usage of MongoTemplate [DATAMONGO-1061]

[spring-projects-issues]

Bogdan Apetrei opened DATAMONGO-1061 and commented

In the scenario when we have one mongodb factory used by two mongo templates defined as below:

          <bean id="mongoTemplateReadFromPrimary" class="org.springframework.data.mongodb.core.MongoTemplate">
              <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
          </bean>

          <bean id="mongoTemplateReadFromSecondary" class="org.springframework.data.mongodb.core.MongoTemplate">
              <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
              <property name="readPreference" value="SECONDARY" />
          </bean>

operations using mongoTemplateReadFromPrimary will be executed also on secondaries.

This can cause unreliable reads since there is a small delay in the replication process.

This behaviour comes in contradiction with:

- MongoDB documentation:

    "By default, an application directs its read operations to the primary member in a replica set.
    Reading from the primary guarantees that read operations reflect the latest version of a document."

- Behaviour when only one mongoTemplate is used, when all reads go to primary

- Spring Data MongoDB documentation, where the need to explicitly specify the readPreference is not documented.
    (at least I was not able to find it)

Problem is hard to identify when the application is highly concurrent. (concurrency was excluded from test scenarios)

I attached the project I used for the test scenarios. You can find there both the java code but also the docker files
and the code used to create the mongod, mongos and mongo config servers

---

Attachments:

• MongoDB-Docker-dev (1).zip (13.32 kB)

Referenced from: pull request #386

[spring-projects-issues]

Bogdan Apetrei commented

Test scenario

Input:

• one mongo:db-factory with one mongoTemplate with no readPreference explicit value.
• 50 queries executed sequentially.

Results:

• all 50 queries are executed on primary server

• executing command: sudo docker logs rs1_srv1 | grep appDomain returns 50 lines similar to:

  Wed Sep 24 18:14:35.773 [conn28] query database.appDomain query: { $comment: "QUERY_ON_PRIMARY", $query: {} } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:13 nreturned:0 reslen:20 0ms

• executing command: sudo docker logs rs1_srv1 | grep appDomain | wc -l returns value 50

Note:

• this is expected since from MongoDB documentation we have:

  By default, an application directs its read operations to the primary member in a replica set. Reading from the primary guarantees that read operations reflect the latest version of a document.

---

---

Test scenario

Motivation:

• to scale MongoDB for read operations it is decided that some of the read operations can be performed on the secondaries while others more critical remain on the primary

Input:

• one mongo:db-factory with two mongoTemplate, code as below

  <bean id="mongoTemplateReadFromPrimary" class="org.springframework.data.mongodb.core.MongoTemplate">
  <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
  </bean>
  <bean id="mongoTemplateReadFromSecondary" class="org.springframework.data.mongodb.core.MongoTemplate">
  <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
  <property name="readPreference" value="SECONDARY" />
  </bean>

• 50 queries executed sequentially using each of the templates

Results:

• executing on primary

  sudo docker logs rs1_srv1 | grep appDomain
  Wed Sep 24 18:49:53.957 [conn26] query database.appDomain query: { $comment: "QUERY_ON_PRIMARY", $query: {} } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) W:179 r:20 nreturned:0 reslen:20 0ms
  sudo docker logs rs1_srv1 | grep appDomain | wc -l
  1

• executing on secondary 1

  sudo docker logs rs1_srv2 | grep appDomain
  Wed Sep 24 18:49:53.960 [conn32] query database.appDomain query: { $comment: "QUERY_ON_SECONDARY", $query: {}, $readPreference: { mode: "secondaryPreferred" } } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) W:114 r:19 nreturned:0 reslen:20 0ms
  Wed Sep 24 18:49:53.961 [conn32] query database.appDomain query: { $comment: "QUERY_ON_PRIMARY", $query: {}, $readPreference: { mode: "secondaryPreferred" } } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) r:16 nreturned:0 reslen:20 0ms
  ........ //following 97 lines

  sudo docker logs rs1_srv2 | grep appDomain | wc -l
  99

• executing on secondary 2

  sudo docker logs rs1_srv3 | grep appDomain | wc -l
  0

Observation:

• we see that queries which should be executed on primary are instead executed on the secondary server

---

---

Test scenario
Motivation:

• the same as above

Input:

• one mongo:db-factory with two mongoTemplate, code as below, read preference specified explicitly for both templates

  <bean id="mongoTemplateReadFromPrimary" class="org.springframework.data.mongodb.core.MongoTemplate">
  <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
  <property name="readPreference" value="PRIMARY" />
  </bean>
  <bean id="mongoTemplateReadFromSecondary" class="org.springframework.data.mongodb.core.MongoTemplate">
  <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
  <property name="readPreference" value="SECONDARY" />
  </bean>

• 50 queries executed sequentially using each of the templates

Result:

• executing on primary

  sudo docker logs rs1_srv1 | grep appDomain
  Wed Sep 24 19:10:22.963 [conn23] query database.appDomain query: { $comment: "QUERY_ON_PRIMARY", $query: {} } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) W:124 r:17 nreturned:0 reslen:20 0ms
  ........

  sudo docker logs rs1_srv1 | grep appDomain | wc -l
  50

• executing on secondary 1

  sudo docker logs rs1_srv2 | grep appDomain
  Wed Sep 24 19:10:22.965 [conn17] query database.appDomain query: { $comment: "QUERY_ON_SECONDARY", $query: {}, $readPreference: { mode: "secondaryPreferred" } } ntoreturn:0 ntoskip:0 nscanned:0 keyUpdates:0 locks(micros) W:96 r:19 nreturned:0 reslen:20 0ms

  sudo docker logs rs1_srv2 | grep appDomain | wc -l
  50

• executing on secondary 2

  sudo docker logs rs1_srv3 | grep appDomain | wc -l
  0

Observation:

• we see that queries which should be executed on primary are executed only on primary server

[spring-projects-issues]

Agoston Horvath commented

I've just bumped into the exact same thing. I've made 2 mongoTemplates, one with readPreference: secondaryPreferred, and one with readPreference: primaryPreferred (for reads and writes, respectively). Dashboard however shows that all reads to to primary. Looking into it, it all boils down to MongoTemplate.prepareCollection() gets the cached DBCollection from the mongo driver and sets the read preference on it.

This means that one can't have 2 different read preferences per collection. This is kind of a blocker, since for the usage pattern 'read-munge-persist', using optimistic locking, one needs a 'primaryPreferred' read preference, while for a regular 'read' pattern, the delay to the secondary nodes is acceptable.

UPDATE: A sort workaround is to create 2 drivers for the same database, setting one of them with a global readpreference primaryPreferred, the other to secondaryPreferred. This is rather dirty though

[spring-projects-issues]

Bogdan Apetrei commented

DATAMONGO-1061 - change in MongoTemplate for setting readPreference when multiple templates are used with different read preferences

[spring-projects-issues]

Bogdan Apetrei commented

Hello Agoston,

As a workaround I found that if the readPreference is set explicitly for both templates read operations will go fine. Example:

 
<bean id="mongoTemplateReadFromPrimary" class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
<property name="readPreference" value="PRIMARY" />
</bean>

<bean id="mongoTemplateReadFromSecondary" class="org.springframework.data.mongodb.core.MongoTemplate">
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
<property name="readPreference" value="SECONDARY" />
</bean>

Indeed problem looks to be in MongoTemplate.prepareCollection() where we have:

protected void prepareCollection(DBCollection collection) {
     if (this.readPreference != null) {
          collection.setReadPreference(readPreference);
     }
}

Best regards,
Bogdan

[spring-projects-issues]

Agoston Horvath commented

Bogdan,

I wish it was that simple. But the mongo driver itself (as of version 3.2.1) caches the DBCollections in itself in DB.getCollection():

public DBCollection getCollection(final String name) {
        DBCollection collection = collectionCache.get(name);
        if (collection != null) {
            return collection;
        }
        [...]

So for now, having 2 drivers (with a new mongo driver instance in each) is the only option to circumvent this.

I would also raise priority on this one. Very often, updates cannot be used because of complex rules, in which case, the only option is to read in the whole document, do the changes in java, and then use .save() (with a @Version field of course to avoid race conditions). In this case, readPreference=secondaryPreferred would mean that when executing this logic rapidly over the same document, we'd be always reading the stale/outdated version of the object.

But if one switches to primaryPreferred, then also for simple queries the primary node is used, which effectively makes the replica slaves useless.

This 2 clients hack works, but I think it is really kludgy for such a simple requirement

[spring-projects-issues]

Oliver Drotbohm commented

Have you guys filed this with the MongoDB guys? I think that caching mutable objects is a receipt for disaster fundamentally. The only way I can imagine working around this on our side is synchronizing on the collection name but I don't think we'd want to do that for performance reasons.

As indicated by Agoston Horvath, always setting the value doesn't really help as we're still subject to conflicts in concurrent scenarios

[spring-projects-issues]

Oliver Drotbohm commented

I've filed a ticket with the MongoDB guys

[spring-projects-issues]

Bogdan Apetrei commented

Hi Oliver,

I agree with you, problem is now more complicated because of the changes done in MongoDB Java Driver (version 3.x) but in the same time it was observed on older versions of MongoDB Spring Data (ex: 1.6) with older version of MongoDB Java Driver (ex: 2.13) before caching was implemented in the driver.

I think a fix in the MongoDB Spring Data would be required also after the fix in the MongoDB Java Driver.

Thank you,
Bogdan

[spring-projects-issues]

Oliver Drotbohm commented

If the MongoDB guys decide to leave things as they are (which is the way it currently looks), there's probably nothing we can realistically do except document the limitation. If the issue is fixed in the driver, all you need to do is to upgrade to a version of the driver including the fix

[spring-projects-issues]

Agoston Horvath commented

Thanks for picking this up, and so quickly!

Since there's no quick fix, here's the copy-pasteable @Configuration that I made to make the 2 driver hack look nice (from far, far away):

@Configuration
@EnableMongoRepositories(basePackages = "com.bla")
public class MongoDBConfiguration extends AbstractMongoConfiguration {

    @Override
    protected String getDatabaseName() {
        return database;
    }

    /**
     * We make 2 clients - one with readPreference primaryPreferred, one with readPreference secondaryPreferred.
     * This is to work around the limitation in mongo driver v3.2.1 that is able to set read preference per-collection only,
     * not per-query.
     */
    @Override
    @Bean
    public Mongo mongo() throws Exception {
        final MongoClient client = isBlank(username) ?
                new MongoClient(getServers(), getClientOptions()) :
                new MongoClient(getServers(), singletonList(MongoCredential.createCredential(username, database, password)), getClientOptions());

        client.setReadPreference(ReadPreference.secondaryPreferred());
        return client;
    }

    @Bean
    public Mongo mongoPrimary() throws Exception {
        final MongoClient client = isBlank(username) ?
                new MongoClient(getServers(), getClientOptions()) :
                new MongoClient(getServers(), singletonList(MongoCredential.createCredential(username, database, password)), getClientOptions());

        client.setReadPreference(ReadPreference.primaryPreferred());
        return client;
    }

    /**
     * mongoTemplate has its read preference to secondary, e.g. when consistency is no issue
     * (this is the preferred template)
     */
    @Bean
    public MongoTemplate mongoTemplate() throws Exception {
        return new MongoTemplate(mongoDbFactory(), mappingMongoConverter());
    }

    /** Exact copy of mongoDbFactory(), but uses the writeMongo() client instead of the mongo() client
     * NB: will need to re-copy this when updating spring-mongo-data */
    @Bean
    public MongoDbFactory mongoDbWriteFactory() throws Exception {
        return new SimpleMongoDbFactory(mongoPrimary(), getDatabaseName(), getUserCredentials(), getAuthenticationDatabaseName());
    }

    /**
     * mongoPrimaryTemplate has its read preference to primary, e.g. when reading from the DB as part of an update operation.
     * (only use this when necessary, as it puts extra load on the master, e.g. to perform a read-update-write cycle with optimisitic locking)
     */
    @Bean
    public MongoTemplate mongoPrimaryTemplate() throws Exception {
        return new MongoTemplate(mongoDbWriteFactory(), mappingMongoConverter());
    }
}

[mp911de]

Fixed via #4286.