random mapping record error

[cqgong]

Hi, I am also facing an error in which a record could not be mapped. The tricky part is that this error is intermittent after upgrading SDN to 6.3.1(spring boot 2.7.1), sometime happens sometimes not, it is working well with SDN 6.2.4(spring boot 2.6.7)

controller (full page is here)

@RequestMapping
public void Image(){
    ReactionLikeEvent rle = getReactionLikeEvent(identifier);
}

private ReactionLikeEvent getReactionLikeEvent(String id) {
        ReactionLikeEvent rle;
        try {
            rle = databaseObjectService.findById(id);
        } catch (ClassCastException e) {
            throw new DiagramExporterException(String.format("The identifier '%s' does not correspond to a 'ReactionLikeEvent'", id));
        }
        if (rle == null) throw new NotFoundException(String.format("Identifier '%s' not found", id));
        return rle;
}

service(full page is here)

@Service
public class DatabaseObjectService {

    private final DatabaseObjectRepository databaseObjectRepository;

    public DatabaseObjectService(DatabaseObjectRepository databaseObjectRepository) {
        this.databaseObjectRepository = databaseObjectRepository;
    }

    public <T extends DatabaseObject> T findById(Object identifier) {
        T rtn = null;
        String id = DatabaseObjectUtils.getIdentifier(identifier);
        if (DatabaseObjectUtils.isStId(id)) {
            rtn = databaseObjectRepository.findByStId(id);
        } else if (DatabaseObjectUtils.isDbId(id)) {
            rtn = databaseObjectRepository.findByDbId(Long.parseLong(id));
        }
        if (rtn != null) rtn.isLoaded = true;
        return rtn;
    }
}

repository(full page is here)

@Repository
public class DatabaseObjectRepository {

    private final Neo4jTemplate neo4jTemplate;
    private final Neo4jClient neo4jClient;

    @Value("${spring.data.neo4j.database:graph.db}")
    private String databaseName;

    @Autowired
    public DatabaseObjectRepository(Neo4jTemplate neo4jTemplate, Neo4jClient neo4jClient) {
        this.neo4jTemplate = neo4jTemplate;
        this.neo4jClient = neo4jClient;
    }

    public <T extends DatabaseObject> T findByStId(String stId) {
        String query = "MATCH (a:DatabaseObject{stId:$stId})-[r]-(m) RETURN a, COLLECT(r), COLLECT(m)";
        return (T) neo4jTemplate.findOne(query, Map.of("stId", stId), DatabaseObject.class).orElse(null);
    }
}

data model:
ReactioneLikeEvent class.

Event class

DatabaseObject class

error message is, it is not easy to produce the error as it happens randomly.

org.springframework.data.mapping.MappingException: Error mapping Record<{a: node<1630761>, COLLECT(r): [relationship<6837398>, relationship<6837397>, relationship<6837396>, relationship<6837395>, relationship<6837392>, relationship<6837391>, relationship<6837390>, relationship<6837389>, relationship<6837388>, relationship<6837387>, relationship<6837386>, relationship<6837385>, relationship<6837384>, relationship<6837383>, relationship<6837382>, relationship<6837381>, relationship<6837380>, relationship<6837379>, relationship<6837378>], COLLECT(m): [node<1630669>, node<1629976>, node<1630748>, node<1630762>, node<8>, node<1629971>, node<789490>, node<1629968>, node<1630684>, node<1629858>, node<789284>, node<135593>, node<245>, node<1630748>, node<789282>, node<1630741>, node<1630738>, node<167296>, node<1630743>]}>
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.read(DefaultNeo4jEntityConverter.java:122)
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.read(DefaultNeo4jEntityConverter.java:71)
	at org.springframework.data.neo4j.core.mapping.Schema.lambda$getRequiredMappingFunctionFor$0(Schema.java:96)
	at org.springframework.data.neo4j.core.PreparedQuery$AggregatingMappingFunction.apply(PreparedQuery.java:246)
	at org.springframework.data.neo4j.core.PreparedQuery$AggregatingMappingFunction.apply(PreparedQuery.java:158)
	at org.springframework.data.neo4j.core.DelegatingMappingFunctionWithNullCheck.apply(DelegatingMappingFunctionWithNullCheck.java:45)
	at org.springframework.data.neo4j.core.DelegatingMappingFunctionWithNullCheck.apply(DelegatingMappingFunctionWithNullCheck.java:35)
	at org.springframework.data.neo4j.core.DefaultNeo4jClient$DefaultRecordFetchSpec.one(DefaultNeo4jClient.java:455)
	at java.base/java.util.Optional.flatMap(Optional.java:294)
	at org.springframework.data.neo4j.core.Neo4jTemplate$DefaultExecutableQuery.getSingleResult(Neo4jTemplate.java:1030)
	at org.springframework.data.neo4j.core.Neo4jTemplate.findOne(Neo4jTemplate.java:246)
	at org.reactome.server.graph.repository.DatabaseObjectRepository.findByStId(DatabaseObjectRepository.java:36)
	at org.reactome.server.graph.repository.DatabaseObjectRepository$$FastClassBySpringCGLIB$$64a0d279.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
	at org.reactome.server.graph.repository.DatabaseObjectRepository$$EnhancerBySpringCGLIB$$c29edc64.findByStId(<generated>)
	at org.reactome.server.graph.service.DatabaseObjectService.findById_aroundBody0(DatabaseObjectService.java:32)
	at org.reactome.server.graph.service.DatabaseObjectService.findById_aroundBody1$advice(DatabaseObjectService.java:33)
	at org.reactome.server.graph.service.DatabaseObjectService.findById(DatabaseObjectService.java:1)
	at org.reactome.server.service.controller.exporter.ImageExporterController.getReactionLikeEvent(ImageExporterController.java:262)
    at org.reactome.server.service.controller.exporter.ImageExporterController.reactionImage(ImageExporterController.java:221)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
    at org.springframework.test.web.servlet.TestDispatcherServlet.service(TestDispatcherServlet.java:72)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
	at org.springframework.mock.web.MockFilterChain$ServletFilterProxy.doFilter(MockFilterChain.java:167)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.reactome.server.service.utils.CustomRequestFilter.doFilterInternal(CustomRequestFilter.java:34)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.reactome.server.service.utils.CustomRequestFilter.doFilterInternal(CustomRequestFilter.java:34)
	at org.springframework.test.web.servlet.setup.PatternMappingFilterProxy.doFilter(PatternMappingFilterProxy.java:100)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:199)
	at org.reactome.server.service.utils.BaseTest.mockMvcGetResult(BaseTest.java:69)
	at org.reactome.server.service.controller.exporter.ImageExporterControllerTest.reactionImage(ImageExporterControllerTest.java:38)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
Caused by: org.springframework.data.mapping.model.MappingInstantiationException: Failed to instantiate org.reactome.server.graph.domain.model.ReactionLikeEvent using constructor public org.reactome.server.graph.domain.model.ReactionLikeEvent() with arguments 
	at org.springframework.data.mapping.model.ClassGeneratingEntityInstantiator$MappingInstantiationExceptionEntityInstantiator.createInstance(ClassGeneratingEntityInstantiator.java:358)
	at org.springframework.data.mapping.model.ClassGeneratingEntityInstantiator.createInstance(ClassGeneratingEntityInstantiator.java:102)
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.instantiate(DefaultNeo4jEntityConverter.java:462)
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.lambda$map$2(DefaultNeo4jEntityConverter.java:286)
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.map(DefaultNeo4jEntityConverter.java:305)
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.map(DefaultNeo4jEntityConverter.java:259)
	at org.springframework.data.neo4j.core.mapping.DefaultNeo4jEntityConverter.read(DefaultNeo4jEntityConverter.java:120)
	... 134 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.reactome.server.graph.domain.model.ReactionLikeEvent]: Class is abstract
	... 141 common frames omitted

References:

our service Github repo:  https://github.com/reactome/content-service 

graph core Github repo: https://github.com/reactome/graph-core/

it would be great if you have any ideas about it and thank you so much for your help.

[meistermeier]

That's interesting, because the part that will determine the right implementation of the DatabaseObject / ReactionLikeEvent has not been changed between 6.2.x and 6.3.x.
The code looks ok and it should definitely determine the right type to instantiate.
Are you sure that your database does not accidentally contain a node with the missing concrete label?
If you can reproduce it locally, it would be helpful to know what the results around those lines are

spring-data-neo4j/src/main/java/org/springframework/data/neo4j/core/mapping/DefaultNeo4jEntityConverter.java

Lines 282 to 284 in dabbecf

	List<String> allLabels = getLabels(queryResult, nodeDescription);
	NodeDescriptionAndLabels nodeDescriptionAndLabels = nodeDescriptionStore
	.deriveConcreteNodeDescription(nodeDescription, allLabels);

To give me some context, does this happen only in concurrency scenarios or also in "single threaded" tests?

[meistermeier]

Sorry, I have to correct me: Was just looking at the head of 6.2.x and not 6.2.4 in special. You are right. The determination there is quite different and our goal was to improve it and not make it worse (of course).
I will try create a simpler reproducer from the given domain class constellation and hope I will get some insights what might be the problem.

[meistermeier]

I tried my best to figure out how to re-produce the problem, but still no chance. Could you maybe help me in providing some context?
Is this random error only occurring when running the tests (fresh application start), or does this also happen "suddenly" in a running application where it has worked before (without restart)?

[cqgong]

I tried my best to figure out how to re-produce the problem, but still no chance. Could you maybe help me in providing some context? Is this random error only occurring when running the tests (fresh application start), or does this also happen "suddenly" in a running application where it has worked before (without restart)?

Yes, actually it almost happens everywhere that the library gets involved, even for standalone java applications. You won't see this error when the application starts. I've found another one yesterday, we deployed exactly the same war file on different servers, dev is not working and production is working, the error message is basically the same just different class name, the dirty fix is redeploying the war file again, however, it won't fix the issue and might happen again.

[meistermeier]

But when it happens, it happens right from the start? SDN does not map correctly first and then suddenly fails for the same class?
Adding some thoughts I had, maybe this also helps you to help me :)

1. The deployment looks like the one in the repository, right? (Can only access graph-core). So the Main application is always "on top" of the domain package.
   The only scenario where I could force the same error is that I move the concrete implementations in a sibling package to Main, so that they won't get discovered on start. The DatabaseObject (and all entities defined in there) will get picked up if it gets found within the Main application package. But as I said, this is something I really have to force SDN to do.
2. I am not very familiar with AspectJ and Spring, but I assume that the autoconfiguration still kicks in as usual. I mention this because the basePackages / value of the @EnableNeo4jRepositories in Main and the graph config points to the repositories instead of the domain classes. This value gets ignored if Spring Boot autoconfiguration is running, so it should not affect the domain setup, if it still runs as I expect.

[cqgong]

But when it happens, it happens right from the start? SDN does not map correctly first and then suddenly fails for the same class? Adding some thoughts I had, maybe this also helps you to help me :)

1. The deployment looks like the one in the repository, right? (Can only access graph-core). So the Main application is always "on top" of the domain package.
   The only scenario where I could force the same error is that I move the concrete implementations in a sibling package to Main, so that they won't get discovered on start. The DatabaseObject (and all entities defined in there) will get picked up if it gets found within the Main application package. But as I said, this is something I really have to force SDN to do.
2. I am not very familiar with AspectJ and Spring, but I assume that the autoconfiguration still kicks in as usual. I mention this because the basePackages / value of the @EnableNeo4jRepositories in Main and the graph config points to the repositories instead of the domain classes. This value gets ignored if Spring Boot autoconfiguration is running, so it should not affect the domain setup, if it still runs as I expect.

I am sorry I couldn't provide you with useful information, I think it happens when querying the graph database, you won't see this error message when our application starts. I have created a demo project with only one main class to allow you to produce the same error message, however, you have to run our graph database locally, here is the database dump file, and the demo project is below.

demo.tar.gz

[meistermeier]

Thanks for the input. I am still without any modifications not running into this problem.
Even after ~50 runs it still ends correctly (?) with
Display name is Deletion of residues 1091 to 1091.
TBH I do not know how to continue this right now. We can leave it open for now because I also want to get this fixed. Probably checking against different JVMs to figure out if they might trigger another behaviour.

[cqgong]

Thanks for the input. I am still without any modifications not running into this problem. Even after ~50 runs it still ends correctly (?) with Display name is Deletion of residues 1091 to 1091. TBH I do not know how to continue this right now. We can leave it open for now because I also want to get this fixed. Probably checking against different JVMs to figure out if they might trigger another behaviour.

Thank you so much for your reply, may I know which JVMs are you using? We use Java 11.

[meistermeier]

I tested 11 - 18 from different vendors.

[EliotRagueneau]

Hi, sorry to get involved in the middle of the thread, but may I ask you wich version of Neo4j are you using to perform the tests? Maybe the issue is linked to this?

[EliotRagueneau]

Hi everyone!
I have started investigating what is going on internally in SDN when we are observing these issues:
The problem seems to come from NodeDescription: our parent abstract class is having in our model is supposed to have 104 children classes
However when trying to computeConcreteNodeDescription, we are trying to assess which of the children classes of the declared returned type is fiting to the received data. The problem is that the haystack returned by entityDescription.getChildNodeDescriptionsInHierarchy() only has 45 classes among the expected 104.

Among those 45, none are belonging to the tree of the target class, so the only common label is the primary root (Database Object), and therefore it picks a random subclass of DatabaseObject for the mapping. Sometimes it picks one of our abstract classes and so it completely fails, and sometimes it just picks another class, but not the actual one we received from neo4j, then things are working because we only look at the displayName which is at root property, but any omore specific property accession would definetly fails.
The question is therefore why entityDescription.getChildNodeDescriptionsInHierarchy() only returns 45 node classes when it should reurn 104. I'll keep digging to understand whall subclasses are not there.

I've modified the demo file to showcase that even when it looks like the. mapping is working, the type is actually never the one expected : FragmentDeletionModification
demo.tgz

[EliotRagueneau]

I tried upgrading SDN to 6.3.6 as some changes where released then are regarding the NodeDescriptionStore and how they map nodes #2619. However this didn't solved the issue.
I have also tried by referencing the missing class in a Repository and it seems to work. So my assumption is that the NodeDescriptions children gets populated according to the presence of a Repository directly referencing the Model.
Somehow this behaviour only occurs when we are using our graph defining package as a dependency. When performing the queries from the graph package itself, then all children classes are well referenced.

Therefore the issue might also come from a Spring Boot optimization when using a dependency which leads to not "index" classes which are not directly in use.

[EliotRagueneau]

I have managed to fix the issue:
The problem was that when we are using our graph-library as a dependency in executable jar, we were initializing the Neo4j connection using a Configuation class which was lacking @EntityScan(basePackages = "org.reactome.server.graph.domain.model"). Somehow wihtin the graph package context, all entities were well indexed in NodeDescriptionStore, but when used as a dependency, then only Entities indexed by Repositories were indexed, probably thanks to @EnableNeo4jRepositories(basePackages = "org.reactome.server.graph.repository") annotation which we did have.

Then adding the @EntityScan solved the issue when running from IDE, but when running the using the executable jar repackaged by spring-boot-maven-plugin, then not all entities where indexed. This also got solved by also adding @Configuration to the base Configuration class, which is I guess targeted by spring-boot-maven-plugin to recognise which configuration classes to consider for the spring-context.

TLDR: Be sure to have a Configuration class in your graph library package which contains the following annotations:

@Configuration // Important when repackaging using spring-boot-maven-plugin
@EntityScan(basePackages = "org.reactome.server.graph.domain.model") // Important to be sure to index all Entities 

@ComponentScan(basePackages = "org.reactome.server.graph")
@EnableNeo4jRepositories(basePackages = "org.reactome.server.graph.repository")
@EnableSpringConfigured
@EnableAutoConfiguration
public class GraphCoreNeo4jConfig {}

[EliotRagueneau]

After further inspection, the issue is not is not entirely fixed: When running it from our spring-boot application context, the haystack for children of DatabaseObjects only gets populated with between 95 and 101 objects instead of the expected 102... The big problem being that this number is random, so depending on luck, some queries might work and some not because the return type does not get picked up by the NodeDescriptionStore, even after explicit declaration of @EntityScan

[meistermeier]

Thanks for the deep analysis of the problem. And I am sorry that the (correct) scanning configuration did not solve all problems.
The set of entities will get passed into SDN in a "random" order. So it seems that this might cause a problem.
Although I think you took care of this: Are you sure that all 102 expected entities are within the scanned path?
Also, I will have a look at the scanning / registration process again today.

[EliotRagueneau]

@meistermeier I managed to fix this last issue by ensuring we are using the same version of spring-data-neo4j on both our graph library and the packages that are using it: their version were overridden by spring-boot-parent 2.7.1 and failing, but now that they are all using spring-boot-parent 2.7.6, it looks like the number always stays at 102 which is the correct number (they were 2 classes from the snaphots which were defined in another path).
So I'm not sure why the mismatch of version resulte in some entities not getting indexed, but it seems that it got fix in SDN 6.3.6, or the conflict of version in maven resulted in random classes getting though the index.

[meistermeier]

Issue was solved by the user. Closing this now.