java.lang.IllegalStateException: Session/EntityManager is closed

[Jotschi]

I'm currently writing a PoC for Hibernate Reactive + Vert.x and encountered an issue when loading data. I see Session/EntityManager is closed whenever I concurrently access the REST API.

I was also able to reproduce the issue in the boiler plate howto application from Vert.x.

Steps to reproduce

• Import and run: https://github.com/vertx-howtos/hibernate-reactive-howto
• Create one product via

POST http://localhost:8080/products
{
   "name": "test",
   "price": 0
}

• Run wrk -d 10s -c 20 http://localhost:8080/products/1
• Observe the errors in the server log

Stacktrace

• https://gist.github.com/Jotschi/d82e906ecde9db4b2f2232306cff5c26

Notes

I have tried to create reproducer tests for this which do not require the use of wrk but so far it has proven hard to reliably reproduce the fault with a test.

Bumping the hibernate reactive dependency in the howto project to 1.1.0.Final did not affect the issue.

The example uses persist(product).call(session::flush) the javadoc states to use #map. I'm not that familiar with Mutiny and not sure whether both equivalent.

[DavideD]

I'm having a look but the fact that the session is closed makes me think that something hasn't been chain correctly in the stream.

[DavideD]

persist(product).call(session::flush)

yes, this is correct

[DavideD]

I'm having a look but the fact that the session is closed makes me think that something hasn't been chain correctly in the stream.

Nevermind, this is something else

[markusdlugi]

Hi @Jotschi, can you maybe try to set the HR version in your project to 1.0.0.CR9 and check if the problem still happens there?

We're seeing similar issues in our applications where HR will often throw such errors when things happen concurrently, but only in Quarkus 2.4.2.Final (which uses HR 1.0.1.Final). In Quarkus 2.3.1.Final (using HR 1.0.0.CR9) everything's working perfectly fine for us.

If the behavior's the same for @Jotschi, I guess some change in between those versions must be responsible for the problems, right, @DavideD? ORM is only one revision apart in both Quarkus versions (5.6.1.Final vs. 5.6.0.Final).

[DavideD]

I will check later but I guess it's possible

[Jotschi]

@markusdlugi The project is the official Vert.x hibernate reactive howto. I tested your suggested changes:

It seems the dependencies are not compatible. If I switch to 1.0.0.CR9 I get a NoSuchMethodError.
I tested mutiny-vertx 2.14.2 (Vert.x 4.1.5) and 2.16.0 (Vert.x 4.2.1)

java.lang.NoSuchMethodError: 'void io.vertx.core.net.impl.pool.ConnectionPool.acquire(io.vertx.core.impl.EventLoopContext, io.vertx.core.net.impl.pool.PoolWaiter$Listener, int, io.vertx.core.Handler)'

What Vert.x version are you suggesting? I can try to select the matching Vert.x version and report back to you.

[markusdlugi]

@Jotschi, Quarkus version 2.3.1.Final uses Vert.x 4.1.4 and mutiny-vertx 2.13.0

[Jotschi]

@markusdlugi With mutiny-vertx 2.13.0 and HR 1.0.0.CR9 I don't see the error.

[markusdlugi]

@Jotschi, great, thanks for the investigation :)

@DavideD, that should hopefully narrow it down quite a lot, right? So the problem was most likely introduced with either CR10 or Final:
https://github.com/hibernate/hibernate-reactive/milestone/12?closed=1
https://github.com/hibernate/hibernate-reactive/milestone/13?closed=1

[DavideD]

Thanks for investigating.
I have the feeling that's related to this issue: #950

[hantsy]

My Hibernate reactive + Vertx example: https://github.com/hantsy/vertx-sandbox/tree/master/mutiny-spring-hibernate, but not encountered this issue.

[gavinking]

Thanks for investigating.
I have the feeling that's related to this issue: #950

@DavideD I don't understand your comment here. WDYM?

[DavideD]

That's the most involved change between CR9 and CR10. So it might be related to it. That's all

[markusdlugi]

@DavideD I tested it with the reproducer provided by @Jotschi and the issue was actually introduced with 1.0.0.Final. Version CR10 is still working fine, no errors whatsoever. Once you switch the reproducer to Final, the issue happens even with the smallest bit of concurrency and is actually really bad. For example:

 ~ wrk -d 10s -c 20 http://localhost:8080/products/1
Running 10s test @ http://localhost:8080/products/1
  2 threads and 20 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   184.41ms   51.43ms 528.58ms   90.26%
    Req/Sec    57.00     27.67   101.00     64.61%
  1090 requests in 10.03s, 85.73KB read
  Non-2xx or 3xx responses: 1028
Requests/sec:    108.65
Transfer/sec:      8.55KB

So even with just 2 concurrent threads, 1028/1090 requests failed due to this issue, so like a 94% failure rate. Using HR CR9 or CR10, not a single request fails.

I don't have deep knowledge of your code, but if the problem was introduced with 1.0.0.Final then I think only the following change could possibly cause this, right? #1001

[markusdlugi]

And by the way, the latest release 1.1.1.Final is also still affected by this problem. Needless to say, it's not really usable with this rate of errors.

[DavideD]

I will check this issue this week

[DavideD]

@markusdlugi I've just tested this and if I revert #1001 everything works

[DavideD]

@markusdlugi It seems that applying this PR also fix the issue: #1118

I'm prone to say that this is the right fix but I'm not sure what's the difference between local and non local data in a Context.
I will continue tomorrow.

[markusdlugi]

@DavideD Thanks a lot for investigating! I can also test tomorrow with a more complex application in case you need more feedback on this.

[gavinking]

@DavideD the difference is that the local context is local to the particular "logical thread" whereas there is one (non-local) Context per verticle.

[DavideD]

@DavideD Thanks a lot for investigating! I can also test tomorrow with a more complex application in case you need more feedback on this.

More testing and feedback is always appreciated

[DavideD]

Any idea how I can unit test this?

[gavinking]

I'm certain that @Sanne's right, and this problem was a consequence of #948. The Vert.x context and the Vert.x local context are very different things—kinda like the difference between a static variable and a threadlocal—and if we mix them up then obviously everything goes haywire.

I think we need to prioritize an immediate release just to fix this. Because right now anyone who uses withTransaction() or withSession() has a completely broken program.

[gavinking]

(P.S. sorry for being slow to pick up on this, I was very busy recently with the other Hibernate which is doing a major release very soon.)

[DavideD]

We are working on this. I haven't merged the fix yet because I want to have a test in Hibernate Reactive for this.
I've almost finished writing one that should work for the time being

[DavideD]

Reopening this issue because the PR doesn't have a test

[DavideD]

It should be fixed now.
Thanks a lot @Jotschi for letting us know about this issue and patiently wait for a fix.

[markusdlugi]

Great, thanks a lot for fixing this @DavideD & @Sanne !

Will there be a dedicated release for this fix?

And in addition, it would be great if you could also try and get the fix (once it's released) into an upcoming Quarkus bugfix release soon. We have been stuck with Quarkus 2.3 since 2 months due to this issue and would really like to update soon :)

[DavideD]

I'm preparing a release now

[Sanne]

@markusdlugi yes we'll upgrade Quarkus right away, it will be included in 2.7

[DavideD]

I've started the release FYI

[DavideD]

Hibernate Reactive 1.1.2.Final is available. I've sent a PR for Quarkus: quarkusio/quarkus#22898

[gavinking]

Excellent, thanks for all your work on this @DavideD.