Error in reactive flow when adding BlockHound #1444
Comments
|
I played around with this a bit today. Replacing the |
|
Thanks for your investigation. After jumping through the exceptions I can see and your findings regarding the lock, I focussed on: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantLock.html#unlock-- Edit: From ReentrantReadWriteLock Edit 2: Alright, this blog post https://spring.io/blog/2019/12/13/flight-of-the-flux-3-hopping-threads-and-schedulers makes it clear that it can happen every time. |
|
@meistermeier and @seabamirum, many thanks for investigating this, it is very much appreciated! 👍 As far as I can tell, BlockHound catches a blocking call and throws an exception. However, it gets swallowed by another error coming from a try {
write.lock(); // throws reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
work.run();
} finally {
write.unlock(); // throws java.lang.IllegalMonitorStateException because the lock has not actually been acquired by this thread
}This may be whitelisted: BlockHound.install(builder -> builder.allowBlockingCallsInside("org.neo4j.driver.internal.async.pool.NettyChannelTracker", "doInWriteLock"));However, there are several places with locks in the driver. Perhaps it is worth making a BlockHound integration similar to what was done here? reactor/BlockHound#75 What do you think? |
|
Before converting a number of my Flux.flatMaps to Flux.concatMaps, I was observing mysterious application hangs on a test server that was not running Blockhound. However, the tests all seem to pass with Blockhound disabled, so if telling it to ignore the blocking lock() method prevents the subsequent IllegalMonitorStateException then maybe that's all that is needed? |
|
With Blockhound enabled, my application grinds to a halt with JMeter running 15 threads. So it must be related to the swallowed exception that you found and locks not getting released. If a custom integration works, it seems less risky than replacing the locks with other synchronization mechanisms. However, |
|
@seabamirum, we have introduced an initial experimental support for BlockHound in the driver. If this is something that is easy enough for you to test, please build the driver and give it a try. mvn clean install -DskipTestsObviously If we detect another issue, we are more than happy to take a look as well. |
|
@seabamirum, the next driver release will also come with this update: #1457 |
Original report: spring-projects/spring-data-neo4j#2755
Investigating the issue, I got rid of all Spring Data Neo4j bits and can reproduce the problem with the (reactive) session of the Java driver alone.
There are different outcomes observable:
where I suspect
to be the root problem.
Added dependency:
Driver version: 5.10.0
Reproducer: https://github.com/meistermeier/neo4j-driver-reactive-blockhound
The text was updated successfully, but these errors were encountered: