Skip to content

ConcurrentModificationException in BookmarkManager #1601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
harrel56 opened this issue Jan 4, 2025 · 2 comments · Fixed by #1604
Closed

ConcurrentModificationException in BookmarkManager #1601

harrel56 opened this issue Jan 4, 2025 · 2 comments · Fixed by #1604

Comments

@harrel56
Copy link

harrel56 commented Jan 4, 2025

Hello team,

I have an issue that ConcurrentModificationException gets thrown when executing multiple concurrent queries (reads).

Neo4j Version: 5.25.1 Community
Neo4j Mode: Single instance
Driver version: Java driver 5.26.3
Operating System: Windows 11 and Debian 12

I'm running these queries in a pooled virtual thread executor limited to 128 threads, heavy load.
TBH I don't even use bookmarks explicitly - this is the code which runs the query:

    public boolean exists(Gav gav) {
        Map<String, Object> gavData = objectMapper.convertValue(gav, new TypeReference<>() {});
        gavData.computeIfAbsent("classifier", k -> "");
        return !driver.executableQuery("""
                        MATCH (root:Artifact)
                        WHERE
                            root.groupId = $props.groupId
                            AND root.artifactId = $props.artifactId
                            AND root.version = $props.version
                            AND root.classifier = $props.classifier
                        RETURN root""")
                .withParameters(Map.of("props", gavData))
                .execute()
                .records()
                .isEmpty();
    }

Stacktrace:

java.util.ConcurrentModificationException
	at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1606)
	at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1629)
	at java.base/java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1079)
	at java.base/java.util.AbstractCollection.addAll(AbstractCollection.java:337)
	at java.base/java.util.HashSet.<init>(HashSet.java:122)
	at org.neo4j.driver.internal.async.NetworkSession.determineBookmarks(NetworkSession.java:593)
	at org.neo4j.driver.internal.async.NetworkSession.lambda$beginTransactionAsync$21(NetworkSession.java:291)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1194)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnectionProvider.release(PooledBoltConnectionProvider.java:609)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnectionProvider.lambda$release$39(PooledBoltConnectionProvider.java:613)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnection.lambda$close$17(PooledBoltConnection.java:304)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:931)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2401)
	at java.base/java.util.concurrent.CompletableFuture$MinimalStage.whenComplete(CompletableFuture.java:2992)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnection.lambda$close$18(PooledBoltConnection.java:304)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:978)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:955)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.api.BasicResponseHandler.onComplete(BasicResponseHandler.java:149)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.runIgnoringError(BoltConnectionImpl.java:665)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.onComplete(BoltConnectionImpl.java:653)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.lambda$new$0(BoltConnectionImpl.java:555)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.handleSummary(BoltConnectionImpl.java:659)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.onResetSummary(BoltConnectionImpl.java:618)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$9.onSummary(BoltConnectionImpl.java:326)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$9.onSummary(BoltConnectionImpl.java:316)
	at org.neo4j.driver.internal.bolt.basicimpl.messaging.v3.BoltProtocolV3.lambda$reset$8(BoltProtocolV3.java:310)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.basicimpl.handlers.ResetResponseHandler.resetCompleted(ResetResponseHandler.java:61)
	at org.neo4j.driver.internal.bolt.basicimpl.handlers.ResetResponseHandler.onSuccess(ResetResponseHandler.java:40)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.InboundMessageDispatcher.handleSuccessMessage(InboundMessageDispatcher.java:82)
	at org.neo4j.driver.internal.bolt.basicimpl.messaging.common.CommonMessageReader.unpackSuccessMessage(CommonMessageReader.java:59)
	at org.neo4j.driver.internal.bolt.basicimpl.messaging.common.CommonMessageReader.read(CommonMessageReader.java:49)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:78)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:33)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.MessageDecoder.channelRead(MessageDecoder.java:40)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1503)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1366)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1415)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1575)

Side note: Unfortunately, the original cause gets stripped down (I couldn't see the exact location where the exception is being thrown) - the stacktrace I pasted here was obtained via debugging.

While this exception frequency differs between my local runs on Windows 11 vs remote Debian 12 it's probably the matter of computation power.
While analyzing the stacktraces the Neo4jBookmarkManager code indeed looks faulty (please correct if it's not the case): calling Collections.unmodifiableSet(this.bookmarks) doesn't make the returned collection thread-safe - the underlying collection can still be modified which will be reflected on returned unmodifiable view. I believe that any such changes while iterating over unmodifiable collection (copy constructor) will likely result in ConcurrentModificationException.

I would really appreciate any kind of workaround (like opting-out of these bookmarks?).

Thanks in advance!

PS probably not necessary but here is the link to the whole codebase: https://github.com/harrel56/jar-hell/blob/97ed256128b9cca7cfaa8db8565ea59bf5d304c9/server/src/main/java/dev/harrel/jarhell/repo/ArtifactRepository.java#L77

@harrel56
Copy link
Author

harrel56 commented Jan 5, 2025

Well, I did disable the bookmark manager explicitly and the issue is gone - so in terms of a workaround I'm all good. Adjusted code:

    public boolean exists(Gav gav) {
        Map<String, Object> gavData = objectMapper.convertValue(gav, new TypeReference<>() {});
        gavData.computeIfAbsent("classifier", k -> "");
        try (var session = session()) {
            return session.executeRead(tx -> {
                Result res = tx.run(new Query("""
                        MATCH (root:Artifact)
                        WHERE
                            root.groupId = $props.groupId
                            AND root.artifactId = $props.artifactId
                            AND root.version = $props.version
                            AND root.classifier = $props.classifier
                        RETURN root""",
                        parameters("props", gavData))
                );
                return res.hasNext();
            });
        }
    }

    private Session session() {
        return driver.session(SessionConfig.builder().withBookmarkManager(null).build());
    }

But still, I think the original concurrency issue should be addressed

@injectives
Copy link
Contributor

@harrel56 , thanks for reporting this issue. #1604 should fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants