NullPointerException in NettyUtils.isConnectionResetException() and hung future

[bsfarrell]

Describe the bug

As part of a call to putObject, the following stack trace was logged:

2023-04-21 11:15:40.607 WARN  [aws-java-sdk-NettyEventLoop-1-2] io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:580) - An exception was thrown by software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor$$Lambda$2618/0x00000008015aa270.operationComplete()
java.lang.NullPointerException: Cannot invoke "String.contains(java.lang.CharSequence)" because the return value of "java.lang.Throwable.getMessage()" is null
        at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.isConnectionResetException(NettyUtils.java:84) ~[netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:76) ~[netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:309) ~[netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$writeRequest$13(NettyRequestExecutor.java:264) ~[netty-nio-client-2.20.33.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3.error(HttpStreamsHandler.java:309) ~[netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber.onError(HandlerSubscriber.java:263) ~[netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor$StreamedRequest$1.onError(NettyRequestExecutor.java:470) ~[netty-nio-client-2.20.33.jar:?]
        ...
        ... my Subscription implementation calls Subscriber::onError with an exception with a null message
        ...
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber.maybeRequestMore(HandlerSubscriber.java:303) [netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber.maybeStart(HandlerSubscriber.java:238) [netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber.provideSubscription(HandlerSubscriber.java:225) [netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber.access$000(HandlerSubscriber.java:41) [netty-nio-client-2.20.33.jar:?]
        at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$1.run(HandlerSubscriber.java:213) [netty-nio-client-2.20.33.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) [netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518) [netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [netty-all-4.1.42.Final.jar:4.1.42.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.42.Final.jar:4.1.42.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]

At this point the future returned by S3AsyncClient#putObject is effectively hung and never completes.

I am able to work around this by changing my subscription implementation to create an exception with a non-null message.

Expected Behavior

1. No NPE. It's not safe to assume that all exceptions have a message. This specific NPE was previously reported in A null pointer occurs when putObject #3671.
2. Regardless of any unchecked exceptions that are thrown in the internal processing of a request and response, the future returned by S3AsyncClient#putObject should always complete with either a result or an error. This hang behavior was previously reported in NullpointerException in NettyUtils.closedChannelMessage #3435.

Current Behavior

See stack trace in the description.

Reproduction Steps

I have not taken the time to write a self-contained reproduction because the cause of the NPE is clear:

private static boolean isConnectionResetException(Throwable originalCause) {
    return originalCause instanceof IOException && originalCause.getMessage().contains("Connection reset by peer");
}

originalCause.getMessage() can be null.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.20.3

JDK version used

17.0.2

Operating System and version

MacOS

[martinKindall]

@debora-ito I can work in this issue, the fix is simple. I noticed the method NettyUtils::decorateException is not unit-tested, so I can add proper tests too in NettyUtilsTest.java

[zoewangg]

The fix has been merged. Could you try with the latest version?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.