Error in writing Kinesis using Asynclinet - Caused by: java.nio.channels.ClosedChannelException: null

[dmunivel]

The Java client using AWS SDK (2.9.17) to write singleRecord to Kinesis from one AWS account to another AWS account through proxy . suddenly we are seeing lot of exception -

Caused by: java.io.IOException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:267)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:252)
	... 27 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null

Code we are using -

PutRecordResponse putRecordIntoKinesis(final PutRecordRequest record) {
try {
return this.kinesisAsyncClient.putRecord(record).join();
} catch (CompletionException e) {
throw new SdkCommonException(ex);
}

Expected Behavior

Should not throw exception.

Current Behavior

Throws Exception

Complete error stack -

Caused by: software.amazon.awssdk.core.exception.SdkClientException: null
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
	at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:253)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:139)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
	at software.amazon.awssdk.http.nio.netty.internal.CancellableAcquireChannelPool.lambda$acquire$1(CancellableAcquireChannelPool.java:56)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
	at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.ensureAcquiredChannelIsHealthy(HealthCheckedChannelPool.java:110)
	at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda$tryAcquire$1(HealthCheckedChannelPool.java:93)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
	at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35)
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:501)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	... 1 common frames omitted
Caused by: java.io.IOException: The channel was closed. This may have been done by the client (e.g. because the request was aborted), by the service (e.g. because the request took too long or the client tried to write on a read-only socket), or by an intermediary party (e.g. because the channel was idle for too long).
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.decorateException(NettyRequestExecutor.java:267)
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:252)
	... 27 common frames omitted
Caused by: java.nio.channels.ClosedChannelException: null
	at io.netty.handler.codec.http2.Http2StreamChannelBootstrap.findCtx(Http2StreamChannelBootstrap.java:156)
	at io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:120)
	at io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:110)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$bootstrapChildChannel$3(MultiplexedChannelRecord.java:136)
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.lambda$asyncPromiseNotifyingBiConsumer$1(NettyUtils.java:84)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.createChildChannel0(MultiplexedChannelRecord.java:125)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$createChildChannel$1(MultiplexedChannelRecord.java:118)
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:119)
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:135)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.createChildChannel(MultiplexedChannelRecord.java:117)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.acquire(MultiplexedChannelRecord.java:83)
	at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquire0(Http2MultiplexedChannelPool.java:101)
	at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.lambda$acquire$0(Http2MultiplexedChannelPool.java:90)
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:119)
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.doInEventLoop(NettyUtils.java:135)
	at software.amazon.awssdk.http.nio.netty.internal.http2.Http2MultiplexedChannelPool.acquire(Http2MultiplexedChannelPool.java:90)
	at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire0(BetterFixedChannelPool.java:169)
	at software.amazon.awssdk.http.nio.netty.internal.utils.BetterFixedChannelPool.acquire(BetterFixedChannelPool.java:138)
	at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.acquire0(HttpOrHttp2ChannelPool.java:78)
	at software.amazon.awssdk.http.nio.netty.internal.http2.HttpOrHttp2ChannelPool.lambda$acquire$0(HttpOrHttp2ChannelPool.java:67)
	at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
	... 6 common frames omitted

After enable the aws sdk debug logs -

{ [-]
   context: default
   exception: java.nio.channels.ClosedChannelException: null
	at io.netty.handler.codec.http2.Http2StreamChannelBootstrap.findCtx(Http2StreamChannelBootstrap.java:156)
	at io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:120)
	at io.netty.handler.codec.http2.Http2StreamChannelBootstrap.open(Http2StreamChannelBootstrap.java:110)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$bootstrapChildChannel$3(MultiplexedChannelRecord.java:136)
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.lambda$asyncPromiseNotifyingBiConsumer$1(NettyUtils.java:84)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.createChildChannel0(MultiplexedChannelRecord.java:125)
	at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.lambda$createChildChannel$1(MultiplexedChannelRecord.java:118)
	at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(Thread.java:748)

   level: DEBUG
   logger: software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor
   message: Failed to create connection to https://kinesis.us-east-1.amazonaws.com/
   thread: aws-java-sdk-NettyEventLoop-2-0
   timestamp: 2021-03-19 19:20:24,280
}

Steps to Reproduce

we are getting this exception in prod environment, the tasks are running in ECS both region, the request rate is 40 TPS. we are unable to reproduce this issue in QA environment.

Context

current application is customer facing application if any error happens we need to manually re-process the events from our s3 bucket.

If we re-cycle the container we can see exception is not shows up minimum certain period of time (4 to 5 hours) after that this exception start to throws again for that we need to go for re-cycle the container.

Your Environment

• AWS Java SDK version used: 2.9.17
• JDK version used: 1.8
• Operating System and version: ECS EC2 instance

[aaoswal]

Hi @dmunivel,
I am unable to reproduce the issue on my end. I know you mentioned not being able to produce this issue in QA environment but there have been multiple changes done to Kinesis after 2.9.17 so have you tried updating to one of the later versions of the SDK to see if that has resolved the issue on your end.

If the issue still persists, I would like to request you to please provide us with a simple sample code instead including the reproduction steps to allow me to reproduce the problem and help assist you faster.

Thank you for your patience.

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.