Code is waiting on environmentFuture.join()

[kogantihemanth]

We are trying to publish our service metrics in EMF and we are using aws-embedded-metrics-java library to publish metrics in EMF. We are seeing a lot of rejected execution logs in our server logs.

rejected execution of com.amazonaws.juno.metric.JunoMetricEmittingService$2@4aee6a4 on OpenSearchThreadPoolExecutor[name = ip-10-100-176-158.ec2.internal/metric_emitter, queue capacity = 100, org.opensearch.common.util.concurrent.OpenSearchThreadPoolExecutor@47e7c2f8[Running, pool size = 1, active threads = 1, queued tasks = 100, completed tasks = 2]]

And, when we took a look at the thread dump, we find that code is waiting on environmentFuture.join();

public void flush() {
        Environment environment;
        try {
            environment = environmentFuture.join();
        } catch (Exception ex) {
            log.info("Failed to resolve environment. Fallback to default environment: ", ex);
            environment = environmentProvider.getDefaultEnvironment();
        }
        ISink sink = environment.getSink();
        configureContextForEnvironment(context, environment);
        sink.accept(context);
        context = context.createCopyWithContext();
    }

Here is the thread dump which we took to understand why the thread was stuck.

"opensearch[ip-10-100-153-121.ec2.internal][metric_emitter][T#1]" #56 daemon prio=5 os_prio=0 cpu=18.74ms elapsed=15355.90s tid=0x00007fde5403b6d0 nid=0x106 waiting on condition  [0x00007fdde64d1000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x00000007c7a99b08> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
        at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1860)
        at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3137)
        at java.util.concurrent.CompletableFuture.waitingGet([email protected]/CompletableFuture.java:1887)
        at java.util.concurrent.CompletableFuture.join([email protected]/CompletableFuture.java:2107)
        at software.amazon.cloudwatchlogs.emf.logger.MetricsLogger.flush(MetricsLogger.java:58)
        at com.amazonaws.juno.metric.JunoMetricEMFEvent.emitMetrics(JunoMetricEMFEvent.java:28)
        at com.amazonaws.juno.metric.JunoMetricEmittingService$2.doRun(JunoMetricEmittingService.java:76)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:800)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:630)
        at java.lang.Thread.run([email protected]/Thread.java:832)

We have set AWS_EMF_AGENT_ENDPOINT environment variable and our fluent-bit configuration is listening to correct port that we setup. We were able to do this in one other package with exact same configurations. Are we missing any extra configuration that needs to be setup or is there something else that is going wrong ?

[markkuhn]

Hi @kogantihemanth. Can you try upgrading to the latest release and letting us know if you're still having this issue?

[markkuhn]

Closing for now. If you experience this error again please open a new issue.