Bug: trace data compatible issue with AWS console UI

[singlewind]

Bug description

AWS console could not load the trace data

Making the trace data posted by tracer to be compatible by AWS console UI.

Expected Behavior

The json data has been loaded by AWS console UI and generate UI elements.

Current Behavior

AWS console UI shows "Unexpected error has occurred
We're unable to load segment's data. Try again later."

Possible Solution

Not sure whether this is the UI issue or trace data format issue. I have pulled the data trace data manually via aws cli. Sounds good to me. Don't know why console failed to load. Too many segments or the start_time and end_time format?

{ "Traces": [ { "Id": "1-61dc455f-6ea9978529a7aed4065a86cf", "Duration": 0.479, "LimitExceeded": false, "Segments": [ { "Id": "3fee87845621506a", "Document": "{\"id\":\"3fee87845621506a\",\"name\":\"sqs-lambda-handlerE1533BD5-akmebiECiMpA\",\"start_time\":1.641825631034E9,\"trace_id\":\"1-61dc455f-6ea9978529a7aed4065a86cf\",\"end_time\":1.641825631198E9,\"http\":{\"response\":{\"status\":200}},\"aws\":{\"request_id\":\"6a836865-974b-5013-9193-3233d0c185e4\"},\"origin\":\"AWS::Lambda\",\"resource_arn\":\"arn:aws:lambda:ap-southeast-2:610395700153:function:sqs-lambda-handlerE1533BD5-akmebiECiMpA\"}" }, { "Id": "0eff643e2830bbd2", "Document": "{\"id\":\"0eff643e2830bbd2\",\"name\":\"sqs-lambda-handlerE1533BD5-akmebiECiMpA\",\"start_time\":1.6418256310533319E9,\"trace_id\":\"1-61dc455f-6ea9978529a7aed4065a86cf\",\"end_time\":1.641825631513453E9,\"parent_id\":\"3fee87845621506a\",\"aws\":{\"account_id\":\"610395700153\",\"function_arn\":\"arn:aws:lambda:ap-southeast-2:610395700153:function:sqs-lambda-handlerE1533BD5-akmebiECiMpA\",\"resource_names\":[\"sqs-lambda-handlerE1533BD5-akmebiECiMpA\"]},\"origin\":\"AWS::Lambda::Function\",\"subsegments\":[{\"id\":\"62d806947df6f181\",\"name\":\"Invocation\",\"start_time\":1.6418256310533895E9,\"end_time\":1.6418256311972835E9,\"aws\":{\"function_arn\":\"arn:aws:lambda:ap-southeast-2:610395700153:function:sqs-lambda-handlerE1533BD5-akmebiECiMpA\"},\"subsegments\":[{\"id\":\"e3ee4d9cca8accd5\",\"name\":\"## batch\",\"start_time\":1.641825631054E9,\"end_time\":1.641825631196E9,\"annotations\":{\"ColdStart\":false,\"totalFaiedJobs\":1.0,\"Service\":\"sqs-lambda-invoke\",\"totalBatchSize\":50.0},\"subsegments\":[{\"id\":\"f0652d84169a996c\",\"name\":\"### f602c275-83c4-4e45-9825-bf9d6ea777fb\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"67b897e5e882a245\",\"name\":\"### 271ff700-c7b6-437e-bf6d-cd0ee6698a9e\",\"start_time\":1.641825631094E9,\"end_time\":1.641825631113E9},{\"id\":\"5e806469077d4542\",\"name\":\"### 8deca156-71a1-40a3-bdfe-1f1d0e237c47\",\"start_time\":1.641825631153E9,\"end_time\":1.641825631153E9},{\"id\":\"25bf93038f65a12c\",\"name\":\"### efda4f46-0825-426c-89d0-cd076d617120\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"a132489e76bf33c5\",\"name\":\"### 3f45022b-16f0-411c-858b-b47c0281f006\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"ad4508687ca26e6d\",\"name\":\"### eaa547ce-a28c-4b07-8381-af33fa5e1019\",\"start_time\":1.641825631113E9,\"end_time\":1.641825631113E9},{\"id\":\"0f095a0d6f0a2593\",\"name\":\"### 191b5466-916b-4309-9245-e252b4b336b9\",\"start_time\":1.641825631114E9,\"end_time\":1.641825631153E9},{\"id\":\"6553077afc5c9e6d\",\"name\":\"### 693f2a0c-3b79-4330-9370-02c98fc7b99d\",\"start_time\":1.641825631193E9,\"end_time\":1.641825631193E9},{\"id\":\"54b6dcbffa698cf9\",\"name\":\"### d5d3b9b6-9761-4896-93de-935d32146336\",\"start_time\":1.641825631196E9,\"end_time\":1.641825631196E9},{\"id\":\"a220867c48681e91\",\"name\":\"### dc74463a-0c2a-4918-8def-51c64c72fed2\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"ccb50bb7266f1bc9\",\"name\":\"### 01703b9b-0a7f-4b3e-8b43-abb7aa369e29\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"0358667981d35bff\",\"name\":\"### ad7b1c89-ebf1-4f4a-91a5-a112d9338b82\",\"start_time\":1.641825631193E9,\"end_time\":1.641825631194E9},{\"id\":\"cc36a18bd22777cb\",\"name\":\"### b16b92d3-b40f-4b16-b1db-cd8cc2262330\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"9f2a186cd7426b60\",\"name\":\"### 894f116d-1dc2-4dd6-ab25-34d9553971a8\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"b01913b2be67c344\",\"name\":\"### 9e9bbc82-5481-4296-8d68-d8dbc860e8a1\",\"start_time\":1.641825631196E9,\"end_time\":1.641825631196E9},{\"id\":\"9c67703bbcb5d57e\",\"name\":\"### a5591dc2-74ba-4c65-94ba-c3a3510b9b2e\",\"start_time\":1.641825631113E9,\"end_time\":1.641825631114E9},{\"id\":\"d42ffde662cbb653\",\"name\":\"### 15f23cf8-68dc-42a5-ba3e-f413c8569172\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"0879fb72a910fdae\",\"name\":\"### 45de1af6-e5e9-43f4-80e8-46537489a4ba\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"849889d9611adf66\",\"name\":\"### ef695279-5fd9-4a42-817d-581024a50e29\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"9013530cf8b7131a\",\"name\":\"### eb69290f-9c88-4bb5-9775-ed4519bab4c5\",\"start_time\":1.641825631153E9,\"end_time\":1.641825631153E9},{\"id\":\"3e925859d9b39b52\",\"name\":\"### e7b78285-5b4e-4c65-9ef2-d514b6d5b9d0\",\"start_time\":1.641825631193E9,\"end_time\":1.641825631193E9},{\"id\":\"0617a9e883751e51\",\"name\":\"### 078f03b1-73ed-4623-bffa-46c9fba3ac62\",\"start_time\":1.641825631196E9,\"end_time\":1.641825631196E9},{\"id\":\"a689ca5772c0ee60\",\"name\":\"### d41b583a-1695-4acd-a853-28c80accbd57\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"68c535dad839a28f\",\"name\":\"### 63f51f5a-da92-459b-bc8b-10876b35beb6\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"1462e0b751a752e2\",\"name\":\"### a5e1b9cb-b210-47b9-85e6-ba5588fa8874\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"4e4cc616b32f388e\",\"name\":\"### b2e6b873-7cff-49e4-bc12-20845ac08cac\",\"start_time\":1.641825631193E9,\"end_time\":1.641825631193E9},{\"id\":\"9b4419747a335e8d\",\"name\":\"### 359e2261-01c6-463f-9d33-46a587be084f\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"54b462ea9e163119\",\"name\":\"### 61b33fed-770f-42e8-8cca-44471007179a\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"866716491c53b445\",\"name\":\"### 67e8b264-3c5f-467e-bcd3-9c0914b76542\",\"start_time\":1.641825631153E9,\"end_time\":1.641825631154E9},{\"id\":\"0dc60489e159fc3f\",\"name\":\"### da5ba9cf-8494-4310-ac90-b8acf9925226\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"0f3ea5e5e7c2f7a6\",\"name\":\"### 84b1d20b-e3a3-46a8-96e7-10aec9fb06eb\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"7b700f2e29e6944a\",\"name\":\"### 5c18a5c5-3581-4f9f-a03b-4c6018a1b9db\",\"start_time\":1.641825631093E9,\"end_time\":1.641825631093E9},{\"id\":\"c1b4d5e404e1ad76\",\"name\":\"### 9bf3310a-4dc8-407b-8849-bdc4cdd4f870\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"e525867f043498ea\",\"name\":\"### b65cb8a7-0794-4e88-a2d7-16685e576553\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"a88a5d8722b775b8\",\"name\":\"### d575ca14-d7b5-4159-a9d6-27bb84495306\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631196E9},{\"id\":\"0f2346e2822ea804\",\"name\":\"### f838f989-95a3-44e0-9473-a492a96e7f46\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"e8044f139c2ea8a6\",\"name\":\"### 0c0d66f5-34c2-4ca5-a974-e77a3c37217d\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"e452cbd1d787b91c\",\"name\":\"### 2dcd6dc2-f463-4592-bf27-0de7105fb684\",\"start_time\":1.641825631193E9,\"end_time\":1.641825631193E9},{\"id\":\"135c59a90e06a768\",\"name\":\"### 2dca5657-b482-4260-a42f-f8d25f3bd225\",\"start_time\":1.641825631054E9,\"end_time\":1.641825631054E9},{\"id\":\"dd43e9d72d026fac\",\"name\":\"### b001fe75-90ee-4da9-a3ee-bdaa6f1396e5\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"bd17c5f02bc6f449\",\"name\":\"### 0b07a876-d1c4-4618-83e2-081f87b31ec0\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"55b6177a6a443c0f\",\"name\":\"### 7962ace0-4e40-48cb-bf1a-7fd48661183b\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"498e0e96ee5679a0\",\"name\":\"### ac57d4e0-2bf0-42a8-a5e7-afe0caf73b25\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"48df7db1ed34ae34\",\"name\":\"### fb2fa23c-5d8b-41ef-a5fe-f556a59e4e22\",\"start_time\":1.641825631154E9,\"end_time\":1.641825631193E9},{\"id\":\"744e1d25bbb49a34\",\"name\":\"### 4b8b7d1f-3a8d-4c79-827f-435672773955\",\"start_time\":1.641825631055E9,\"end_time\":1.641825631055E9},{\"id\":\"27f64dc72f0b6f23\",\"name\":\"### ee005a67-3a97-4e8d-8ff7-6ec71895d2df\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"08441777deb9ba06\",\"name\":\"### 6e5b35d2-81df-44fc-a7c8-4ef181bbb84b\",\"start_time\":1.641825631196E9,\"end_time\":1.641825631196E9},{\"id\":\"83d86acd9d1f31ba\",\"name\":\"### d0d7edba-6633-4641-bae9-53b0c0024ae3\",\"start_time\":1.641825631056E9,\"end_time\":1.641825631056E9},{\"id\":\"4d11b4b4d414e732\",\"name\":\"### 81512844-ccc8-4546-97f2-84e846707303\",\"start_time\":1.641825631194E9,\"end_time\":1.641825631194E9},{\"id\":\"5b418dbb9ba2b608\",\"name\":\"### 27e21671-638d-4694-8021-4133c8a0a3f3\",\"start_time\":1.641825631057E9,\"end_time\":1.641825631093E9}]}]},{\"id\":\"abb65bfb4a5f818d\",\"name\":\"Overhead\",\"start_time\":1.6418256311973307E9,\"end_time\":1.6418256315133874E9,\"aws\":{\"function_arn\":\"arn:aws:lambda:ap-southeast-2:610395700153:function:sqs-lambda-handlerE1533BD5-akmebiECiMpA\"}}]}" } ] } ], "UnprocessedTraceIds": [] }

[singlewind]

A screenshot can explain more things.

[dreamorosi]

Hello @singlewind thanks for reporting this. The trace data emitted by Tracer should already be compatible with the X-Ray service console, so if this is happening there's definitely an issue.

Would it be possible for you to share a minimal example of how you're using the library so that we can try to reproduce the issue and see what's happening?

[singlewind]

@dreamorosi , here is the code.

import { SQSHandler, SQSEvent, SQSBatchResponse, SQSRecord, Context, SQSBatchItemFailure } from 'aws-lambda'
import { Tracer } from '@aws-lambda-powertools/tracer'
import { Logger } from '@aws-lambda-powertools/logger'
import { Metrics, MetricUnits } from '@aws-lambda-powertools/metrics'
import { randomNumber } from './utils'

const serviceName = 'sqs-lambda-invoke'
const tracer = new Tracer({ serviceName })
const logger = new Logger({ serviceName })
const metrics = new Metrics({ serviceName, namespace: 'batchProcess' })

export const handler: SQSHandler = async (event: SQSEvent, context: Context) => {
  const parentSegment = tracer.getSegment()
  const processSegment = parentSegment.addNewSubsegment('## batch')
  tracer.setSegment(processSegment)
  tracer.annotateColdStart()
  tracer.addServiceNameAnnotation()
  tracer.putAnnotation('totalBatchSize', event.Records.length)

  metrics.addMetric('total', MetricUnits.Count, event.Records.length)

  const batchItemFailures: SQSBatchItemFailure[] = []

  event.Records.forEach((record) => {
    const workerSegment = processSegment.addNewSubsegment(`### ${ record.messageId }`)
    tracer.setSegment(workerSegment)

    const dice = randomNumber(100, 2000)
    if (dice > 1900) {
      batchItemFailures.push({ itemIdentifier: record.messageId })
      logger.error(`${record.messageId} failed`)
    } else {
      logger.info(`${record.messageId} processed`)
    }

    workerSegment.close()
    tracer.setSegment(processSegment)
  })

  tracer.putAnnotation('totalFaiedJobs', batchItemFailures.length)
  processSegment.close()
  tracer.setSegment(parentSegment)

  metrics.addMetric('failed', MetricUnits.Count, batchItemFailures.length)
  metrics.addMetric('succeed', MetricUnits.Count, event.Records.length - batchItemFailures.length)
  metrics.publishStoredMetrics()

  return {
    batchItemFailures,
  }
}

[dreamorosi]

@singlewind great, thanks for providing this. Please allow some time for us to investigate, the team will get back to you on this.

[singlewind]

@dreamorosi no problem at all. I've reviewed the data been generated and compared to the schema. No obvious things get my attention. Of cause, I have limited knowledge of X-Ray. That's why I also submit a ticket to our AWS support as well. To see what has been missed to generate the UI properly.

[singlewind]

Not sure whether this is relevant. function_arn and resource_names are not properties of aws object based on xray-segmentdocument-schema-v1.0.0.json.

[dreamorosi]

@dreamorosi no problem at all. I've reviewed the data been generated and compared to the schema. No obvious things get my attention. Of cause, I have limited knowledge of X-Ray. That's why I also submit a ticket to our AWS support as well. To see what has been missed to generate the UI properly.

Just to clarify, the JSON data that you shared above comes from the X-Ray service console? (see screenshot below):

And if so, the issue is that it's not rendering properly?

[singlewind]

@dreamorosi the data was pulled from aws xray batch-get-traces command. You just remind me something. After check, actually the x-ray console ui is working correctly. It is the cloudwatch console failed to render. Cloudwatch -> Application Monitoring -> Traces. If that case, it should be a bug exists in AWS Cloudwatch console then.

[singlewind]

We can close that issue now. Thanks @dreamorosi for your help. I will chase AWS Cloudwatch team to fix this issue. The library works nicely. One thing we are still looking forward to see improvement is more friendly for currency. At the moment, tracer object is singleton resource in an instance scope. It creates race situation when run code in currency. Thank you again.

[dreamorosi]

Thanks for the feedback about the concurrency, we'll keep it in mind and we'd appreciate if you could open an RFC issue on this repo to discuss the feature and potentially also expand on some examples of how you'd want to use it.

Regarding the subject of this issue I can confirm that I have been able to see the same issue on the X-Ray console while testing another Lambda function. For now I'll close the issue but if this appears again please don't hesitate to get in touch again and/or reopen the issue.