SQS trace header fails for unsampled traces

[jimjshields]

We're using the XRay SDK for tracing our Django apps, which use SQS for asynchronous work. For the most part, it's sending traces of Django requests successfully and we can see them in the console.

However, we had an unsampled request that happened to interact with SQS, and it failed with this error:

ClientError("An error occurred (InvalidParameterValue) when calling the GetQueueUrl operation: The request has a 'X-Amzn-Trace-Id' HTTP header which is reserved for AWS X-Ray trace header and has an invalid value 'Root=dummy;Parent={real_parent_id};Sampled=0'",)

It seems like unsampled requests get the Root of dummy, which seems to cause this validation exception in SQS. Ideally, it'd either have a real ID, or that header wouldn't be injected for unsampled requests.

We have:

• python: 3.6.5
• aws-xray-sdk: 2.4.2
• boto3: 1.9.230
• botocore: 1.12.253
• django: 2.0.4

Here's our configuration in settings.py:

XRAY_RECORDER = {
    'AWS_XRAY_DAEMON_ADDRESS': f"{EC2_PRIVATE_IP}:{XRAY_PORT}",
    'AWS_XRAY_TRACING_NAME': xray_tracing_name,
    'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR',
    'PATCH_MODULES': [
        'boto3',
        'requests'
    ],
    'PLUGINS': ('EC2Plugin', 'ECSPlugin')
}

And the SDK is in our INSTALLED_APPS and MIDDLEWARE.

Let me know if you need more information. Thanks for the work on this library, I think it'll help us a lot!

[srprash]

Hi @jimjshields
Thanks for bringing up this issue. If a trace is unsampled, the python SDK marks the traceid as dummy which fails a backend validation.
We have figured out the root cause of the issue and are working on rolling out a fix soon. I'd request you to please stay tuned for any updates.

Thanks.

[jimjshields]

That's great, thanks @srprash! Looking forward to a fix.

[myyee]

@srprash Do you have an ETA for the fix? It is affecting our production. Thanks.

[bhautikpip]

Hi @myyee,

SQS is aware of this issue and is in the process of rolling out a fix.

[myyee]

Hi @bhautikpip ,
The X-Ray Python SDK crashed due to this underlying SQS issue. I ended up with nothing in the X-Ray because it fails to send the traces over. This is a totally hidden issue until someone informed me that things are not working correctly. Would you make an enhancement to the SDK so that it fails gracefully and still be able to send out traces to the X-Ray? Thanks.

[bhautikpip]

Hi @myyee,

Since this behavior is only visible in dummy segments. You might not have visibility of this issue on the console because XRay SDK never send out dummy segments to the daemon.

[bhautikpip]

Hello,

Sorry for the inconvenience because of this issue. SQS has informed us that the issue is fixed. Please let us know if you're still encountering this issue.

[srprash]

Hi @xrn
This issue was regarding a bug that was introduced as part of our integration with SQS, where if an instrumented SQS client was making a call and the trace is unsampled, the application breaks with the error mentioned in the original post. This was fixed on our end.
I believe the problem you're describing relates to this issue.

[sachdevajanit123]

I can still see the issue , If any of my requests isnot sampled
An error occurred (InvalidParameterValue) when calling the DeleteMessage operation: The request has a 'X-Amzn-Trace-Id' HTTP header which is reserved for AWS X-Ray trace header and has an invalid value 'Root=dummy;Parent=39d23fb90647b57f;Sampled=1'

[srprash]

Hi @sachdevajanit123 ,
What's the version of the SDK you're using? This issue was fixed in the v2.6.0