"failed to send data to X-Ray daemon." -> "Message too long"

[tobiasmcnulty]

We've enabled the record_full_document parameter for the pymongo patch method (https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/ext/pymongo/patch.py#L12-L14) and are seeing the below exception when attempting to record a large query/response from mongo.

Would it be possible to update the UDP emitter to truncate the data or split it up so we can see at least some of it in the console?

Many thanks!

failed to send data to X-Ray daemon.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aws_xray_sdk/core/emitters/udp_emitter.py", line 63, in _send_data
    self._port))
OSError: [Errno 90] Message too long

[tobiasmcnulty]

I should add, we're also seeing traces with no segments in the console, which is maybe related?

(ID has been obfuscated.)

[awssandra]

Hi tobiasmcnulty,

I'd suggest using the streaming threshold in the recorder first. The idea is that subsegments can be batched and sent independently of their segment, as long as they're complete. This can help alleviate the UDP size limit when working with large segment documents.

Relevant Note
More Notes

We have similar logic on Lambda, where we force every subsegment to send when complete

So, I'd expect something like the following to work...

from aws_xray_sdk.core import xray_recorder

xray_recorder.configure(
    //... configs here
    streaming_threshold=0
    //Magic number here, default is 30. Setting to 0 will force a subsegment to send as soon as it's closed.
)

Let me know if this helps at all.

Thanks!
Sandra

[dpoirier]

@awssandra It looks as if the streaming_threshold won't be used unless it's truthy:

aws-xray-sdk-python/aws_xray_sdk/core/recorder.py

Line 162 in 37e57bb

if streaming_threshold:

So I wouldn't think setting it to 0 would have any effect. Would setting it to 1 or -1 be almost as good?

This might be a bug - maybe it should be checking if streaming_threshold is not None.

(I work with @tobiasmcnulty.)

[awssandra]

Hi dpoirier,

That looks like a bug to me - zero should be an acceptable value (I am most knowledgeable of the Node SDK so here is the setting logic there).

Setting to "1" should work in a very similar fashion.
Feel free to open a PR to correct this - but either way I'll try to get this prioritized on my end.

Sorry for the issue,
Sandra

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.