X-Ray migration on Django: XRayTracedConn subclassing of psycopg's connection throws exception

[alexklemas]

After an update to my code base (my branch was 2 weeks behind), suddenly my working branch failed to build my core application's Docker container (using containers, all local, including the X-Ray Daemon), the output shows this:

Full output:

ALEXKLEMAS: docker_wrapper_recorder
ALEXKLEMAS: docker_XRAY mode has been enabled
ALEXKLEMAS: VERIFYING SERVICE PASSED ALONG: Project_WRAPPER_MDW
ALEXKLEMAS: docker_wrapper_recorder
ALEXKLEMAS: docker_XRAY mode has been enabled
ALEXKLEMAS: VERIFYING SERVICE PASSED ALONG: Project_WRAPPER_PRICER
ALEXKLEMAS: docker_wrapper_recorder
ALEXKLEMAS: docker_XRAY mode has been enabled
ALEXKLEMAS: VERIFYING SERVICE PASSED ALONG: Project_WRAPPER_COMMON
ALEXKLEMAS: docker_wrapper_recorder
ALEXKLEMAS: docker_XRAY mode has been enabled
ALEXKLEMAS: VERIFYING SERVICE PASSED ALONG: Project_WRAPPER_COLLECTORS
ALEXKLEMAS: docker_wrapper_recorder
ALEXKLEMAS: docker_XRAY mode has been enabled
ALEXKLEMAS: VERIFYING SERVICE PASSED ALONG: Project_WRAPPER_BASIC
ALEXKLEMAS: docker_wrapper_recorder
ALEXKLEMAS: docker_XRAY mode has been enabled
ALEXKLEMAS: VERIFYING SERVICE PASSED ALONG: Project_WRAPPER_ENVIRONMENT

Executing Django Command: /opt/app/work_project/work_company/project/manage.py migrate
Traceback (most recent call last):
File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/app/work_project/work_company/project/manage.py", line 39, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 85, in wrapped
res = handle_func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 92, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
self.loader = MigrationLoader(self.connection)
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 53, in __init__
self.build_graph()
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 216, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
if self.has_table():
File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 55, in has_table
with self.connection.cursor() as cursor:
File "/usr/local/lib/python3.6/site-packages/aws_xray_sdk/ext/django/db.py", line 81, in cursor
original_cursor = getattr(self, attr)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 235, in _cursor
self.ensure_connection()
File "/usr/local/lib/python3.6/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/usr/local/lib/python3.6/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python3.6/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 206, in get_new_connection
psycopg2.extras.register_default_jsonb(conn_or_curs=connection, loads=lambda x: x)
File "/usr/local/lib/python3.6/site-packages/psycopg2/_json.py", line 156, in register_default_jsonb
loads=loads, oid=JSONB_OID, array_oid=JSONBARRAY_OID, name='jsonb')
File "/usr/local/lib/python3.6/site-packages/psycopg2/_json.py", line 125, in register_json
register_type(JSON, not globally and conn_or_curs or None)
TypeError: argument 2 must be a connection, cursor or None

And upon finding this (still open) bug #148, tried to see if the subclassing was also False:

Python 3.6.9 (default, Jul 17 2020, 12:50:27) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from aws_xray_sdk.ext.dbapi2 import XRayTracedConn
>>> from psycopg2.extensions import connection
>>> issubclass(XRayTracedConn, connection)  
False
>>> exit()

I'm not sure, but it sure looks like the same issue, 2 years later, is still ongoing.

I read https://docs.aws.amazon.com/xray-sdk-for-python/latest/reference/frameworks.html#local-development and here's my Django config:

# ##### AWS XRAY Specifics ###############################

XRAY_RECORDER = {
    "AUTO_INSTRUMENT": True,
    "AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
    "AWS_XRAY_DAEMON_ADDRESS": "127.0.0.1:2000",
    "AWS_XRAY_TRACING_NAME": "Project_DJANGO",
    "PLUGINS": ("ECSPlugin", "ElasticBeanstalkPlugin"),
    "SAMPLING": False,
}

And my psycopg2 level is 2.8.5 (#148 reported value of 2.7.7)

This is blocking our usage of X-Ray (which was working just fine), has there been any progress on it, or a possible workaround (didn't quite get the workaround mentioned in #148)?

[lupengamzn]

Hi @alexklemas ,

Seems like it's an issue raised by the incompatibility of XRayTracedConn and psycopg2.extensions.connection due to wrapt is not properly subclassing the connection or cursor object.

Have you tried the method mentioned in #148 to mitigate this issue?

We'll need some further investigation on this and will get back to you soon

Thanks for the patience

[srprash]

Hi @alexklemas
I'm closing this one to de-dupe and tracking in the issue #243. Please find any updates there and feel free to add any comments. Thanks!