Django integration failing on exception

[adamchainz]

I'm using Django 2.0 on Python 3.6, with a view that raises PermissionDenied, which Django will convert to a 403 response, the XRayMiddleware dies:

[ERROR]	2018-08-29T15:48:07.91Z	ec1dfd19-aba2-11e8-870f-1f51d52b6dae	Internal Server Error: /graphql/
Traceback (most recent call last):
File "/var/task/django/core/handlers/base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/var/task/django/views/generic/base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File <redacted>
raise PermissionDenied('Not authenticated')
django.core.exceptions.PermissionDenied: Not authenticated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/task/django/core/handlers/exception.py", line 35, in inner
response = get_response(request)
File "/var/task/django/core/handlers/base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/var/task/django/core/handlers/base.py", line 168, in process_exception_by_middleware
response = middleware_method(request, exception)
File "/var/task/aws_xray_sdk/ext/django/middleware.py", line 84, in process_exception
segment.put_http_meta(http.STATUS, 500)
File "/var/task/aws_xray_sdk/core/models/facade_segment.py", line 42, in put_http_meta
raise FacadeSegmentMutationException(MUTATION_UNSUPPORTED_MESSAGE)
aws_xray_sdk.core.exceptions.exceptions.FacadeSegmentMutationException: FacadeSegments cannot be mutated.

I tried to replicate it in the test suite, but this passes for me (using Python 2.7 rather than 3.X, because I only have 3.7 on my local machine and various parts of this test suite don't work on 3.7):

diff --git a/tests/ext/django/app/views.py b/tests/ext/django/app/views.py
index 6bb5edf..dd05ce3 100644
--- a/tests/ext/django/app/views.py
+++ b/tests/ext/django/app/views.py
@@ -1,5 +1,6 @@
 import sqlite3
 
+from django.core.exceptions import PermissionDenied
 from django.http import HttpResponse
 from django.conf.urls import url
 from django.views.generic import TemplateView
@@ -13,6 +14,10 @@ def ok(request):
     return HttpResponse(status=200)
 
 
+def denied(request):
+    raise PermissionDenied('Not allowed')
+
+
 def fault(request):
     {}['key']
 
@@ -24,11 +29,9 @@ def call_db(request):
     return HttpResponse(status=201)
 
 
-# def template(request):
-
-
 urlpatterns = [
     url(r'^200ok/$', ok, name='200ok'),
+    url(r'^403denied/$', denied, name='403denied'),
     url(r'^500fault/$', fault, name='500fault'),
     url(r'^call_db/$', call_db, name='call_db'),
     url(r'^template/$', IndexView.as_view(), name='template'),
diff --git a/tests/ext/django/test_middleware.py b/tests/ext/django/test_middleware.py
index 66e9648..d1eea72 100644
--- a/tests/ext/django/test_middleware.py
+++ b/tests/ext/django/test_middleware.py
@@ -1,5 +1,5 @@
 import django
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from django.test import TestCase
 
 from aws_xray_sdk.core import xray_recorder
@@ -42,6 +42,18 @@ class XRayTestCase(TestCase):
         assert request['client_ip'] == '127.0.0.1'
         assert response['status'] == 404
 
+    def test_denied(self):
+        self.client.get('/403denied/')
+        segment = xray_recorder.emitter.pop()
+        assert segment.error
+
+        request = segment.http['request']
+        response = segment.http['response']
+
+        assert request['method'] == 'GET'
+        assert request['client_ip'] == '127.0.0.1'
+        assert response['status'] == 403
+
     def test_fault(self):
         url = reverse('500fault')
         try:

It could be related to changes in Python 3.X, or Django 2.0 which isn't being tested here yet and I made #85 for.

[haotianw465]

Hi, one quick question, are you using any serverless framework to deploy your Django app to AWS Lambda?
aws_xray_sdk.core.exceptions.exceptions.FacadeSegmentMutationException: FacadeSegments cannot be mutated. The facade segments will only be created when the SDK detects it is running inside a Lambda container.

[adamchainz]

No we just wrap it with a small WSGI conversion layer adapted from what other frameworks use (apig-wsgi) and bundle a custom zip.

[chanchiem]

With the latest update (SDK 2.4.1) that allows middlewares to run on Lambda, I wonder if this issue has been fixed.

[millarm]

I think I've got an instance of this, from code that does raise Http404 in a view.

I think the bug is here:

    def process_exception(self, request, exception):
        """
        Add exception information and fault flag to the
        current segment.
        """
        segment = xray_recorder.current_segment()
        segment.put_http_meta(http.STATUS, 500)

        stack = stacktrace.get_stacktrace(limit=xray_recorder._max_trace_back)
        segment.add_exception(exception, stack)

Which fails if this is running in an AWS Lambda environment, as it's not dealing with the subsegment.

By analogy with the Flask middleware the code should be:

    def process_exception(self, request, exception):
        """
        Add exception information and fault flag to the
        current segment.
        """
        if not exception:
            return
        segment = None
        try:
            if self.in_lambda_ctx:
                segment = self._recorder.current_subsegment()
            else:
                segment = self._recorder.current_segment()
        except Exception:
            pass
        if not segment:
            return

        segment.put_http_meta(http.STATUS, 500)
        stack = stacktrace.get_stacktrace(limit=self._recorder._max_trace_back)
        segment.add_exception(exception, stack)
        if self.in_lambda_ctx:
            self._recorder.end_subsegment()
        else:
            self._recorder.end_segment()

But... I haven't yet written a test case to prove this!

[millarm]

The above is in aws_xray_sdk.ext.django.middleware.XRayMiddleware

[millarm]

Real time update: Workaround, replace the middleware with this middleware:

from aws_xray_sdk.ext.django.middleware import XRayMiddleware
from aws_xray_sdk.core.models import http
from aws_xray_sdk.core.utils import stacktrace


class PatchedXRayMiddleware(XRayMiddleware):

    def process_exception(self, request, exception):
        """
        Add exception information and fault flag to the
        current segment.
        """
        if not exception:
            return
        segment = None
        try:
            if self.in_lambda_ctx:
                segment = self._recorder.current_subsegment()
            else:
                segment = self._recorder.current_segment()
        except Exception:
            pass
        if not segment:
            return

        segment.put_http_meta(http.STATUS, 500)
        stack = stacktrace.get_stacktrace(limit=self._recorder._max_trace_back)
        segment.add_exception(exception, stack)
        if self.in_lambda_ctx:
            self._recorder.end_subsegment()
        else:
            self._recorder.end_segment()

[chanchiem]

Hey,

Thank you very much for the update and catching the issue! We will address this issue and create a pull request to fix it :)

[chanchiem]

The patch release has been pushed, so this change is now live on version 2.4.2! :)