SyncRemoteRepository (PublicTask) fail when an Exception is raised

[humitos]

PublicTask is used to run a Celery Task and communicate the result to the FE. When the result is an Exception we are having an EncodeError since the exception is not being able to be encoded properly by kombu:

kombu.exceptions.EncodeError: Object of type 'Exception' is not JSON serializable

This error happens at https://github.com/rtfd/readthedocs.org/blob/eb6dd0b3ddf0c8dee32f0aa7c617eeaa78373e4f/readthedocs/core/utils/tasks/public.py#L83 when the info['error'] is populated.

(I found this by debugging for a while, the original exception that I saw was at https://github.com/rtfd/readthedocs.org/blob/eb6dd0b3ddf0c8dee32f0aa7c617eeaa78373e4f/readthedocs/core/utils/tasks/retrieve.py#L28 saying that EncodeError is not iterable)

I'm almost sure that this start happening when we upgrade to Celery 4 and the default serializer became json.

To reproduce this,

1. run runserver with celery (ALWAYS_EAGER=False)
2. raise an exception in GithubService.sync method
3. click on sync while importing a project

Related: #4076
Related: https://github.com/readthedocs/readthedocs-corporate/issues/323
Reported upstream: celery/kombu#573 (comment)

[humitos]

This is already fixed in #4078 with one solution but it's not the best: to be able to get info data, status and the exception raised we are marking a FAILED task as SUCCESS.

[xrmx]

Got this too, any idea on where the EncodeError comes from? Also this #4078 applied we have the /api/v2/jobs/status/ still returning this:

File "/readthedocs/restapi/views/task_views.py" in job_status
  46.         task_name, state, public_data, error = get_public_task_data(request, task_id)

File "/readthedocs/core/utils/tasks/public.py" in get_public_task_data
  130.         task, state, info = get_task_data(task_id)

File "/readthedocs/core/utils/tasks/retrieve.py" in get_task_data
  28.     if 'task_name' not in info:

Exception Type: TypeError at /api/v2/jobs/status/<task-id>/
Exception Value: argument of type 'EncodeError' is not iterable

[humitos]

@xrmx yes. The problem is with Celery 4 (it didn't happen with Celery 3) and it's because we try to encode the exception inside the task data. The PR linked in my previous comment fixes the problem in one way. Not the best one, though.

[stsewd]

I'll take a look at this after #4656

[stsewd]

Ok, I just saw the linked issue celery/kombu#868 and the PR #4078

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

This is pretty old and we are not facing this issue anymore.