Notify users about the usage of deprecated webhooks

docs/webhooks.rst

@@ -36,6 +36,8 @@ As an example, the URL pattern looks like this: *readthedocs.org/api/v2/webhook/
 
 Use this URL when setting up a new webhook with your provider -- these steps vary depending on the provider:
 
+.. _webhook-integration-github:
+
 GitHub
 ~~~~~~
 
@@ -53,6 +55,8 @@ For a 403 error, it's likely that the Payload URL is incorrect.
 
 .. note:: The webhook token, intended for the GitHub **Secret** field, is not yet implemented.
 
+.. _webhook-integration-bitbucket:
+
 Bitbucket
 ~~~~~~~~~
 
@@ -63,6 +67,8 @@ Bitbucket
 * Under **Triggers**, **Repository push** should be selected
 * Finish by clicking **Save**
 
+.. _webhook-integration-gitlab:
+
 GitLab
 ~~~~~~
 
@@ -73,6 +79,8 @@ GitLab
 * Leave the default **Push events** selected and mark **Tag push events** also
 * Finish by clicking **Add Webhook**
 
+.. _webhook-integration-generic:
+
 Using the generic API integration
 ---------------------------------
 
@@ -136,3 +144,40 @@ Resyncing webhooks
 It might be necessary to re-establish a webhook if you are noticing problems.
 To resync a webhook from Read the Docs, visit the integration detail page and
 follow the directions for re-syncing your repository webhook.
+
+Troubleshooting
+---------------
+
+My project isn't automatically building
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If your project isn't automatically building, you can check your integration on
+Read the Docs to see the payload sent to our servers. If there is no recent
+activity on your Read the Docs project webhook integration, then it's likely
+that your VCS provider is not configured correctly. If there is payload
+information on your Read the Docs project, you might need to verify that your
+versions are configured to build correctly.
+
+Either way, it may help to either resync your webhook intergration (see
+`Resyncing webhooks`_ for information on this process), or set up an entirely
+new webhook intergration.
+
+.. _webhook-github-services:
+
+I was warned I shouldn't use GitHub Services
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Last year, GitHub announced that effective Jan 31st, 2019, GitHub Services will stop
+working [1]_. This means GitHub will stop sending notifications to Read the Docs
+for projects configured with the ``ReadTheDocs`` GitHub Service. If your project
+has been configured on Read the Docs for a long time, you are most likely still
+using this service to automatically build your project on Read the Docs.
+
+In order for your project to continue automatically building, you will need to
+configure your GitHub repository with a new webhook. You can use either a
+connected GitHub account and a :ref:`GitHub webhook integration <webhook-integration-github>`
+on your Read the Docs project, or you can use a
+:ref:`generic webhook integraiton <webhook-integration-generic>` without a connected
+account.
+
+.. [1] https://developer.github.com/changes/2018-04-25-github-services-deprecation/

readthedocs/core/views/hooks.py

@@ -17,6 +17,7 @@
 
 from readthedocs.builds.constants import LATEST
 from readthedocs.core.utils import trigger_build
+from readthedocs.notifications.decorators import notify_deprecated_endpoint
 from readthedocs.projects import constants
 from readthedocs.projects.models import Feature, Project
 from readthedocs.projects.tasks import sync_repository_task
@@ -126,6 +127,7 @@ def log_info(project, msg):
                      msg=msg))
 
 
+@notify_deprecated_endpoint
 def _build_url(url, projects, branches):
     """
     Map a URL onto specific projects to build that are linked to that URL.
@@ -341,6 +343,7 @@ def bitbucket_build(request):
 
 
 @csrf_exempt
+@notify_deprecated_endpoint
 def generic_build(request, project_id_or_slug=None):
     """
     Generic webhook build endpoint.

readthedocs/notifications/decorators.py

@@ -0,0 +1,51 @@
+# -*- coding: utf-8 -*-
+
+import logging
+
+from django.db.models import Q
+from django.http import HttpRequest
+
+from readthedocs.projects.models import Project
+from readthedocs.projects.notifications import DeprecatedWebhookEndpointNotification
+
+log = logging.getLogger(__name__)
+
+
+def notify_deprecated_endpoint(function):
+    """
+    Decorator to notify owners that the endpoint is DEPRECATED.
+
+    .. note::
+
+        See the class ``DeprecatedWebhookEndpointNotification`` which contains
+        all the logic to send notification for this messages properly without
+        spamming.
+    """
+    def wrap(*args, **kwargs):
+        # Called from ``generic_build``
+        project_id_or_slug = kwargs.get('project_id_or_slug')
+
+        if project_id_or_slug:
+            projects = Project.objects.filter(
+                Q(pk=project_id_or_slug) | Q(slug=project_id_or_slug),
+            )
+        else:
+            # Called from ``_build_url``
+            projects = args[1]  # ``projects`` argument
+
+        if projects:
+            for project in projects:
+                # Send one notification to each owner of the project
+                for user in project.users.all():
+                    notification = DeprecatedWebhookEndpointNotification(
+                        project,
+                        HttpRequest(),
+                        user,
+                    )
+                    notification.send()
+        else:
+            log.info('Projects not found when hitting deprecated webhook')
+
+        return function(*args, **kwargs)
+
+    return wrap

readthedocs/projects/notifications.py

@@ -1,6 +1,10 @@
+# -*- coding: utf-8 -*-
 """Project notifications"""
 
 from __future__ import absolute_import
+from datetime import timedelta
+from django.utils import timezone
+from messages_extends.models import Message
 from readthedocs.notifications import Notification
 from readthedocs.notifications.constants import REQUIREMENT
 
@@ -11,3 +15,69 @@ class ResourceUsageNotification(Notification):
     context_object_name = 'project'
     subject = 'Builds for {{ project.name }} are using too many resources'
     level = REQUIREMENT
+
+
+class DeprecatedWebhookEndpointNotification(Notification):
+
+    """
+    Notification for the usage of deprecated webhook endpoints.
+
+    Each time that a view decorated with ``notify_deprecated_endpoint`` is hit,
+    a new instance of this class is created. Then, ``.send`` is called and the
+    ``SiteBackend`` will create (avoiding duplication) a site notification and
+    the ``EmailBackend`` will do nothing (because of ``send_email=False``).
+
+    Besides, a ``message_extends.models.Message`` object is created to track
+    sending an email if this endpoint is hit again after ``email_period``. When,
+    ``.send`` is call and the ``email_period`` was reach from the
+    ``Message.created`` time we mark ``send_email=True`` in this instance and
+    call the super ``.send`` method that will effectively send the email and
+    mark the message as ``extra_tags='email_sent'``.
+    """
+
+    name = 'deprecated_webhook_endpoint'
+    context_object_name = 'project'
+    subject = '{{ project.name }} project webhook needs to be updated'
+    send_email = False
+    email_period = timedelta(days=7)
+    level = REQUIREMENT
+
+    def __init__(self, context_object, request, user=None):
+        super(DeprecatedWebhookEndpointNotification, self).__init__(
+            context_object,
+            request,
+            user,
+        )
+        self.message, created = self._create_message()
+
+        # Mark this notification to be sent as email the first time that it's
+        # created (user hits this endpoint for the first time)
+        if created:
+            self.send_email = True
+
+    def _create_message(self):
+        # Each time this class is instantiated we create a new Message (it's
+        # de-duplicated by using the ``message``, ``user`` and ``extra_tags``
+        # status)
+        return Message.objects.get_or_create(
+            message='{}: {}'.format(self.name, self.get_subject()),
+            level=self.level,
+            user=self.user,
+            extra_tags='email_delayed',
+        )
+
+    def send(self, *args, **kwargs):  # noqa
+        if self.message.created + self.email_period < timezone.now():
+            # Mark this instance to really send the email and rely on the
+            # EmailBackend to effectively send the email
+            self.send_email = True
+
+            # Mark the message as sent and send the email
+            self.message.extra_tags = 'email_sent'
+            self.message.save()
+
+            # Create a new Message with ``email_delayed`` so we are prepared to
+            # de-duplicate the following one
+            self._create_message()
+
+        super(DeprecatedWebhookEndpointNotification, self).send(*args, **kwargs)

readthedocs/rtd_tests/tests/test_notifications.py

@@ -2,16 +2,22 @@
 """Notification tests"""
 
 from __future__ import absolute_import
+from datetime import timedelta
 import mock
 import django_dynamic_fixture as fixture
+from django.http import HttpRequest
 from django.test import TestCase
 from django.test.utils import override_settings
 from django.contrib.auth.models import User, AnonymousUser
+from django.utils import timezone
+from freezegun import freeze_time
 from messages_extends.models import Message as PersistentMessage
 
 from readthedocs.notifications import Notification, SiteNotification
 from readthedocs.notifications.backends import EmailBackend, SiteBackend
 from readthedocs.notifications.constants import ERROR, INFO_NON_PERSISTENT, WARNING_NON_PERSISTENT
+from readthedocs.projects.models import Project
+from readthedocs.projects.notifications import DeprecatedWebhookEndpointNotification
 from readthedocs.builds.models import Build
 
 
@@ -222,3 +228,92 @@ def test_message(self):
         with mock.patch('readthedocs.notifications.notification.log') as mock_log:
             self.assertEqual(self.n.get_message(False), '')
             mock_log.error.assert_called_once()
+
+
+class DeprecatedWebhookEndpointNotificationTests(TestCase):
+
+    def setUp(self):
+        PersistentMessage.objects.all().delete()
+
+        self.project = fixture.get(Project)
+        self.user = fixture.get(User)
+        self.request = HttpRequest()
+
+        self.notification = DeprecatedWebhookEndpointNotification(
+            self.project,
+            self.request,
+            self.user,
+        )
+
+    def test_deduplication(self):
+        self.assertEqual(PersistentMessage.objects.filter(user=self.user).count(), 1)
+        for x in range(5):
+            DeprecatedWebhookEndpointNotification(
+                self.project,
+                self.request,
+                self.user,
+            )
+        self.assertEqual(PersistentMessage.objects.filter(user=self.user).count(), 1)
+        DeprecatedWebhookEndpointNotification(
+            self.project,
+            self.request,
+            fixture.get(User),
+        )
+        self.assertEqual(PersistentMessage.objects.count(), 2)
+        self.notification.message.extra_tags = 'email_sent'
+        self.notification.message.save()
+        DeprecatedWebhookEndpointNotification(
+            self.project,
+            self.request,
+            self.user,
+        )
+        self.assertEqual(PersistentMessage.objects.filter(user=self.user).count(), 2)
+        self.assertEqual(PersistentMessage.objects.count(), 3)
+
+    @mock.patch('readthedocs.notifications.backends.send_email')
+    def test_send_email(self, send_email):
+        # After creating the notification object, ``send_email=True``
+        self.assertTrue(self.notification.send_email)
+
+        # Calling ``.send`` on this instance will send the email and the
+        # ``Message`` will stay in ``email_delayed`` status.
+        self.notification.send()
+        self.notification.message.refresh_from_db()
+        self.assertEqual(self.notification.message.extra_tags, 'email_delayed')
+        self.assertTrue(send_email.called)
+        send_email.reset_mock()
+
+        # Hit the endpoint twice
+        for x in range(2):
+            notification = DeprecatedWebhookEndpointNotification(
+                self.project,
+                self.request,
+                self.user,
+            )
+        # A SiteNotification is created after calling ``.send`` so we filter for
+        # the message also to be sure that no new notification was created
+        self.assertEqual(PersistentMessage.objects.filter(
+            user=self.user,
+            message__startswith=DeprecatedWebhookEndpointNotification.name).count(),
+            1,
+        )
+        self.assertFalse(notification.send_email)  # second notification
+        notification.send()
+        self.assertFalse(send_email.called)
+        self.assertEqual(notification.message.extra_tags, 'email_delayed')
+
+        # In 7 days, when the ``.send`` method is called from a new created
+        # instance, this method will send the email and mark this ``Message`` as
+        # ``email_sent``
+        with freeze_time(timezone.now() + timedelta(days=7)):
+            notification.send()
+
+        self.assertTrue(send_email.called)
+        self.assertEqual(notification.message.extra_tags, 'email_sent')
+        # A new Message object is created
+        self.assertEqual(PersistentMessage.objects.filter(
+            user=self.user,
+            message__startswith=DeprecatedWebhookEndpointNotification.name).count(),
+            2,
+        )
+        self.assertEqual(PersistentMessage.objects.last().extra_tags, 'email_delayed')

readthedocs/templates/projects/notifications/deprecated_webhook_endpoint_email.html

@@ -0,0 +1,8 @@
+<p>Your project, {{ project.name }}, is currently using GitHub Services to trigger builds on Read the Docs. Effective January 31, 2019, GitHub will no longer process requests using the Services feature, and so Read the Docs will not receive notifications on updates to your repository.</p>
+
+<p>To continue building your Read the Docs project on changes to your repository, you will need to add a new webhook on your GitHub repository. You can either connect your GitHub account and configure a GitHub webhook integration, or you can add a generic webhook integration.</p>
+
+<p>You can find more information on our webhook intergrations in our documentation, at:</p>
+
+{% comment %}Plain text link because of text version of email{% endcomment %}
+<p><a href="https://docs.readthedocs.io/en/latest/webhooks.html">https://docs.readthedocs.io/en/latest/webhooks.html</a></p>

readthedocs/templates/projects/notifications/deprecated_webhook_endpoint_site.html

@@ -0,0 +1 @@
+Your project, {{ project.name }}, needs to be reconfigured in order to continue building automatically after Jan 31st. For more information, <a href="https://docs.readthedocs.io/en/latest/webhooks.html#webhook-github-services">see our documentation on webhook integrations</a>.

requirements/testing.txt

@@ -19,6 +19,8 @@ Mercurial==4.4.2
 yamale==1.8.0
 pytest-mock==1.10.0
 
+freezegun==0.3.11
+
 # local debugging tools
 datadiff
 ipdb