diff --git a/docs/_static/images/webhooks-activity.png b/docs/_static/images/webhooks-activity.png new file mode 100644 index 00000000000..b8ad92a8446 Binary files /dev/null and b/docs/_static/images/webhooks-activity.png differ diff --git a/docs/_static/images/webhooks-events.png b/docs/_static/images/webhooks-events.png new file mode 100644 index 00000000000..26344db2464 Binary files /dev/null and b/docs/_static/images/webhooks-events.png differ diff --git a/docs/_static/images/webhooks-payload.png b/docs/_static/images/webhooks-payload.png new file mode 100644 index 00000000000..2534e873fbd Binary files /dev/null and b/docs/_static/images/webhooks-payload.png differ diff --git a/docs/_static/images/webhooks-secret.png b/docs/_static/images/webhooks-secret.png new file mode 100644 index 00000000000..0f14b1cd965 Binary files /dev/null and b/docs/_static/images/webhooks-secret.png differ diff --git a/docs/build-notifications.rst b/docs/build-notifications.rst index d57b726c12a..5e8522235e7 100644 --- a/docs/build-notifications.rst +++ b/docs/build-notifications.rst @@ -1,13 +1,13 @@ -Enabling Build Notifications -============================ +Build Notifications and Webhooks +================================ .. note:: - Currently we don't send notifications when - a :doc:`build from a pull request fails `. + Currently we don't send notifications or trigger webhooks + on :doc:`builds from pull requests `. -Using email ------------ +Email notifications +------------------- Read the Docs allows you to configure emails that can be sent on failing builds. This makes sure you know when your builds have failed. @@ -20,31 +20,259 @@ Take these steps to enable build notifications using email: You should now get notified by email when your builds fail! -Using webhook -------------- +Build Status Webhooks +--------------------- Read the Docs can also send webhooks when builds are triggered, successful or failed. Take these steps to enable build notifications using a webhook: -* Go to :guilabel:`Admin` > :guilabel:`Notifications` in your project. -* Fill in the **URL** field under the **New Webhook Notifications** heading -* Submit the form +* Go to :guilabel:`Admin` > :guilabel:`Webhooks` in your project. +* Fill in the **URL** field and select what events will trigger the webhook +* Modify the payload or leave the default (see below) +* Click on :guilabel:`Save` + +.. figure:: /_static/images/webhooks-events.png + :align: center + :alt: URL and events for a webhook + + URL and events for a webhook + +Every time one of the checked events triggers, +Read the Docs will send a POST request to your webhook URL. +The default payload will look like this: + +.. code-block:: json + + { + "event": "build:triggered", + "name": "docs", + "slug": "docs", + "version": "latest", + "commit": "2552bb609ca46865dc36401dee0b1865a0aee52d", + "build": "15173336", + "start_date": "2021-11-03T16:23:14", + "build_url": "https://readthedocs.org/projects/docs/builds/15173336/", + "docs_url": "https://docs.readthedocs.io/en/latest/" + } + +When a webhook is sent, a new entry will be added to the +"Recent Activity" table. By clicking on each individual entry, +you will see the server response, the webhook request, and the payload. + +.. figure:: /_static/images/webhooks-activity.png + :align: center + :alt: Activity of a webhook + + Activity of a webhook + +Custom payload examples +~~~~~~~~~~~~~~~~~~~~~~~ + +You can customize the payload of the webhook to suit your needs, +as long as it is valid JSON. Below you have a couple of examples, +and in the following section you will find all the available variables. + +.. figure:: /_static/images/webhooks-payload.png + :width: 80% + :align: center + :alt: Custom payload + + Custom payload -The project name, slug and its details for the build will be sent as POST request to your webhook URL: +Slack ++++++ .. code-block:: json + { + "attachments": [ { - "name": "Read the Docs", - "slug": "rtd", - "build": { - "id": 6321373, - "commit": "e8dd17a3f1627dd206d721e4be08ae6766fda40", - "state": "finished", - "success": false, - "date": "2017-02-15 20:35:54" - } + "color": "#db3238", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": "*Read the Docs build failed*" + } + }, + { + "type": "section", + "fields": [ + { + "type": "mrkdwn", + "text": "*Project*: <{{ project.url }}|{{ project.name }}>" + }, + { + "type": "mrkdwn", + "text": "*Version*: {{ version.name }} ({{ build.commit }})" + }, + { + "type": "mrkdwn", + "text": "*Build*: <{{ build.url }}|{{ build.id }}>" + } + ] + } + ] } + ] + } + +More information on `the Slack Incoming Webhooks documentation `_. + +Discord ++++++++ + +.. code-block:: json + + { + "username": "Read the Docs", + "content": "Read the Docs build failed", + "embeds": [ + { + "title": "Build logs", + "url": "{{ build.url }}", + "color": 15258703, + "fields": [ + { + "name": "*Project*", + "value": "{{ project.url }}", + "inline": true + }, + { + "name": "*Version*", + "value": "{{ version.name }} ({{ build.commit }})", + "inline": true + }, + { + "name": "*Build*", + "value": "{{ build.url }}" + } + ] + } + ] + } + +More information on `the Discord webhooks documentation `_. + +Variable substitutions reference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``{{ event }}`` + Event that triggered the webhook, one of ``build:triggered``, ``build:failed``, or ``build:passed``. + +``{{ build.id }}`` + Build ID. + +``{{ build.commit }}`` + Commit corresponding to the build, if present (otherwise ``""``). + +``{{ build.url }}`` + URL of the build, for example ``https://readthedocs.org/projects/docs/builds/15173336/``. + +``{{ build.docs_url }}`` + URL of the documentation corresponding to the build, + for example ``https://docs.readthedocs.io/en/latest/``. -You should now get notified on your webhook when your builds start and finish (failure/success)! +``{{ build.start_date }}`` + Start date of the build (UTC, ISO format), for example ``2021-11-03T16:23:14``. + +``{{ organization.name }}`` + Organization name (Commercial only). + +``{{ organization.slug }}`` + Organization slug (Commercial only). + +``{{ project.slug }}`` + Project slug. + +``{{ project.name }}`` + Project name. + +``{{ project.url }}`` + URL of the project :term:`dashboard`. + +``{{ version.slug }}`` + Version slug. + +``{{ version.name }}`` + Version name. + +Validating the payload +~~~~~~~~~~~~~~~~~~~~~~ + +After you add a new webhook, Read the Docs will generate a secret key for it +and uses it to generate a hash signature (HMAC-SHA256) for each payload +that is included in the ``X-Hub-Signature`` header of the request. + +.. figure:: /_static/images/webhooks-secret.png + :width: 80% + :align: center + :alt: Webhook secret + + Webhook secret + +We highly recommend using this signature +to verify that the webhook is coming from Read the Docs. +To do so, you can add some custom code on your server, +like this: + +.. code-block:: python + + import hashlib + import hmac + import os + + + def verify_signature(payload, request_headers): + """ + Verify that the signature of payload is the same as the one coming from request_headers. + """ + digest = hmac.new( + key=os.environ["WEBHOOK_SECRET"].encode(), + msg=payload.encode(), + digestmod=hashlib.sha256, + ) + expected_signature = digest.hexdigest() + + return hmac.compare_digest( + request_headers["X-Hub-Signature"].encode(), + expected_signature.encode(), + ) + +Legacy webhooks +~~~~~~~~~~~~~~~ + +Webhooks created before the custom payloads functionality was added to Read the Docs +send a payload with the following structure: + +.. code-block:: json + + { + "name": "Read the Docs", + "slug": "rtd", + "build": { + "id": 6321373, + "commit": "e8dd17a3f1627dd206d721e4be08ae6766fda40", + "state": "finished", + "success": false, + "date": "2017-02-15 20:35:54" + } + } + +To migrate to the new webhooks and keep a similar structure, +you can use this payload: + +.. code-block:: json + + { + "name": "{{ project.name }}", + "slug": "{{ project.slug }}", + "build": { + "id": "{{ build.id }}", + "commit": "{{ build.commit }}", + "state": "{{ event }}", + "date": "{{ build.start_date }}" + } + } diff --git a/readthedocs/__init__.py b/readthedocs/__init__.py index 8f8c9ee7c80..198abfeea67 100644 --- a/readthedocs/__init__.py +++ b/readthedocs/__init__.py @@ -1,5 +1,3 @@ -# -*- coding: utf-8 -*- - """Read the Docs.""" import os.path diff --git a/readthedocs/builds/tasks.py b/readthedocs/builds/tasks.py index edc38c75ab7..f2397e07801 100644 --- a/readthedocs/builds/tasks.py +++ b/readthedocs/builds/tasks.py @@ -3,9 +3,13 @@ from datetime import datetime, timedelta from io import BytesIO +import requests from celery import Task from django.conf import settings +from django.urls import reverse +from django.utils.translation import ugettext_lazy as _ +from readthedocs import __version__ from readthedocs.api.v2.serializers import BuildSerializer from readthedocs.api.v2.utils import ( delete_versions_from_db, @@ -25,11 +29,11 @@ from readthedocs.builds.models import Build, Version from readthedocs.builds.utils import memcache_lock from readthedocs.core.permissions import AdminPermission -from readthedocs.core.utils import trigger_build -from readthedocs.oauth.models import RemoteRepository +from readthedocs.core.utils import send_email, trigger_build +from readthedocs.integrations.models import HttpExchange from readthedocs.oauth.notifications import GitBuildStatusFailureNotification from readthedocs.projects.constants import GITHUB_BRAND, GITLAB_BRAND -from readthedocs.projects.models import Project +from readthedocs.projects.models import Project, WebHookEvent from readthedocs.storage import build_commands_storage from readthedocs.worker import app @@ -453,3 +457,178 @@ def send_build_status(build_pk, commit, status, link_to_build=False): build.project.slug ) return False + + +@app.task(queue='web') +def send_build_notifications(version_pk, build_pk, event): + version = Version.objects.get_object_or_log(pk=version_pk) + if not version or version.type == EXTERNAL: + return + + build = Build.objects.filter(pk=build_pk).first() + if not build: + return + + sender = BuildNotificationSender( + version=version, + build=build, + event=event, + ) + sender.send() + + +class BuildNotificationSender: + + webhook_timeout = 2 + + def __init__(self, version, build, event): + self.version = version + self.build = build + self.project = version.project + self.event = event + + def send(self): + """ + Send email and webhook notifications for `project` about the `build`. + + Email notifications are only send for build:failed events. + Webhooks choose to what events they subscribe to. + """ + if self.event == WebHookEvent.BUILD_FAILED: + email_addresses = ( + self.project.emailhook_notifications.all() + .values_list('email', flat=True) + ) + for email in email_addresses: + try: + self.send_email(email) + except Exception: + log.exception( + 'Failed to send email notification. ' + 'email=%s project=%s version=%s build=%s', + email, self.project.slug, self.version.slug, self.build.pk, + ) + + webhooks = ( + self.project.webhook_notifications + .filter(events__name=self.event) + ) + for webhook in webhooks: + try: + self.send_webhook(webhook) + except Exception: + log.exception( + 'Failed to send webhook. webhook=%s project=%s version=%s build=%s', + webhook.id, self.project.slug, self.version.slug, self.build.pk, + ) + + def send_email(self, email): + """Send email notifications for build failures.""" + # We send only what we need from the Django model objects here to avoid + # serialization problems in the ``readthedocs.core.tasks.send_email_task`` + protocol = 'http' if settings.DEBUG else 'https' + context = { + 'version': { + 'verbose_name': self.version.verbose_name, + }, + 'project': { + 'name': self.project.name, + }, + 'build': { + 'pk': self.build.pk, + 'error': self.build.error, + }, + 'build_url': '{}://{}{}'.format( + protocol, + settings.PRODUCTION_DOMAIN, + self.build.get_absolute_url(), + ), + 'unsubscribe_url': '{}://{}{}'.format( + protocol, + settings.PRODUCTION_DOMAIN, + reverse('projects_notifications', args=[self.project.slug]), + ), + } + + if self.build.commit: + title = _('Failed: {project[name]} ({commit})').format( + commit=self.build.commit[:8], + **context, + ) + else: + title = _('Failed: {project[name]} ({version[verbose_name]})').format( + **context + ) + + log.info( + 'Sending email notification. email=%s project=%s version=%s build=%s', + email, self.project.slug, self.version.slug, self.build.id, + ) + send_email( + email, + title, + template='projects/email/build_failed.txt', + template_html='projects/email/build_failed.html', + context=context, + ) + + def send_webhook(self, webhook): + """ + Send webhook notification. + + The payload is signed using HMAC-SHA256, + for users to be able to verify the authenticity of the request. + + Webhooks that don't have a payload, + are from the old implementation, for those we keep sending the + old default payload. + + An HttpExchange object is created for each transaction. + """ + payload = webhook.get_payload( + version=self.version, + build=self.build, + event=self.event, + ) + if not payload: + # Default payload from old webhooks. + payload = json.dumps({ + 'name': self.project.name, + 'slug': self.project.slug, + 'build': { + 'id': self.build.id, + 'commit': self.build.commit, + 'state': self.build.state, + 'success': self.build.success, + 'date': self.build.date.strftime('%Y-%m-%d %H:%M:%S'), + }, + }) + + headers = { + 'content-type': 'application/json', + 'User-Agent': f'Read-the-Docs/{__version__} ({settings.PRODUCTION_DOMAIN})', + 'X-RTD-Event': self.event, + } + if webhook.secret: + headers['X-Hub-Signature'] = webhook.sign_payload(payload) + + try: + log.info( + 'Sending webhook notification. webhook=%s project=%s version=%s build=%s', + webhook.pk, self.project.slug, self.version.slug, self.build.pk, + ) + response = requests.post( + webhook.url, + data=payload, + headers=headers, + timeout=self.webhook_timeout, + ) + HttpExchange.objects.from_requests_exchange( + response=response, + related_object=webhook, + ) + except Exception: + log.exception( + 'Failed to POST to webhook url. webhook=%s url=%s', + webhook.pk, webhook.url, + ) diff --git a/readthedocs/core/utils/__init__.py b/readthedocs/core/utils/__init__.py index 094d99a3e28..6ef577877ed 100644 --- a/readthedocs/core/utils/__init__.py +++ b/readthedocs/core/utils/__init__.py @@ -56,10 +56,10 @@ def prepare_build( """ # Avoid circular import from readthedocs.builds.models import Build - from readthedocs.projects.models import Feature, Project + from readthedocs.builds.tasks import send_build_notifications + from readthedocs.projects.models import Feature, Project, WebHookEvent from readthedocs.projects.tasks import ( send_external_build_status, - send_notifications, update_docs_task, ) @@ -123,8 +123,12 @@ def prepare_build( ) if build and version.type != EXTERNAL: - # Send Webhook notification for build triggered. - send_notifications.delay(version.pk, build_pk=build.pk, email=False) + # Send notifications for build triggered. + send_build_notifications.delay( + version_pk=version.pk, + build_pk=build.pk, + event=WebHookEvent.BUILD_TRIGGERED, + ) options['priority'] = CELERY_HIGH if project.main_language_project: diff --git a/readthedocs/integrations/models.py b/readthedocs/integrations/models.py index f8daf557385..8ae5d9c43e3 100644 --- a/readthedocs/integrations/models.py +++ b/readthedocs/integrations/models.py @@ -1,5 +1,3 @@ -# -*- coding: utf-8 -*- - """Integration models for external services.""" import json @@ -96,6 +94,25 @@ def from_exchange(self, req, resp, related_object, payload=None): self.delete_limit(related_object) return obj + def from_requests_exchange(self, response, related_object): + """ + Create an exchange object from a requests' response. + + :param response: The result from calling request.post() or similar. + :param related_object: Object to use for generic relationship. + """ + request = response.request + obj = self.create( + related_object=related_object, + request_headers=request.headers or {}, + request_body=request.body or '', + status_code=response.status_code, + response_headers=response.headers, + response_body=response.text, + ) + self.delete_limit(related_object) + return obj + def delete_limit(self, related_object, limit=10): if isinstance(related_object, Integration): queryset = self.filter(integrations=related_object) diff --git a/readthedocs/projects/admin.py b/readthedocs/projects/admin.py index b1154470085..3d34929e8d0 100644 --- a/readthedocs/projects/admin.py +++ b/readthedocs/projects/admin.py @@ -21,12 +21,13 @@ EmailHook, EnvironmentVariable, Feature, - HTTPHeader, HTMLFile, + HTTPHeader, ImportedFile, Project, ProjectRelationship, WebHook, + WebHookEvent, ) from .notifications import ( DeprecatedBuildWebhookNotification, @@ -480,4 +481,5 @@ class EnvironmentVariableAdmin(admin.ModelAdmin): admin.site.register(Feature, FeatureAdmin) admin.site.register(EmailHook) admin.site.register(WebHook) +admin.site.register(WebHookEvent) admin.site.register(HTMLFile, ImportedFileAdmin) diff --git a/readthedocs/projects/forms.py b/readthedocs/projects/forms.py index d1f854d92b0..63e2a88837d 100644 --- a/readthedocs/projects/forms.py +++ b/readthedocs/projects/forms.py @@ -1,4 +1,5 @@ """Project forms.""" +import json from random import choice from re import fullmatch from urllib.parse import urlparse @@ -444,23 +445,49 @@ def save(self): class WebHookForm(forms.ModelForm): - """Project webhook form.""" + project = forms.CharField(widget=forms.HiddenInput(), required=False) + + class Meta: + model = WebHook + fields = ['project', 'url', 'events', 'payload', 'secret'] + widgets = { + 'events': forms.CheckboxSelectMultiple, + } def __init__(self, *args, **kwargs): self.project = kwargs.pop('project', None) super().__init__(*args, **kwargs) - def save(self, commit=True): - self.webhook = WebHook.objects.get_or_create( - url=self.cleaned_data['url'], - project=self.project, - )[0] - self.project.webhook_notifications.add(self.webhook) + if self.instance and self.instance.pk: + # Show secret in the detail form, but as readonly. + self.fields['secret'].disabled = True + else: + # Don't show the secret in the creation form. + self.fields.pop('secret') + self.fields['payload'].initial = json.dumps({ + 'event': '{{ event }}', + 'name': '{{ project.name }}', + 'slug': '{{ project.slug }}', + 'version': '{{ version.slug }}', + 'commit': '{{ build.commit }}', + 'build': '{{ build.id }}', + 'start_date': '{{ build.start_date }}', + 'build_url': '{{ build.url }}', + 'docs_url': '{{ build.docs_url }}', + }, indent=2) + + def clean_project(self): return self.project - class Meta: - model = WebHook - fields = ['url'] + def clean_payload(self): + """Check if the payload is a valid json object and format it.""" + payload = self.cleaned_data['payload'] + try: + payload = json.loads(payload) + payload = json.dumps(payload, indent=2) + except Exception: + raise forms.ValidationError(_('The payload must be a valid JSON object.')) + return payload class TranslationBaseForm(forms.Form): diff --git a/readthedocs/projects/migrations/0083_init_generic_webhooks.py b/readthedocs/projects/migrations/0083_init_generic_webhooks.py new file mode 100644 index 00000000000..3f1d6beb89a --- /dev/null +++ b/readthedocs/projects/migrations/0083_init_generic_webhooks.py @@ -0,0 +1,64 @@ +# Generated by Django 2.2.24 on 2021-09-27 18:11 + +from django.db import migrations, models +import django.utils.timezone +import django_extensions.db.fields + + +class Migration(migrations.Migration): + + dependencies = [ + ('projects', '0082_add_extra_history_fields'), + ] + + operations = [ + migrations.CreateModel( + name='WebHookEvent', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(choices=[('build:triggered', 'Build triggered'), ('build:passed', 'Build passed'), ('build:failed', 'Build failed')], max_length=256, unique=True)), + ], + ), + migrations.AddField( + model_name='emailhook', + name='created', + field=django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, default=django.utils.timezone.now, verbose_name='created', null=True), + preserve_default=False, + ), + migrations.AddField( + model_name='emailhook', + name='modified', + field=django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified', null=True), + ), + migrations.AddField( + model_name='webhook', + name='created', + field=django_extensions.db.fields.CreationDateTimeField(auto_now_add=True, default=django.utils.timezone.now, verbose_name='created', null=True), + preserve_default=False, + ), + migrations.AddField( + model_name='webhook', + name='modified', + field=django_extensions.db.fields.ModificationDateTimeField(auto_now=True, verbose_name='modified', null=True), + ), + migrations.AddField( + model_name='webhook', + name='payload', + field=models.TextField(blank=True, help_text='JSON payload to send to the webhook. Check the docs for available substitutions.', max_length=25000, null=True, verbose_name='JSON payload'), + ), + migrations.AddField( + model_name='webhook', + name='secret', + field=models.CharField(blank=True, help_text='Secret used to sign the payload of the webhook', max_length=255, null=True), + ), + migrations.AlterField( + model_name='webhook', + name='url', + field=models.URLField(help_text='URL to send the webhook to', max_length=600, verbose_name='URL'), + ), + migrations.AddField( + model_name='webhook', + name='events', + field=models.ManyToManyField(help_text='Events to subscribe', related_name='webhooks', to='projects.WebHookEvent'), + ), + ] diff --git a/readthedocs/projects/migrations/0084_create_webhook_events.py b/readthedocs/projects/migrations/0084_create_webhook_events.py new file mode 100644 index 00000000000..738b68fd4c0 --- /dev/null +++ b/readthedocs/projects/migrations/0084_create_webhook_events.py @@ -0,0 +1,21 @@ +# Generated by Django 2.2.24 on 2021-09-27 20:43 + +from django.db import migrations + + +def forwards_func(apps, schema_editor): + """Create supported events for webhooks.""" + WebHookEvent = apps.get_model('projects', 'WebHookEvent') + for event in ['build:triggered', 'build:failed', 'build:passed']: + WebHookEvent.objects.get_or_create(name=event) + + +class Migration(migrations.Migration): + + dependencies = [ + ('projects', '0083_init_generic_webhooks'), + ] + + operations = [ + migrations.RunPython(forwards_func), + ] diff --git a/readthedocs/projects/migrations/0085_subscribe_old_webhooks_to_events.py b/readthedocs/projects/migrations/0085_subscribe_old_webhooks_to_events.py new file mode 100644 index 00000000000..36b9d9819df --- /dev/null +++ b/readthedocs/projects/migrations/0085_subscribe_old_webhooks_to_events.py @@ -0,0 +1,29 @@ +# Generated by Django 2.2.24 on 2021-09-28 19:44 + +from django.db import migrations + + +def forwards_func(apps, schema_editor): + """Migrate old webhooks to subscribe to events instead.""" + WebHook = apps.get_model('projects', 'WebHook') + WebHookEvent = apps.get_model('projects', 'WebHookEvent') + old_webhooks = WebHook.objects.filter(events__isnull=True) + default_events = [ + WebHookEvent.objects.get(name='build:triggered'), + WebHookEvent.objects.get(name='build:passed'), + WebHookEvent.objects.get(name='build:failed'), + ] + for webhook in old_webhooks: + webhook.events.set(default_events) + webhook.save() + + +class Migration(migrations.Migration): + + dependencies = [ + ('projects', '0084_create_webhook_events'), + ] + + operations = [ + migrations.RunPython(forwards_func), + ] diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py index 96f3fb1e006..5ee7744c2da 100644 --- a/readthedocs/projects/models.py +++ b/readthedocs/projects/models.py @@ -1,5 +1,7 @@ """Project models.""" import fnmatch +import hashlib +import hmac import logging import os import re @@ -10,14 +12,19 @@ from django.conf import settings from django.conf.urls import include from django.contrib.auth.models import User +from django.contrib.contenttypes.fields import GenericRelation from django.core.validators import MaxValueValidator, MinValueValidator from django.db import models from django.db.models import Prefetch from django.urls import re_path, reverse +from django.utils.crypto import get_random_string from django.utils.functional import cached_property from django.utils.translation import ugettext_lazy as _ from django.views import defaults -from django_extensions.db.fields import CreationDateTimeField +from django_extensions.db.fields import ( + CreationDateTimeField, + ModificationDateTimeField, +) from django_extensions.db.models import TimeStampedModel from taggit.managers import TaggableManager @@ -1450,7 +1457,20 @@ def processed_json(self): return self.get_processed_json() -class Notification(models.Model): +class Notification(TimeStampedModel): + # TODO: Overridden from TimeStampedModel just to allow null values, + # remove after deploy. + created = CreationDateTimeField( + _('created'), + null=True, + blank=True, + ) + modified = ModificationDateTimeField( + _('modified'), + null=True, + blank=True, + ) + project = models.ForeignKey( Project, related_name='%(class)s_notifications', @@ -1469,14 +1489,137 @@ def __str__(self): return self.email +class WebHookEvent(models.Model): + + BUILD_TRIGGERED = 'build:triggered' + BUILD_PASSED = 'build:passed' + BUILD_FAILED = 'build:failed' + + EVENTS = ( + (BUILD_TRIGGERED, _('Build triggered')), + (BUILD_PASSED, _('Build passed')), + (BUILD_FAILED, _('Build failed')), + ) + + name = models.CharField( + max_length=256, + unique=True, + choices=EVENTS, + ) + + def __str__(self): + return self.name + + class WebHook(Notification): + url = models.URLField( + _('URL'), max_length=600, help_text=_('URL to send the webhook to'), ) + secret = models.CharField( + help_text=_('Secret used to sign the payload of the webhook'), + max_length=255, + blank=True, + null=True, + ) + events = models.ManyToManyField( + WebHookEvent, + related_name='webhooks', + help_text=_('Events to subscribe'), + ) + payload = models.TextField( + _('JSON payload'), + help_text=_( + 'JSON payload to send to the webhook. ' + 'Check the docs for available substitutions.', # noqa + ), + blank=True, + null=True, + max_length=25000, + ) + exchanges = GenericRelation( + 'integrations.HttpExchange', + related_query_name='webhook', + ) + + def save(self, *args, **kwargs): + if not self.secret: + self.secret = get_random_string(length=32) + super().save(*args, **kwargs) + + def get_payload(self, version, build, event): + """ + Get the final payload replacing all placeholders. + + Placeholders are in the ``{{ foo }}`` or ``{{foo}}`` format. + """ + if not self.payload: + return None + + project = version.project + organization = project.organizations.first() + + organization_name = '' + organization_slug = '' + if organization: + organization_slug = organization.slug + organization_name = organization.name + + # Commit can be None, display an empty string instead. + commit = build.commit or '' + protocol = 'http' if settings.DEBUG else 'https' + project_url = f'{protocol}://{settings.PRODUCTION_DOMAIN}{project.get_absolute_url()}' + build_url = f'{protocol}://{settings.PRODUCTION_DOMAIN}{build.get_absolute_url()}' + build_docsurl = project.get_docs_url( + version_slug=version.slug, + external=version.is_external, + ) + + # Remove timezone and microseconds from the date, + # so it's more readable. + start_date = build.date.replace( + tzinfo=None, + microsecond=0 + ).isoformat() + + substitutions = { + 'event': event, + 'build.id': build.id, + 'build.commit': commit, + 'build.url': build_url, + 'build.docs_url': build_docsurl, + 'build.start_date': start_date, + 'organization.name': organization_name, + 'organization.slug': organization_slug, + 'project.slug': project.slug, + 'project.name': project.name, + 'project.url': project_url, + 'version.slug': version.slug, + 'version.name': version.verbose_name, + } + payload = self.payload + # Small protection for DDoS. + max_substitutions = 99 + for substitution, value in substitutions.items(): + # Replace {{ foo }}. + payload = payload.replace(f'{{{{ {substitution} }}}}', str(value), max_substitutions) + # Replace {{foo}}. + payload = payload.replace(f'{{{{{substitution}}}}}', str(value), max_substitutions) + return payload + + def sign_payload(self, payload): + """Get the signature of `payload` using HMAC-SHA1 with the webhook secret.""" + digest = hmac.new( + key=self.secret.encode(), + msg=payload.encode(), + digestmod=hashlib.sha256, + ) + return digest.hexdigest() def __str__(self): - return self.url + return f'{self.project.slug} {self.url}' class Domain(TimeStampedModel, models.Model): diff --git a/readthedocs/projects/tasks.py b/readthedocs/projects/tasks.py index 26215e12de7..ff24ede494a 100644 --- a/readthedocs/projects/tasks.py +++ b/readthedocs/projects/tasks.py @@ -17,11 +17,9 @@ from collections import Counter, defaultdict from fnmatch import fnmatch -import requests from celery.exceptions import SoftTimeLimitExceeded from django.conf import settings from django.db.models import Q -from django.urls import reverse from django.utils import timezone from django.utils.translation import ugettext_lazy as _ from slumber.exceptions import HttpClientError @@ -43,7 +41,6 @@ from readthedocs.builds.models import APIVersion, Build, Version from readthedocs.builds.signals import build_complete from readthedocs.config import ConfigError -from readthedocs.core.utils import send_email from readthedocs.doc_builder.config import load_yaml_config from readthedocs.doc_builder.environments import ( DockerBuildEnvironment, @@ -60,7 +57,7 @@ ) from readthedocs.doc_builder.loader import get_builder_class from readthedocs.doc_builder.python_environments import Conda, Virtualenv -from readthedocs.projects.models import APIProject, Feature +from readthedocs.projects.models import APIProject, Feature, WebHookEvent from readthedocs.projects.signals import ( after_build, before_build, @@ -640,7 +637,11 @@ def run( self.setup_env.update_build(BUILD_STATE_FINISHED) # Send notifications for unhandled errors - self.send_notifications(version_pk, build_pk, email=True) + self.send_notifications( + version_pk, + build_pk, + event=WebHookEvent.BUILD_FAILED, + ) return False def run_setup(self, record=True): @@ -715,7 +716,11 @@ def run_setup(self, record=True): # triggered before the previous one has finished (e.g. two webhooks, # one after the other) if not isinstance(environment.failure, VersionLockedError): - self.send_notifications(self.version.pk, self.build['id'], email=True) + self.send_notifications( + self.version.pk, + self.build['id'], + event=WebHookEvent.BUILD_FAILED, + ) return False @@ -823,7 +828,11 @@ def run_build(self, record): if self.build_env.failed: # Send Webhook and email notification for build failure. - self.send_notifications(self.version.pk, self.build['id'], email=True) + self.send_notifications( + self.version.pk, + self.build['id'], + event=WebHookEvent.BUILD_FAILED, + ) if self.commit: send_external_build_status( @@ -834,7 +843,11 @@ def run_build(self, record): ) elif self.build_env.successful: # Send Webhook notification for build success. - self.send_notifications(self.version.pk, self.build['id'], email=False) + self.send_notifications( + self.version.pk, + self.build['id'], + event=WebHookEvent.BUILD_PASSED, + ) # Push cached environment on success for next build self.push_cached_environment() @@ -1321,12 +1334,16 @@ def build_docs_class(self, builder_class): builder.move() return success - def send_notifications(self, version_pk, build_pk, email=False): - """Send notifications on build failure.""" + def send_notifications(self, version_pk, build_pk, event): + """Send notifications to all subscribers of `event`.""" # Try to infer the version type if we can # before creating a task. if not self.version or self.version.type != EXTERNAL: - send_notifications.delay(version_pk, build_pk=build_pk, email=email) + build_tasks.send_build_notifications.delay( + version_pk=version_pk, + build_pk=build_pk, + event=event, + ) def is_type_sphinx(self): """Is documentation type Sphinx.""" @@ -1652,123 +1669,6 @@ def _sync_imported_files(version, build): ) -@app.task(queue='web') -def send_notifications(version_pk, build_pk, email=False): - version = Version.objects.get_object_or_log(pk=version_pk) - - if not version or version.type == EXTERNAL: - return - - build = Build.objects.get(pk=build_pk) - - for hook in version.project.webhook_notifications.all(): - webhook_notification(version, build, hook.url) - - if email: - for email_address in version.project.emailhook_notifications.all().values_list( - 'email', - flat=True, - ): - email_notification(version, build, email_address) - - -def email_notification(version, build, email): - """ - Send email notifications for build failure. - - :param version: :py:class:`Version` instance that failed - :param build: :py:class:`Build` instance that failed - :param email: Email recipient address - """ - log.debug( - LOG_TEMPLATE, - { - 'project': version.project.slug, - 'version': version.slug, - 'msg': 'sending email to: %s' % email, - } - ) - - # We send only what we need from the Django model objects here to avoid - # serialization problems in the ``readthedocs.core.tasks.send_email_task`` - context = { - 'version': { - 'verbose_name': version.verbose_name, - }, - 'project': { - 'name': version.project.name, - }, - 'build': { - 'pk': build.pk, - 'error': build.error, - }, - 'build_url': 'https://{}{}'.format( - settings.PRODUCTION_DOMAIN, - build.get_absolute_url(), - ), - 'unsub_url': 'https://{}{}'.format( - settings.PRODUCTION_DOMAIN, - reverse('projects_notifications', args=[version.project.slug]), - ), - } - - if build.commit: - title = _( - 'Failed: {project[name]} ({commit})', - ).format(commit=build.commit[:8], **context) - else: - title = _('Failed: {project[name]} ({version[verbose_name]})').format( - **context - ) - - send_email( - email, - title, - template='projects/email/build_failed.txt', - template_html='projects/email/build_failed.html', - context=context, - ) - - -def webhook_notification(version, build, hook_url): - """ - Send webhook notification for project webhook. - - :param version: Version instance to send hook for - :param build: Build instance that failed - :param hook_url: Hook URL to send to - """ - project = version.project - - data = json.dumps({ - 'name': project.name, - 'slug': project.slug, - 'build': { - 'id': build.id, - 'commit': build.commit, - 'state': build.state, - 'success': build.success, - 'date': build.date.strftime('%Y-%m-%d %H:%M:%S'), - }, - }) - log.debug( - LOG_TEMPLATE, - { - 'project': project.slug, - 'version': '', - 'msg': 'sending notification to: %s' % hook_url, - } - ) - try: - requests.post( - hook_url, - data=data, - headers={'content-type': 'application/json'} - ) - except Exception: - log.exception('Failed to POST on webhook url: url=%s', hook_url) - - # Random Tasks @app.task() def remove_dirs(paths): diff --git a/readthedocs/projects/urls/private.py b/readthedocs/projects/urls/private.py index e584e7d30e8..1d3a6551614 100644 --- a/readthedocs/projects/urls/private.py +++ b/readthedocs/projects/urls/private.py @@ -39,13 +39,18 @@ ProjectUpdate, ProjectUsersCreateList, ProjectUsersDelete, - ProjectVersionDeleteHTML, ProjectVersionCreate, + ProjectVersionDeleteHTML, ProjectVersionDetail, RegexAutomationRuleCreate, RegexAutomationRuleUpdate, SearchAnalytics, TrafficAnalyticsView, + WebHookCreate, + WebHookDelete, + WebHookExchangeDetail, + WebHookList, + WebHookUpdate, ) urlpatterns = [ @@ -337,3 +342,33 @@ ] urlpatterns += automation_rule_urls + +webhook_urls = [ + url( + r'^(?P[-\w]+)/webhooks/$', + WebHookList.as_view(), + name='projects_webhooks', + ), + url( + r'^(?P[-\w]+)/webhooks/create/$', + WebHookCreate.as_view(), + name='projects_webhooks_create', + ), + url( + r'^(?P[-\w]+)/webhooks/(?P[-\w]+)/edit/$', + WebHookUpdate.as_view(), + name='projects_webhooks_edit', + ), + url( + r'^(?P[-\w]+)/webhooks/(?P[-\w]+)/delete/$', + WebHookDelete.as_view(), + name='projects_webhooks_delete', + ), + url( + r'^(?P[-\w]+)/webhooks/(?P[-\w]+)/exchanges/(?P[-\w]+)/$', # noqa + WebHookExchangeDetail.as_view(), + name='projects_webhooks_exchange', + ), +] + +urlpatterns += webhook_urls diff --git a/readthedocs/projects/views/private.py b/readthedocs/projects/views/private.py index aa205528509..f87c80196b4 100644 --- a/readthedocs/projects/views/private.py +++ b/readthedocs/projects/views/private.py @@ -5,7 +5,7 @@ from allauth.socialaccount.models import SocialAccount from django.conf import settings from django.contrib import messages -from django.db.models import Count +from django.db.models import Count, Q from django.http import ( Http404, HttpResponse, @@ -18,7 +18,7 @@ from django.utils import timezone from django.utils.safestring import mark_safe from django.utils.translation import ugettext_lazy as _ -from django.views.generic import ListView, TemplateView, View +from django.views.generic import ListView, TemplateView from formtools.wizard.views import SessionWizardView from vanilla import ( CreateView, @@ -40,7 +40,6 @@ ) from readthedocs.core.history import UpdateChangeReasonPostView from readthedocs.core.mixins import ListViewWithForm, PrivateViewMixin -from readthedocs.core.utils import trigger_build from readthedocs.core.utils.extend import SettingsOverrideObject from readthedocs.integrations.models import HttpExchange, Integration from readthedocs.oauth.services import registry @@ -517,7 +516,6 @@ class ProjectNotifications(ProjectNotificationsMixin, TemplateView): template_name = 'projects/project_notifications.html' email_form = EmailHookForm - webhook_form = WebHookForm def get_email_form(self): project = self.get_project() @@ -526,37 +524,36 @@ def get_email_form(self): project=project, ) - def get_webhook_form(self): - project = self.get_project() - return self.webhook_form( - self.request.POST or None, - project=project, - ) - def post(self, request, *args, **kwargs): if 'email' in request.POST: email_form = self.get_email_form() if email_form.is_valid(): email_form.save() - elif 'url' in request.POST: - webhook_form = self.get_webhook_form() - if webhook_form.is_valid(): - webhook_form.save() return HttpResponseRedirect(self.get_success_url()) + def _has_old_webhooks(self): + """ + Check if the project has webhooks from the old implementation created. + + Webhooks from the old implementation don't have a custom payload. + """ + project = self.get_project() + return ( + project.webhook_notifications + .filter(Q(payload__isnull=True) | Q(payload='')) + .exists() + ) + def get_context_data(self, **kwargs): context = super().get_context_data() project = self.get_project() emails = project.emailhook_notifications.all() - urls = project.webhook_notifications.all() - context.update( { 'email_form': self.get_email_form(), - 'webhook_form': self.get_webhook_form(), 'emails': emails, - 'urls': urls, + 'has_old_webhooks': self._has_old_webhooks(), }, ) return context @@ -582,6 +579,65 @@ def post(self, request, *args, **kwargs): return HttpResponseRedirect(self.get_success_url()) +class WebHookMixin(ProjectAdminMixin, PrivateViewMixin): + + model = WebHook + lookup_url_kwarg = 'webhook_pk' + form_class = WebHookForm + + def get_success_url(self): + return reverse( + 'projects_webhooks', + args=[self.get_project().slug], + ) + + +class WebHookList(WebHookMixin, ListView): + + pass + + +class WebHookCreate(WebHookMixin, CreateView): + + def get_success_url(self): + return reverse( + 'projects_webhooks_edit', + args=[self.get_project().slug, self.object.pk], + ) + + +class WebHookUpdate(WebHookMixin, UpdateView): + + def get_success_url(self): + return reverse( + 'projects_webhooks_edit', + args=[self.get_project().slug, self.object.pk], + ) + + +class WebHookDelete(WebHookMixin, DeleteView): + + http_method_names = ['post'] + + +class WebHookExchangeDetail(WebHookMixin, DetailView): + + model = HttpExchange + lookup_url_kwarg = 'webhook_exchange_pk' + webhook_url_kwarg = 'webhook_pk' + template_name = 'projects/webhook_exchange_detail.html' + + def get_queryset(self): + return self.model.objects.filter(webhook=self.get_webhook()) + + def get_webhook(self): + return get_object_or_404( + WebHook, + pk=self.kwargs[self.webhook_url_kwarg], + project=self.get_project(), + ) + + class ProjectTranslationsMixin(ProjectAdminMixin, PrivateViewMixin): def get_success_url(self): diff --git a/readthedocs/rtd_tests/base.py b/readthedocs/rtd_tests/base.py index 0a6d5c34b43..b8b1f8e95d9 100644 --- a/readthedocs/rtd_tests/base.py +++ b/readthedocs/rtd_tests/base.py @@ -65,7 +65,7 @@ class WizardTestCase(RequestFactoryTestMixin, TestCase): wizard_class_slug = None wizard_class = None - @patch('readthedocs.projects.views.private.trigger_build', lambda x: None) + @patch('readthedocs.core.utils.trigger_build', lambda x: None) def post_step(self, step, **kwargs): """ Post step form data to `url`, using supplementary `kwargs` diff --git a/readthedocs/rtd_tests/tests/test_build_notifications.py b/readthedocs/rtd_tests/tests/test_build_notifications.py index a92131f485a..3ef8c3a0e58 100644 --- a/readthedocs/rtd_tests/tests/test_build_notifications.py +++ b/readthedocs/rtd_tests/tests/test_build_notifications.py @@ -1,62 +1,96 @@ """Notifications sent after build is completed.""" - +import hashlib +import hmac +from django.utils import timezone import json +from unittest import mock -import django_dynamic_fixture as fixture +import requests_mock from django.core import mail -from django.test import TestCase -from unittest.mock import patch +from django.test import TestCase, override_settings +from django_dynamic_fixture import get from readthedocs.builds.constants import EXTERNAL from readthedocs.builds.models import Build, Version +from readthedocs.builds.tasks import send_build_notifications from readthedocs.projects.forms import WebHookForm -from readthedocs.projects.models import EmailHook, Project, WebHook -from readthedocs.projects.tasks import send_notifications +from readthedocs.projects.models import ( + EmailHook, + Project, + WebHook, + WebHookEvent, +) +@override_settings( + PRODUCTION_DOMAIN='readthedocs.org', + PUBLIC_DOMAIN='readthedocs.io', + USE_SUBDOMAIN=True, +) class BuildNotificationsTests(TestCase): + def setUp(self): - self.project = fixture.get(Project) - self.version = fixture.get(Version, project=self.project) - self.build = fixture.get(Build, version=self.version) + self.project = get(Project, slug='test', language='en') + self.version = get(Version, project=self.project, slug='1.0') + self.build = get(Build, version=self.version, commit='abc1234567890') - @patch('readthedocs.builds.managers.log') + @mock.patch('readthedocs.builds.managers.log') def test_send_notification_none_if_wrong_version_pk(self, mock_logger): self.assertFalse(Version.objects.filter(pk=345343).exists()) - send_notifications(version_pk=345343, build_pk=None) + send_build_notifications( + version_pk=345343, + build_pk=None, + event=WebHookEvent.BUILD_FAILED, + ) mock_logger.warning.assert_called_with( 'Version not found for given kwargs. %s', {'pk': 345343}, ) - def test_send_notification_none(self): - send_notifications(self.version.pk, self.build.pk) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) self.assertEqual(len(mail.outbox), 0) - def test_send_webhook_notification(self): - fixture.get(WebHook, project=self.project) - with patch('readthedocs.projects.tasks.requests.post') as mock: - mock.return_value = None - send_notifications(self.version.pk, self.build.pk) - mock.assert_called_once() - + @requests_mock.Mocker(kw='mock_request') + def test_send_webhook_notification(self, mock_request): + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], + ) + self.assertEqual(webhook.exchanges.all().count(), 0) + mock_request.post(webhook.url) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + self.assertEqual(webhook.exchanges.all().count(), 1) self.assertEqual(len(mail.outbox), 0) def test_dont_send_webhook_notifications_for_external_versions(self): - fixture.get(WebHook, project=self.project) + webhook = get(WebHook, url='https://example.com/webhook/', project=self.project) self.version.type = EXTERNAL self.version.save() + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + self.assertEqual(webhook.exchanges.all().count(), 0) - with patch('readthedocs.projects.tasks.requests.post') as mock: - mock.return_value = None - send_notifications(self.version.pk, self.build.pk) - mock.assert_not_called() - - self.assertEqual(len(mail.outbox), 0) - - def test_send_webhook_notification_has_content_type_header(self): - hook = fixture.get(WebHook, project=self.project) + def test_webhook_notification_has_content_type_header(self): + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], + ) data = json.dumps({ 'name': self.project.name, 'slug': self.project.slug, @@ -68,49 +102,235 @@ def test_send_webhook_notification_has_content_type_header(self): 'date': self.build.date.strftime('%Y-%m-%d %H:%M:%S'), }, }) - with patch('readthedocs.projects.tasks.requests.post') as mock: - mock.return_value = None - send_notifications(self.version.pk, self.build.pk) - mock.assert_called_once_with( - hook.url, + with mock.patch('readthedocs.builds.tasks.requests.post') as post: + post.return_value = None + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + post.assert_called_once_with( + webhook.url, data=data, - headers={'content-type': 'application/json'} + headers={ + 'content-type': 'application/json', + 'X-Hub-Signature': mock.ANY, + 'User-Agent': mock.ANY, + 'X-RTD-Event': mock.ANY, + }, + timeout=mock.ANY, + ) + + @requests_mock.Mocker(kw='mock_request') + def test_send_webhook_custom_on_given_event(self, mock_request): + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[ + WebHookEvent.objects.get(name=WebHookEvent.BUILD_TRIGGERED), + WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED), + ], + payload='{}', + ) + mock_request.post(webhook.url) + for event, _ in WebHookEvent.EVENTS: + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=event, ) + self.assertEqual(len(mail.outbox), 0) + self.assertEqual(webhook.exchanges.all().count(), 2) + + @requests_mock.Mocker(kw='mock_request') + def test_send_webhook_custom_payload(self, mock_request): + self.build.date = timezone.datetime( + year=2021, month=3, day=15, hour=15, minute=30, second=4, + ) + self.build.save() + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED)], + payload=json.dumps({ + 'message': 'Event {{ event }} triggered for {{ version.slug }}', + 'extra-data': { + 'build_id': '{{build.id}}', + 'build_commit': '{{build.commit}}', + 'build_url': '{{ build.url }}', + 'build_docsurl': '{{ build.docs_url }}', + 'build_start_date': '{{ build.start_date }}', + 'organization_slug': '{{ organization.slug }}', + 'organization_name': '{{ organization.name }}', + 'project_slug': '{{ project.slug }}', + 'project_name': '{{ project.name }}', + 'project_url': '{{ project.url }}', + 'version_slug': '{{ version.slug }}', + 'version_name': '{{ version.name }}', + 'invalid_substitution': '{{ invalid.substitution }}', + } + }), + ) + post = mock_request.post(webhook.url) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + self.assertTrue(post.called_once) + request = post.request_history[0] + self.assertEqual( + request.json(), + { + 'message': f'Event build:failed triggered for {self.version.slug}', + 'extra-data': { + 'build_id': str(self.build.pk), + 'build_commit': self.build.commit, + 'build_url': f'https://readthedocs.org{self.build.get_absolute_url()}', + 'build_docsurl': 'http://test.readthedocs.io/en/1.0/', + 'build_start_date': '2021-03-15T15:30:04', + 'organization_name': '', + 'organization_slug': '', + 'project_name': self.project.name, + 'project_slug': self.project.slug, + 'project_url': f'https://readthedocs.org{self.project.get_absolute_url()}', + 'version_name': self.version.verbose_name, + 'version_slug': self.version.slug, + 'invalid_substitution': '{{ invalid.substitution }}', + }, + } + ) + self.assertEqual(webhook.exchanges.all().count(), 1) - def test_send_email_notification(self): - fixture.get(EmailHook, project=self.project) - send_notifications(self.version.pk, self.build.pk, email=True) + @requests_mock.Mocker(kw='mock_request') + def test_webhook_headers(self, mock_request): + secret = '1234' + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED)], + payload='{"sign": "me"}', + secret=secret, + ) + post = mock_request.post(webhook.url) + signature = hmac.new( + key=secret.encode(), + msg=webhook.payload.encode(), + digestmod=hashlib.sha256, + ).hexdigest() + + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + self.assertTrue(post.called_once) + request = post.request_history[0] + headers = request.headers + self.assertTrue(headers['User-Agent'].startswith('Read-the-Docs/')) + self.assertEqual(headers['X-Hub-Signature'], signature) + self.assertEqual(headers['X-RTD-Event'], WebHookEvent.BUILD_FAILED) + self.assertEqual(webhook.exchanges.all().count(), 1) + + @requests_mock.Mocker(kw='mock_request') + def test_webhook_record_exchange(self, mock_request): + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED)], + payload='{"request": "ok"}', + ) + post = mock_request.post( + webhook.url, + json={'response': 'ok'}, + headers={'X-Greeting': 'Hi!'}, + status_code=201, + ) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + self.assertTrue(post.called_once) + self.assertEqual(webhook.exchanges.all().count(), 1) + exchange = webhook.exchanges.all().first() + self.assertTrue(exchange.request_headers['User-Agent'].startswith('Read-the-Docs/')) + self.assertIn('X-Hub-Signature', exchange.request_headers) + self.assertEqual(exchange.request_body, webhook.payload) + self.assertEqual(exchange.response_headers, {'X-Greeting': 'Hi!'}) + self.assertEqual(exchange.response_body, '{"response": "ok"}') + self.assertEqual(exchange.status_code, 201) + + def test_send_email_notification_on_build_failure(self): + get(EmailHook, project=self.project) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) self.assertEqual(len(mail.outbox), 1) def test_dont_send_email_notifications_for_external_versions(self): - fixture.get(EmailHook, project=self.project) + get(EmailHook, project=self.project) self.version.type = EXTERNAL self.version.save() - send_notifications(self.version.pk, self.build.pk, email=True) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) + self.assertEqual(len(mail.outbox), 0) + + def test_dont_send_email_notifications_for_other_events(self): + """Email notifications are only send for BUILD_FAILED events.""" + get(EmailHook, project=self.project) + for event in [WebHookEvent.BUILD_PASSED, WebHookEvent.BUILD_TRIGGERED]: + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=event, + ) self.assertEqual(len(mail.outbox), 0) - def test_send_email_and_webhook__notification(self): - fixture.get(EmailHook, project=self.project) - fixture.get(WebHook, project=self.project) - with patch('readthedocs.projects.tasks.requests.post') as mock: - mock.return_value = None - send_notifications(self.version.pk, self.build.pk, email=True) - mock.assert_called_once() + @requests_mock.Mocker(kw='mock_request') + def test_send_email_and_webhook_notification(self, mock_request): + get(EmailHook, project=self.project) + webhook = get( + WebHook, + url='https://example.com/webhook/', + project=self.project, + events=[WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], + ) + mock_request.post(webhook.url) + self.assertEqual(len(mail.outbox), 0) + self.assertEqual(webhook.exchanges.all().count(), 0) + send_build_notifications( + version_pk=self.version.pk, + build_pk=self.build.pk, + event=WebHookEvent.BUILD_FAILED, + ) self.assertEqual(len(mail.outbox), 1) + self.assertEqual(webhook.exchanges.all().count(), 1) class TestForms(TestCase): def setUp(self): - self.project = fixture.get(Project) - self.version = fixture.get(Version, project=self.project) - self.build = fixture.get(Build, version=self.version) + self.project = get(Project) + self.version = get(Version, project=self.project) + self.build = get(Build, version=self.version) def test_webhook_form_url_length(self): form = WebHookForm( { 'url': 'https://foobar.com', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], }, project=self.project, ) @@ -119,6 +339,8 @@ def test_webhook_form_url_length(self): form = WebHookForm( { 'url': 'foo' * 500, + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], }, project=self.project, ) diff --git a/readthedocs/rtd_tests/tests/test_privacy_urls.py b/readthedocs/rtd_tests/tests/test_privacy_urls.py index 24b0b9e7e0b..193c5b72ce8 100644 --- a/readthedocs/rtd_tests/tests/test_privacy_urls.py +++ b/readthedocs/rtd_tests/tests/test_privacy_urls.py @@ -18,7 +18,12 @@ from readthedocs.core.utils.tasks import TaskNoPermission from readthedocs.integrations.models import HttpExchange, Integration from readthedocs.oauth.models import RemoteOrganization, RemoteRepository -from readthedocs.projects.models import Domain, EnvironmentVariable, Project +from readthedocs.projects.models import ( + Domain, + EnvironmentVariable, + Project, + WebHook, +) from readthedocs.rtd_tests.utils import create_user @@ -160,7 +165,7 @@ def setUp(self): self.pip.translations.add(self.subproject) self.integration = get(Integration, project=self.pip, provider_data='') # For whatever reason, fixtures hates JSONField - self.webhook_exchange = HttpExchange.objects.create( + self.integration_exchange = HttpExchange.objects.create( related_object=self.integration, request_headers='{"foo": "bar"}', response_headers='{"foo": "bar"}', @@ -175,6 +180,13 @@ def setUp(self): action=VersionAutomationRule.ACTIVATE_VERSION_ACTION, version_type=BRANCH, ) + self.webhook = get(WebHook, project=self.pip) + self.webhook_exchange = HttpExchange.objects.create( + related_object=self.webhook, + request_headers='{"foo": "bar"}', + response_headers='{"foo": "bar"}', + status_code=200, + ) self.default_kwargs = { 'project_slug': self.pip.slug, 'subproject_slug': self.subproject.slug, @@ -186,11 +198,13 @@ def setUp(self): 'build_pk': self.build.pk, 'domain_pk': self.domain.pk, 'integration_pk': self.integration.pk, - 'exchange_pk': self.webhook_exchange.pk, + 'exchange_pk': self.integration_exchange.pk, 'environmentvariable_pk': self.environment_variable.pk, 'automation_rule_pk': self.automation_rule.pk, 'steps': 1, 'invalid_project_slug': 'invalid_slug', + 'webhook_pk': self.webhook.pk, + 'webhook_exchange_pk': self.webhook_exchange.pk, } @@ -253,7 +267,7 @@ def is_admin(self): # ## Private Project Testing ### -@mock.patch('readthedocs.projects.views.private.trigger_build', mock.MagicMock()) +@mock.patch('readthedocs.core.utils.trigger_build', mock.MagicMock()) class PrivateProjectAdminAccessTest(PrivateProjectMixin, TestCase): response_data = { @@ -275,6 +289,7 @@ class PrivateProjectAdminAccessTest(PrivateProjectMixin, TestCase): '/dashboard/pip/version/latest/delete_html/': {'status_code': 405}, '/dashboard/pip/rules/{automation_rule_id}/delete/': {'status_code': 405}, '/dashboard/pip/rules/{automation_rule_id}/move/{steps}/': {'status_code': 405}, + '/dashboard/pip/webhooks/{webhook_id}/delete/': {'status_code': 405}, } def get_url_path_ctx(self): @@ -282,6 +297,7 @@ def get_url_path_ctx(self): 'integration_id': self.integration.id, 'environmentvariable_id': self.environment_variable.id, 'automation_rule_id': self.automation_rule.id, + 'webhook_id': self.webhook.id, 'steps': 1, } @@ -292,7 +308,7 @@ def is_admin(self): return True -@mock.patch('readthedocs.projects.views.private.trigger_build', mock.MagicMock()) +@mock.patch('readthedocs.core.utils.trigger_build', mock.MagicMock()) class PrivateProjectUserAccessTest(PrivateProjectMixin, TestCase): response_data = { @@ -318,6 +334,7 @@ class PrivateProjectUserAccessTest(PrivateProjectMixin, TestCase): '/dashboard/pip/version/latest/delete_html/': {'status_code': 405}, '/dashboard/pip/rules/{automation_rule_id}/delete/': {'status_code': 405}, '/dashboard/pip/rules/{automation_rule_id}/move/{steps}/': {'status_code': 405}, + '/dashboard/pip/webhooks/{webhook_id}/delete/': {'status_code': 405}, } # Filtered out by queryset on projects that we don't own. @@ -328,6 +345,7 @@ def get_url_path_ctx(self): 'integration_id': self.integration.id, 'environmentvariable_id': self.environment_variable.id, 'automation_rule_id': self.automation_rule.id, + 'webhook_id': self.webhook.id, 'steps': 1, } diff --git a/readthedocs/rtd_tests/tests/test_project_forms.py b/readthedocs/rtd_tests/tests/test_project_forms.py index 3921e019bef..11e52d8745f 100644 --- a/readthedocs/rtd_tests/tests/test_project_forms.py +++ b/readthedocs/rtd_tests/tests/test_project_forms.py @@ -27,7 +27,11 @@ UpdateProjectForm, WebHookForm, ) -from readthedocs.projects.models import EnvironmentVariable, Project +from readthedocs.projects.models import ( + EnvironmentVariable, + Project, + WebHookEvent, +) class TestProjectForms(TestCase): @@ -665,7 +669,7 @@ def test_can_change_language_to_self_lang(self): self.assertTrue(form.is_valid()) -class TestNotificationForm(TestCase): +class TestWebhookForm(TestCase): def setUp(self): self.project = get(Project) @@ -674,7 +678,9 @@ def test_webhookform(self): self.assertEqual(self.project.webhook_notifications.all().count(), 0) data = { - 'url': 'http://www.example.com/' + 'url': 'http://www.example.com/', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], } form = WebHookForm(data=data, project=self.project) self.assertTrue(form.is_valid()) @@ -682,7 +688,9 @@ def test_webhookform(self): self.assertEqual(self.project.webhook_notifications.all().count(), 1) data = { - 'url': 'https://www.example.com/' + 'url': 'https://www.example.com/', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_PASSED).id], } form = WebHookForm(data=data, project=self.project) self.assertTrue(form.is_valid()) @@ -693,7 +701,9 @@ def test_wrong_inputs_in_webhookform(self): self.assertEqual(self.project.webhook_notifications.all().count(), 0) data = { - 'url': '' + 'url': '', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], } form = WebHookForm(data=data, project=self.project) self.assertFalse(form.is_valid()) @@ -701,13 +711,41 @@ def test_wrong_inputs_in_webhookform(self): self.assertEqual(self.project.webhook_notifications.all().count(), 0) data = { - 'url': 'wrong-url' + 'url': 'wrong-url', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], } form = WebHookForm(data=data, project=self.project) self.assertFalse(form.is_valid()) self.assertDictEqual(form.errors, {'url': ['Enter a valid URL.']}) self.assertEqual(self.project.webhook_notifications.all().count(), 0) + data = { + 'url': 'https://example.com/webhook/', + 'payload': '{wrong json object}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], + } + form = WebHookForm(data=data, project=self.project) + self.assertFalse(form.is_valid()) + self.assertDictEqual(form.errors, {'payload': ['The payload must be a valid JSON object.']}) + self.assertEqual(self.project.webhook_notifications.all().count(), 0) + + data = { + 'url': 'https://example.com/webhook/', + 'payload': '{}', + 'events': [], + } + form = WebHookForm(data=data, project=self.project) + self.assertFalse(form.is_valid()) + self.assertDictEqual(form.errors, {'events': ['This field is required.']}) + self.assertEqual(self.project.webhook_notifications.all().count(), 0) + + +class TestNotificationForm(TestCase): + + def setUp(self): + self.project = get(Project) + def test_emailhookform(self): self.assertEqual(self.project.emailhook_notifications.all().count(), 0) diff --git a/readthedocs/rtd_tests/tests/test_project_views.py b/readthedocs/rtd_tests/tests/test_project_views.py index bbad4d2e07f..ea2849ddc88 100644 --- a/readthedocs/rtd_tests/tests/test_project_views.py +++ b/readthedocs/rtd_tests/tests/test_project_views.py @@ -1,12 +1,10 @@ from datetime import timedelta from unittest import mock -from allauth.account.models import EmailAddress from allauth.socialaccount.models import SocialAccount from django.contrib.auth.models import User -from django.contrib.messages import constants as message_const from django.http.response import HttpResponseRedirect -from django.test import TestCase +from django.test import TestCase, override_settings from django.urls import reverse from django.utils import timezone from django.views.generic.base import ContextMixin @@ -16,9 +14,10 @@ from readthedocs.builds.models import Build, Version from readthedocs.integrations.models import GenericAPIWebhook, GitHubWebhook from readthedocs.oauth.models import RemoteRepository, RemoteRepositoryRelation +from readthedocs.organizations.models import Organization from readthedocs.projects.constants import PUBLIC from readthedocs.projects.exceptions import ProjectSpamError -from readthedocs.projects.models import Domain, Project +from readthedocs.projects.models import Domain, Project, WebHook, WebHookEvent from readthedocs.projects.views.mixins import ProjectRelationMixin from readthedocs.projects.views.private import ImportWizardView from readthedocs.projects.views.public import ProjectBadgeView @@ -667,3 +666,65 @@ def test_project_filtering_work_with_tags_with_space_in_name(self): pip.tags.add('tag with space') response = self.client.get('/projects/tags/tag-with-space/') self.assertContains(response, '"/projects/pip/"') + + +@override_settings(RTD_ALLOW_ORGANIZATIONS=False) +class TestWebhooksViews(TestCase): + + def setUp(self): + self.user = get(User) + self.project = get(Project, slug='test', users=[self.user]) + self.version = get(Version, slug='1.0', project=self.project) + self.webhook = get(WebHook, project=self.project) + self.client.force_login(self.user) + + def test_list(self): + resp = self.client.get( + reverse('projects_webhooks', args=[self.project.slug]), + ) + self.assertEqual(resp.status_code, 200) + queryset = resp.context['object_list'] + self.assertEqual(queryset.count(), 1) + self.assertEqual(queryset.first(), self.webhook) + + def test_create(self): + self.assertEqual(self.project.webhook_notifications.all().count(), 1) + resp = self.client.post( + reverse('projects_webhooks_create', args=[self.project.slug]), + data = { + 'url': 'http://www.example.com/', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], + }, + ) + self.assertEqual(resp.status_code, 302) + self.assertEqual(self.project.webhook_notifications.all().count(), 2) + + def test_update(self): + self.assertEqual(self.project.webhook_notifications.all().count(), 1) + self.client.post( + reverse('projects_webhooks_edit', args=[self.project.slug, self.webhook.pk]), + data = { + 'url': 'http://www.example.com/new', + 'payload': '{}', + 'events': [WebHookEvent.objects.get(name=WebHookEvent.BUILD_FAILED).id], + }, + ) + self.webhook.refresh_from_db() + self.assertEqual(self.webhook.url, 'http://www.example.com/new') + self.assertEqual(self.project.webhook_notifications.all().count(), 1) + + def test_delete(self): + self.assertEqual(self.project.webhook_notifications.all().count(), 1) + self.client.post( + reverse('projects_webhooks_delete', args=[self.project.slug, self.webhook.pk]), + ) + self.assertEqual(self.project.webhook_notifications.all().count(), 0) + + +@override_settings(RTD_ALLOW_ORGANIZATIONS=True) +class TestWebhooksViewsWithOrganizations(TestWebhooksViews): + + def setUp(self): + super().setUp() + self.organization = get(Organization, owners=[self.user], projects=[self.project]) diff --git a/readthedocs/templates/projects/email/build_failed.html b/readthedocs/templates/projects/email/build_failed.html index dc466abfe1c..9f49a8048d8 100644 --- a/readthedocs/templates/projects/email/build_failed.html +++ b/readthedocs/templates/projects/email/build_failed.html @@ -33,6 +33,6 @@

- You can unsubscribe from these emails in your Notification Settings + You can unsubscribe from these emails in your Notification Settings

{% endblock %} diff --git a/readthedocs/templates/projects/project_edit_base.html b/readthedocs/templates/projects/project_edit_base.html index 2199bd0005d..d1b3f944400 100644 --- a/readthedocs/templates/projects/project_edit_base.html +++ b/readthedocs/templates/projects/project_edit_base.html @@ -26,7 +26,8 @@
  • {% trans "Integrations" %}
  • {% trans "Environment Variables" %}
  • {% trans "Automation Rules" %}
    • -
  • {% trans "Notifications" %}
    • +
  • {% trans "Webhooks" %}
    • +
  • {% trans "Email Notifications" %}
  • {% trans "Traffic Analytics" %}
  • {% trans "Search Analytics" %}
    • {% if USE_PROMOS %} diff --git a/readthedocs/templates/projects/project_notifications.html b/readthedocs/templates/projects/project_notifications.html index f3f7bb6e8b5..b52f22f1d6b 100644 --- a/readthedocs/templates/projects/project_notifications.html +++ b/readthedocs/templates/projects/project_notifications.html @@ -2,20 +2,30 @@ {% load i18n %} -{% block title %}{% trans "Edit Notifications" %}{% endblock %} +{% block title %}{% trans "Email Notifications" %}{% endblock %} {% block nav-dashboard %} class="active"{% endblock %} {% block editing-option-edit-proj %}class="active"{% endblock %} {% block project-notifications-active %}active{% endblock %} -{% block project_edit_content_header %}{% trans "Notifications" %}{% endblock %} +{% block project_edit_content_header %}{% trans "Email Notifications" %}{% endblock %} {% block project_edit_content %}

    - {% trans "Configure Notifications to be sent on build failures." %} + {% trans "Configure email notifications to be sent on build failures." %}

    + {% if has_old_webhooks %} +

    + {% url 'projects_webhooks' project.slug as webhooks_url %} + {% blocktrans trimmed with webhooks_url=webhooks_url %} + Webhooks have been moved to its own page. + Go to the Webhooks tab if you are looking for your webhooks. + {% endblocktrans %} +

    + {% endif %} + {% if emails or urls %}

    {% trans "Existing Notifications" %}

    @@ -69,15 +79,4 @@

    {% trans "New Email Notifications" %}

    - -

    {% trans "New Webhook Notifications" %}

    -

    - {% trans "Add a URL to notify" %} -

    -
    {% csrf_token %} - {{ webhook_form }} -

    - -

    -
    {% endblock %} diff --git a/readthedocs/templates/projects/webhook_exchange_detail.html b/readthedocs/templates/projects/webhook_exchange_detail.html new file mode 100644 index 00000000000..96363648ef5 --- /dev/null +++ b/readthedocs/templates/projects/webhook_exchange_detail.html @@ -0,0 +1,39 @@ +{% extends "projects/project_edit_base.html" %} + +{% load i18n %} + +{% block title %}{% trans "Webhooks" %}{% endblock %} + +{% block nav-dashboard %} class="active"{% endblock %} + +{% block project-webhooks-active %}active{% endblock %} +{% block project_edit_content_header %}{% trans "Exchange" %}{% endblock %} + +{% block project_edit_content %} +

    Response

    + +
    +
    +
    Status:
    +
    {{ httpexchange.status_code }}
    + + {% for header, value in httpexchange.response_headers.items %} +
    {{ header }}:
    +
    {{ value }}
    + {% endfor %} +
    + {{ httpexchange.formatted_response_body }} +
    + +

    Request

    + +
    +
    + {% for header, value in httpexchange.request_headers.items %} +
    {{ header }}:
    +
    {{ value }}
    + {% endfor %} +
    + {{ httpexchange.formatted_request_body }} +
    +{% endblock %} diff --git a/readthedocs/templates/projects/webhook_form.html b/readthedocs/templates/projects/webhook_form.html new file mode 100644 index 00000000000..f51a8705d6d --- /dev/null +++ b/readthedocs/templates/projects/webhook_form.html @@ -0,0 +1,71 @@ +{% extends "projects/project_edit_base.html" %} + +{% load i18n %} + +{% block title %}{% trans "Webhooks" %}{% endblock %} + +{% block nav-dashboard %} class="active"{% endblock %} + +{% block project-webhooks-active %}active{% endblock %} + +{% block project_edit_content_header %}{% trans "Webhooks" %}{% endblock %} + +{% block project_edit_content %} + +

    + {% blocktrans trimmed with docs_url="https://readthedocs.io/page/build-notifications.html#webhooks" %} + We’ll send a POST request to the URL with the JSON payload below on the selected events. + Check our docs for more information. + {% endblocktrans %} +

    + + {# If the webhook was created from the old implementation, it doesn't have a custom payload. #} + {% if object.pk and not object.payload %} +

    + {% blocktrans trimmed with docs_url="https://docs.readthedocs.io/page/build-notifications.html#legacy-webhooks" %} + This webhook was created from our old implementation and doesn't have a custom payload. + Check our docs for the payload description or edit the payload to set a custom one. + {% endblocktrans %} +

    + {% endif %} + +
    + {% csrf_token %} + {{ form.as_p }} + +
    + + {% if object.pk %} +
    + {% csrf_token %} + +
    + +

    {% trans "Recent Activity" %}

    + +
    + +
    + {% endif %} + +{% endblock %} diff --git a/readthedocs/templates/projects/webhook_list.html b/readthedocs/templates/projects/webhook_list.html new file mode 100644 index 00000000000..b7e6c63f373 --- /dev/null +++ b/readthedocs/templates/projects/webhook_list.html @@ -0,0 +1,54 @@ +{% extends "projects/project_edit_base.html" %} + +{% load i18n %} + +{% block title %}{% trans "Webhooks" %}{% endblock %} + +{% block nav-dashboard %} class="active"{% endblock %} + +{% block project-webhooks-active %}active{% endblock %} +{% block project_edit_content_header %}{% trans "Webhooks" %}{% endblock %} + +{% block project_edit_content %} +
    + +
    + +
    +
    +
      + {% for webhook in object_list %} +
    • + + {{ webhook.url|truncatechars:35 }} + + + {% for event in webhook.events.all|slice:":2" %} + {{ event.name }}{% endfor %}{% if webhook.events.all|length > 2 %}...{% endif %} + + +
      • + {% empty %} +
    • +

      + {% trans "No webhooks found." %} +

      +
      • + {% endfor %} +
    +
    +
    +{% endblock %}