Skip to content

Allow to hook the initial build from outside #4033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 67 commits into from
Jun 14, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
3582584
Remove unused basic attribute on trigger build
humitos May 1, 2018
10def53
Split trigger_build to be able to prepare_build first
humitos May 1, 2018
c2c10af
Allow to hook the initial build from outside
humitos Apr 27, 2018
efa2bd3
Remove now unused signal
humitos May 1, 2018
851e1ff
Fix Celery signature creation
humitos May 1, 2018
0f8a055
Fix testcases with basic on trigger_build
humitos May 1, 2018
2fe1220
Fix mock calls in test cases
humitos May 1, 2018
2d7063b
Fix test mock check
humitos May 8, 2018
e4412b3
Use async task to attach webhook
humitos Jun 11, 2018
d5cf6c6
Use proper context object
humitos Jun 5, 2018
14c99f3
Make it compatible with newer Django versions
humitos Jun 5, 2018
90f5ea6
Py2 and Py3 compatible line
humitos Jun 5, 2018
3cff0a7
Dismiss the sticky message and stay in the same page
humitos Jun 5, 2018
da81906
Use Context to render the template
humitos Jun 5, 2018
9cd08f7
Lint errors fixed
humitos Jun 5, 2018
804ef65
Call the task in a sync way properly
humitos Jun 5, 2018
8179559
Update common submodule to latest
humitos Jun 5, 2018
e5fe8f7
Define NonPersistentStorage for one-time notifications
humitos Jun 11, 2018
fca9947
Make string translatable
humitos Jun 11, 2018
38a93e3
Fix merge conflicts
humitos Jun 11, 2018
757f57f
Recover accidentally deleted line
humitos Jun 11, 2018
d804b53
Fixed linting errors
humitos Jun 11, 2018
2f20aed
Replace message_key with reason to be clearer
humitos Jun 12, 2018
90892fb
Rename non persistent storage class
humitos Jun 12, 2018
f72a129
Remove old templates
humitos Jun 12, 2018
8f1f5cd
Make SiteNotification more flexible
humitos Jun 12, 2018
39237cf
Adapt AttachWebhookNotification to the new features
humitos Jun 12, 2018
58ce253
Refactor the task to attach a webhook
humitos Jun 12, 2018
f609a3a
Test cases for SiteNotification and NonPersistentStorage
humitos Jun 12, 2018
864d968
Remove unnecessary lines
humitos Jun 12, 2018
9001a9e
Show a persistent message for invalid project webhook
humitos Jun 12, 2018
1d90204
Improve copy
humitos Jun 13, 2018
0f16e24
Remove unused basic attribute on trigger build
humitos May 1, 2018
16cbf0f
Split trigger_build to be able to prepare_build first
humitos May 1, 2018
2d4b227
Allow to hook the initial build from outside
humitos Apr 27, 2018
4022f28
Remove now unused signal
humitos May 1, 2018
79fcaa3
Fix Celery signature creation
humitos May 1, 2018
26e45b3
Fix testcases with basic on trigger_build
humitos May 1, 2018
cbcf55d
Fix mock calls in test cases
humitos May 1, 2018
ebac662
Fix test mock check
humitos May 8, 2018
bb500ba
Use async task to attach webhook
humitos Jun 11, 2018
1939f9b
Use proper context object
humitos Jun 5, 2018
671c29f
Make it compatible with newer Django versions
humitos Jun 5, 2018
a9a16a6
Py2 and Py3 compatible line
humitos Jun 5, 2018
d8cc092
Dismiss the sticky message and stay in the same page
humitos Jun 5, 2018
f1b2078
Use Context to render the template
humitos Jun 5, 2018
c96fd72
Lint errors fixed
humitos Jun 5, 2018
a2244f7
Call the task in a sync way properly
humitos Jun 5, 2018
a67dde4
Update common submodule to latest
humitos Jun 5, 2018
8416503
Define NonPersistentStorage for one-time notifications
humitos Jun 11, 2018
db4a084
Make string translatable
humitos Jun 11, 2018
9f31115
Fix merge conflicts
humitos Jun 11, 2018
eee7e71
Recover accidentally deleted line
humitos Jun 11, 2018
f8299cc
Fixed linting errors
humitos Jun 11, 2018
c96c4fe
Replace message_key with reason to be clearer
humitos Jun 12, 2018
1ac78a1
Rename non persistent storage class
humitos Jun 12, 2018
7e1c7d9
Remove old templates
humitos Jun 12, 2018
5421b1f
Make SiteNotification more flexible
humitos Jun 12, 2018
c8deddf
Adapt AttachWebhookNotification to the new features
humitos Jun 12, 2018
c85676f
Refactor the task to attach a webhook
humitos Jun 12, 2018
9a54e74
Test cases for SiteNotification and NonPersistentStorage
humitos Jun 12, 2018
48eed16
Remove unnecessary lines
humitos Jun 12, 2018
13ec71c
Show a persistent message for invalid project webhook
humitos Jun 12, 2018
9f8fddf
Improve copy
humitos Jun 13, 2018
5bfd8df
Merge branch 'master' of github.com:rtfd/readthedocs.org into humitos…
humitos Jun 14, 2018
5d1d7c5
Remove fixed template notification about Project.has_valid_webhook
humitos Jun 14, 2018
c87e646
Merge branch 'humitos/async/initial-build' of github.com:rtfd/readthe…
humitos Jun 14, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion common
4 changes: 3 additions & 1 deletion media/css/core.css
Original file line number Diff line number Diff line change
Expand Up @@ -610,7 +610,9 @@ p.build-missing { font-size: .8em; color: #9d9a55; margin: 0 0 3px; }
/* notification box */
.notification { padding: 5px 0; color: #a55; }
.notification-20,
.notification-25 {
.notification-25,
.notification-101,
.notification-102 {
color: #5a5;
}

Expand Down
74 changes: 56 additions & 18 deletions readthedocs/core/utils/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# -*- coding: utf-8 -*-
"""Common utilty functions."""

from __future__ import absolute_import
Expand All @@ -16,7 +17,7 @@
from future.backports.urllib.parse import urlparse
from celery import group, chord

from readthedocs.builds.constants import LATEST
from readthedocs.builds.constants import LATEST, BUILD_STATE_TRIGGERED
from readthedocs.doc_builder.constants import DOCKER_LIMITS


Expand Down Expand Up @@ -75,38 +76,47 @@ def cname_to_slug(host):
return slug


def trigger_build(project, version=None, record=True, force=False, basic=False):
def prepare_build(
project, version=None, record=True, force=False, immutable=True):
"""
Trigger build for project and version.
Prepare a build in a Celery task for project and version.

If project has a ``build_queue``, execute task on this build queue. Queue
will be prefixed with ``build-`` to unify build queue names.
If project has a ``build_queue``, execute the task on this build queue. If
project has ``skip=True``, the build is not triggered.

:param project: project's documentation to be built
:param version: version of the project to be built. Default: ``latest``
:param record: whether or not record the build in a new Build object
:param force: build the HTML documentation even if the files haven't changed
:param immutable: whether or not create an immutable Celery signature
:returns: Celery signature of UpdateDocsTask to be executed
"""
# Avoid circular import
from readthedocs.projects.tasks import UpdateDocsTask
from readthedocs.builds.models import Build

if project.skip:
log.info(
'Build not triggered because Project.skip=True: project=%s',
project.slug,
)
return None

if not version:
version = project.versions.get(slug=LATEST)

kwargs = dict(
pk=project.pk,
version_pk=version.pk,
record=record,
force=force,
basic=basic,
)
kwargs = {
'version_pk': version.pk,
'record': record,
'force': force,
}

build = None
if record:
build = Build.objects.create(
project=project,
version=version,
type='html',
state='triggered',
state=BUILD_STATE_TRIGGERED,
success=True,
)
kwargs['build_pk'] = build.pk
Expand All @@ -121,16 +131,44 @@ def trigger_build(project, version=None, record=True, force=False, basic=False):
if project.container_time_limit:
time_limit = int(project.container_time_limit)
except ValueError:
pass
log.warning('Invalid time_limit for project: %s', project.slug)

# Add 20% overhead to task, to ensure the build can timeout and the task
# will cleanly finish.
options['soft_time_limit'] = time_limit
options['time_limit'] = int(time_limit * 1.2)

update_docs = UpdateDocsTask()
update_docs.apply_async(kwargs=kwargs, **options)
update_docs_task = UpdateDocsTask()

# Py 2.7 doesn't support ``**`` expand syntax twice. We create just one big
# kwargs (including the options) for this and expand it just once.
# return update_docs_task.si(project.pk, **kwargs, **options)
kwargs.update(options)

return update_docs_task.si(project.pk, **kwargs)

return build

def trigger_build(project, version=None, record=True, force=False):
"""
Trigger a Build.

Helper that calls ``prepare_build`` and just effectively trigger the Celery
task to be executed by a worker.

:param project: project's documentation to be built
:param version: version of the project to be built. Default: ``latest``
:param record: whether or not record the build in a new Build object
:param force: build the HTML documentation even if the files haven't changed
:returns: Celery AsyncResult promise
"""
update_docs_task = prepare_build(
project,
version,
record,
force,
immutable=True,
)
return update_docs_task.apply_async()


def send_email(recipient, subject, template, template_html, context=None,
Expand Down
3 changes: 2 additions & 1 deletion readthedocs/notifications/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@
.. _`django-messages-extends`: https://github.com
/AliLozano/django-messages-extends/
"""
from .notification import Notification
from .notification import Notification, SiteNotification
from .backends import send_notification

__all__ = (
'Notification',
'SiteNotification',
'send_notification'
)

Expand Down
16 changes: 14 additions & 2 deletions readthedocs/notifications/backends.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,11 @@ def send_notification(request, notification):
backends = getattr(settings, 'NOTIFICATION_BACKENDS', [])
for cls_name in backends:
backend = import_string(cls_name)(request)
backend.send(notification)
# Do not send email notification if defined explicitly
if backend.name == EmailBackend.name and not notification.send_email:
pass
else:
backend.send(notification)


class Backend(object):
Expand Down Expand Up @@ -96,8 +100,16 @@ def send(self, notification):
req = HttpRequest()
setattr(req, 'session', '')
storage = cls(req)

# Use the method defined by the notification or map a simple level to a
# persistent one otherwise
if hasattr(notification, 'get_message_level'):
level = notification.get_message_level()
else:
level = LEVEL_MAPPING.get(notification.level, INFO_PERSISTENT)

storage.add(
level=LEVEL_MAPPING.get(notification.level, INFO_PERSISTENT),
level=level,
message=notification.render(
backend_name=self.name,
source_format=HTML,
Expand Down
17 changes: 17 additions & 0 deletions readthedocs/notifications/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,20 @@
REQUIREMENT: message_constants.WARNING_PERSISTENT,
ERROR: message_constants.ERROR_PERSISTENT,
}


# Message levels to save the message into the database and mark as read
# immediately after retrieved (one-time shown message)
DEBUG_NON_PERSISTENT = 100
INFO_NON_PERSISTENT = 101
SUCCESS_NON_PERSISTENT = 102
WARNING_NON_PERSISTENT = 103
ERROR_NON_PERSISTENT = 104

NON_PERSISTENT_MESSAGE_LEVELS = (
DEBUG_NON_PERSISTENT,
INFO_NON_PERSISTENT,
SUCCESS_NON_PERSISTENT,
WARNING_NON_PERSISTENT,
ERROR_NON_PERSISTENT,
)
102 changes: 99 additions & 3 deletions readthedocs/notifications/notification.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
# -*- coding: utf-8 -*-
"""Support for templating of notifications."""

from __future__ import absolute_import
from builtins import object
import logging
from django.conf import settings
from django.template import Template, Context
from django.template.loader import render_to_string
from django.db import models
from django.http import HttpRequest

from .backends import send_notification
from . import constants


log = logging.getLogger(__name__)


class Notification(object):

"""
Expand All @@ -28,6 +34,7 @@ class Notification(object):
level = constants.INFO
subject = None
user = None
send_email = True

def __init__(self, context_object, request, user=None):
self.object = context_object
Expand All @@ -45,8 +52,8 @@ def get_context_data(self):
self.context_object_name: self.object,
'request': self.request,
'production_uri': '{scheme}://{host}'.format(
scheme='https', host=settings.PRODUCTION_DOMAIN
)
scheme='https', host=settings.PRODUCTION_DOMAIN,
),
}

def get_template_names(self, backend_name, source_format=constants.HTML):
Expand All @@ -71,7 +78,7 @@ def render(self, backend_name, source_format=constants.HTML):
backend_name=backend_name,
source_format=source_format,
),
context=Context(self.get_context_data()),
context=self.get_context_data(),
)

def send(self):
Expand All @@ -84,3 +91,92 @@ def send(self):
avoided.
"""
send_notification(self.request, self)


class SiteNotification(Notification):

"""
Simple notification to show *only* on site messages.

``success_message`` and ``failure_message`` can be a simple string or a
dictionary with different messages depending on the reason of the failure /
success. The message is selected by using ``reason`` to get the proper
value.

The notification is tied to the ``user`` and it could be sticky, persistent
or normal --this depends on the ``success_level`` and ``failure_level``.

.. note::

``send_email`` is forced to False to not send accidental emails when
only a simple site notification is needed.
"""

send_email = False

success_message = None
failure_message = None

success_level = constants.SUCCESS_NON_PERSISTENT
failure_level = constants.ERROR_NON_PERSISTENT

def __init__(
self, user, success, reason=None, context_object=None,
request=None, extra_context=None):
self.object = context_object

self.user = user or request.user
# Fake the request in case the notification is instantiated from a place
# without access to the request object (Celery task, for example)
self.request = request or HttpRequest()
self.request.user = user

self.success = success
self.reason = reason
self.extra_context = extra_context or {}
super(SiteNotification, self).__init__(context_object, request, user)

def get_context_data(self):
context = super(SiteNotification, self).get_context_data()
context.update(self.extra_context)
return context

def get_message_level(self):
if self.success:
return self.success_level
return self.failure_level
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I like mimicking django class patterns like this.


def get_message(self, success):
if success:
message = self.success_message
else:
message = self.failure_message

msg = '' # default message in case of error
if isinstance(message, dict):
if self.reason:
if self.reason in message:
msg = message.get(self.reason)
else:
# log the error but not crash
log.error(
"Notification {} has no key '{}' for {} messages".format(
self.__class__.__name__,
self.reason,
'success' if self.success else 'failure',
),
)
else:
log.error(
'{}.{}_message is a dictionary but no reason was provided'.format(
self.__class__.__name__,
'success' if self.success else 'failure',
),
)
else:
msg = message

return Template(msg).render(context=Context(self.get_context_data()))

def render(self, *args, **kwargs): # pylint: disable=arguments-differ
return self.get_message(self.success)
Loading