APIv3 "Import Project" endpoint #5857

humitos · 2019-06-27T11:59:04Z

This PR allows POST at /api/v3/projects to Import a Project.

It follows the same idea that importing from the Dashboard:

Create the Project object
Assign the user who makes the request as maintainer
Send project_import Django signal
Trigger an initial build

Note that I had to refactor the function that trigger the initial build to be able to call the same logic from two different places.

ericholscher

This looks good, but I do think it might be useful to keep all the logic together from the view and code. This does it for the trigger_initial_build logic, but it would be useful for the logic around adding the user to the Project, since this varies between sites.

ericholscher · 2019-07-01T18:45:36Z

readthedocs/api/v3/permissions.py

@@ -16,6 +16,7 @@ def has_permission(self, request, view):
        if is_authenticated:
            if view.basename == 'projects' and any([
                    view.action == 'list',
+                    view.action == 'create',  # used to create Form in BrowsableAPIRenderer


Do we want to support that? I feel like creating a project via the web form is probably not going to be the best UX.

I've been thinking on making the browsable API consistent with what the API offers. Besides consistency, it helps to explore it and have a better understanding of what's available and what can be done: not "self-documented" but in that direction.

On the other hand, removing the create from this check it fails with 404 when accessing to /api/v3/projects/ because of an internal DRF check. Actually, that's how I realized that I needed it :)

Overall I'm in favor of making the browsable API be consistent with what the API can do. We still can restrict creating with additional permissions or even turn off the browsable API in prod completely if needed (hopefully not as I do believe the browsable API makes the API more approachable). Generally though I hope regular users are not creating projects in production through the DRF UI.

ericholscher · 2019-07-01T18:47:51Z

readthedocs/api/v3/views.py

+        Return correct serializer depending on the action.
+
+        For GET it returns a serializer with many fields and on PUT/PATCH/POST,
+        it return a serializer to validate just a few fields.


Does this not allow us to update the entire project via API?

No, in theory only the fields from ProjectCreateSerializer.Meta.fields are allowed (name, language, repository and project_url). If there are extra fields passed, they are ignored.

We may want to add more, though.

Seems like a good test case, actually.

ericholscher · 2019-07-01T18:48:11Z

readthedocs/api/v3/views.py

+        For GET it returns a serializer with many fields and on PUT/PATCH/POST,
+        it return a serializer to validate just a few fields.
+        """
+        if self.action in ('list', 'retrieve', 'superproject'):


what is superproject here? They should probably be constants.

superproject is the @action defined in the ProjectViewSet that returns the superproject of a project.

I'm not sure if it makes sense to make it a constant. Although, maybe it's good to add a comment in this line to avoid confusions.

Yea, we should add a comment at least, since it's not like the others.

ericholscher · 2019-07-01T18:50:46Z

readthedocs/api/v3/views.py

+        # TODO: these lines need to be adapted for Corporate
+        project.users.add(request.user)
+        project_import.send(sender=project, request=request)
+        trigger_initial_build(project, request.user)


I wonder if this should all be abstracted into a single function called trigger_build(initial) and used in all the builds, so we only have to change this logic in one place.

ericholscher · 2019-07-01T18:54:23Z

readthedocs/api/v3/views.py

+        headers = self.get_success_headers(serializer.data)
+
+        # TODO: these lines need to be adapted for Corporate
+        project.users.add(request.user)


Where does this happen in the normal import workflow? I can't find it in the code.

Ah, was on the subclass: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/forms.py#L50-L53

So perhaps we should be inheriting form this form for the Form we're using?

I'm not sure I like to inherit a Serializer from a Wizard Form. Although, maybe these common methods could be extracted into a Mixin that it's usage by both. Actually, in that case, there is nothing to change in corporate since it's already done the override.

This involves a bigger refactor than I thought. All the process to import a project is very tied to forms methods. ProjectBasicForm.save links the remote repository.

ImportWizardView manipulates the Basic and the Extra form to make it all fit in one project before saving it plus triggering a new build (done method in the form)

humitos · 2019-07-02T11:24:18Z

This looks good, but I do think it might be useful to keep all the logic together from the view and code

Like a import_project(validated_data, request) helper that, create the project, assign the user, trigger the initial build, etc? I think this can be handy, I need to think a little on the implementation since we probably will need a class so it can be override from corporate (that has some differencies).

humitos · 2019-07-02T12:17:27Z

@ericholscher I refactored the Import Project code by extracting the shared bits into a Mixin. Looks better than the first option that I proposed, but still it's not perfect.

All the manipulation of the fields (from the Form or Serializer) happen on each view and once those things are done, the data is passed to the common method that will finish the import.

humitos · 2019-07-02T12:18:35Z

readthedocs/projects/views/mixins.py

+
+    """Helpers to import a Project."""
+
+    def import_project(self, project, tags, request):


Maybe finish_import_project is a better name here, what do you think @ericholscher? Considering that the Project object is created outside this function and this one does the final steps.

That seems better.

humitos · 2019-07-08T07:39:28Z

readthedocs/api/v3/serializers.py

+            'language',
+            'programming_language',
+            'repository',
+            'homepage',


We can add more fields here. Maybe these are also useful:

default_branch

default_version

tags

analytics_code

show_version_warning

ericholscher

Lots of good stuff here, just a few comments.

We also need to add public-facing docs here, that is the biggest missing piece for me.

ericholscher · 2019-07-08T23:00:16Z

readthedocs/api/v3/views.py

@@ -110,14 +115,13 @@ class ProjectsViewSet(APIv3Settings, NestedViewSetMixin, ProjectQuerySetMixin,
    * Subprojects of a project: ``/api/v3/projects/{project_slug}/subprojects/``
    * Superproject of a project: ``/api/v3/projects/{project_slug}/superproject/``

-    Go to [https://docs.readthedocs.io/en/stable/api/v3.html](https://docs.readthedocs.io/en/stable/api/v3.html)
+    Go to [https://docs.readthedocs.io/page/api/v3.html](https://docs.readthedocs.io/page/api/v3.html)


I think this should auto-link, no? Or you can put <> around it I believe (depending on markdown implementation, 😦)

Yes. It's rendered as Markdown using "some implementation" and [Text](Link) is the format that works.

ericholscher · 2019-07-08T23:00:42Z

readthedocs/api/v3/views.py

+        For GET it returns a serializer with many fields and on PUT/PATCH/POST,
+        it return a serializer to validate just a few fields.
+        """
+        if self.action in ('list', 'retrieve', 'superproject'):


Yea, we should add a comment at least, since it's not like the others.

ericholscher · 2019-07-08T23:01:08Z

readthedocs/api/v3/views.py

+        if self.action in ('list', 'retrieve', 'superproject'):
+            return ProjectSerializer
+
+        if self.action in ('create',):


any reason not to just use a list here? Or check for the value explicitly?

Just for consistency.

ericholscher · 2019-07-08T23:02:19Z

readthedocs/api/v3/views.py

+        the database.
+        """
+        project = serializer.save()
+        self.import_project(project, [], self.request)


What is the empty list for? We should use kwargs to make it more explicit.

I put request as first argument and make tags an optional with None as default. I think it's better now.

ericholscher · 2019-07-08T23:03:19Z

readthedocs/projects/views/mixins.py

+
+    """Helpers to import a Project."""
+
+    def import_project(self, project, tags, request):


That seems better.

ericholscher · 2019-07-08T23:03:42Z

readthedocs/projects/views/mixins.py

+
+    def import_project(self, project, tags, request):
+        """
+        Import a Project into Read the Docs.


As you noted in the PR comment above, we should note what work it assumes has already been done in the code comment as well.

ericholscher · 2019-07-08T23:04:15Z

readthedocs/projects/views/mixins.py

+
+    """Helpers to import a Project."""
+
+    def import_project(self, project, tags, request):


I might put the request first, as that is the normal order in Django in my experience.

ericholscher · 2019-07-08T23:04:46Z

readthedocs/projects/views/mixins.py

+        task_promise = chain(
+            attach_webhook.si(project.pk, user.pk),
+            update_docs,
+        )


Do we need to chain these? Seems like we still want to update the docs even if webhook setup fails, and they don't depend on each other in any real way.

We do a chain just to follow an order. If attach_webhook fails, it handles it by itself, set a message to the user and the build process continue. It does not block the update of the docs.

Anyways, this logic is not something I want to change/refactor in this PR because it's something bigger than this.

ericholscher · 2019-07-08T23:06:21Z

readthedocs/projects/views/private.py

        for field, value in list(form_data.items()):
            if field in extra_fields:
                setattr(project, field, value)
        project.save()

-        # TODO: this signal could be removed, or used for sync task
-        project_import.send(sender=project, request=self.request)
+        self.import_project(project, tags, self.request)


kwargs here too would be good.

humitos · 2019-07-09T08:12:57Z

We also need to add public-facing docs here, that is the biggest missing piece for me.

I was thinking on finish all the docs in another PR (#5895) so we can deploy this without blocking on docs, which are deployed just with a merge. It's on my list to write them.

This is needed to render the Form for POST when using BrowsableAPIRenderer and hitting /projects/

tags is a special attribute and needs to be set via `.add` method.

humitos · 2019-07-09T08:52:25Z

@ericholscher I think I made all the changes needed from your comments. I had some issues when rebasing with master, hopefully I didn't break anything :)

ericholscher · 2019-07-09T19:22:35Z

Why not just merge master into the branch?

ericholscher · 2019-07-09T19:22:57Z

now I can't really see what commits have changed since my review :/

ericholscher · 2019-07-09T19:33:26Z

Looking at it though, the changes look good to me 👍

humitos · 2019-07-09T21:57:32Z

@ericholscher I'm sorry about that. I usually do rebase against master but, I may need to change that flow because of this problem

humitos requested a review from a team June 27, 2019 11:59

humitos mentioned this pull request Jun 27, 2019

Make APIv3 permission classes modular #5858

Closed

humitos force-pushed the humitos/apiv3-import-project branch from d9f08a9 to 6302256 Compare June 27, 2019 12:07

ericholscher reviewed Jul 1, 2019

View reviewed changes

humitos commented Jul 2, 2019

View reviewed changes

humitos commented Jul 8, 2019

View reviewed changes

humitos mentioned this pull request Jul 8, 2019

APIv3 refactor some fields #5868

Merged

humitos force-pushed the humitos/apiv3-import-project branch from e81802b to 83d21ac Compare July 8, 2019 08:24

ericholscher reviewed Jul 8, 2019

View reviewed changes

humitos added 18 commits July 9, 2019 10:22

Make Project.respository.type a ChoiceField

fd6b96c

Grant permissions on 'create' action

f6073e7

This is needed to render the Form for POST when using BrowsableAPIRenderer and hitting /projects/

Use standard DRF status codes

f344d82

Refactor 'trigger_initial_build' to allow calling from other places

9081474

View/Serializer to import a Project from APIv3

6aa7eda

Endpoint to import a project

44d0f2f

Refactor Import Project to share the flow

ff5b088

Re-write test to match the new implementation (Mixin)

a5b0015

Use one serializer to validate input and a different one in the response

2c88524

Remove tags attribute before being set

58bc94e

tags is a special attribute and needs to be set via `.add` method.

Avoid elif when using return

92c0e8a

Use the same attributes for trigger_initial_build

3d70928

Call trigger_build properly

1896e33

More attributes when importing a Project

9a19a0e

Add note about superproject "action"

d550922

Make explicit comparison

f153864

Rename method and make argument a kwarg

28b7606

Solve issues after merge

b29b3b5

humitos force-pushed the humitos/apiv3-import-project branch from c44f039 to b29b3b5 Compare July 9, 2019 08:51

humitos requested a review from ericholscher July 9, 2019 08:51

ericholscher approved these changes Jul 9, 2019

View reviewed changes

humitos merged commit e574168 into master Jul 9, 2019

delete-merged-branch bot deleted the humitos/apiv3-import-project branch July 9, 2019 22:03

humitos mentioned this pull request Oct 1, 2019

APIv3 endpoint: allow to modify a Project once it's imported #5952

Merged


		"""Helpers to import a Project."""

		def import_project(self, project, tags, request):

APIv3 "Import Project" endpoint #5857

APIv3 "Import Project" endpoint #5857

Conversation

humitos commented Jun 27, 2019

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Jul 2, 2019

humitos commented Jul 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Jul 9, 2019 • edited Loading

humitos commented Jul 9, 2019

ericholscher commented Jul 9, 2019

ericholscher commented Jul 9, 2019

ericholscher commented Jul 9, 2019

humitos commented Jul 9, 2019

humitos commented Jul 9, 2019 •

edited

Loading