Skip to content

Remove files from cloud storage when version is wiped. #6186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,13 @@ What commit of Read the Docs is in production?
We deploy readthedocs.org from the `rel` branch in our GitHub repository. You can see the latest commits that have been deployed by looking on GitHub: https://github.com/readthedocs/readthedocs.org/commits/rel


How can I remove stale results from the search?
-----------------------------------------------

If the search results contains stale/outdated results,
you can update the search index by :doc:`wiping the build environment <guides/wipe-environment>` and then rebuilding your docs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this. I mean, if that is happening I would like that users report that to us, since it is a bug from our side. We could remove this or add something like if this is happening very often, report it.

@davidfischer what do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stsewd
I think this is a bug from Sphinx, if a .rst file is deleted, then the .html file for that shouldn't be created or deleted in the next build.

I think it would be better if a user tries wiping the environment and rebuilding the docs, if that doesn't work -- then report to us.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, as we move toward each build being totally fresh and not relying on a previous environment, this problem should naturally go away. It is possible that we need to detect when files aren't generated as part of the build and remove them. Even with this change, that won't be done exactly.



How can I avoid search results having a deprecated version of my docs?
----------------------------------------------------------------------

Expand Down
4 changes: 2 additions & 2 deletions readthedocs/builds/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
VersionAutomationRule,
)
from readthedocs.core.utils import trigger_build
from readthedocs.core.utils.general import wipe_version_via_slugs
from readthedocs.core.utils.general import _wipe_version_helper
from readthedocs.projects.models import HTMLFile
from readthedocs.search.utils import _indexing_helper

Expand Down Expand Up @@ -114,7 +114,7 @@ class VersionAdmin(admin.ModelAdmin):
def wipe_selected_versions(self, request, queryset):
"""Wipes the selected versions."""
for version in queryset:
wipe_version_via_slugs(
_wipe_version_helper(
version_slug=version.slug,
project_slug=version.project.slug
)
Expand Down
26 changes: 24 additions & 2 deletions readthedocs/core/utils/general.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,23 @@

import os

from django.conf import settings
from django.core.files.storage import get_storage_class
from django.shortcuts import get_object_or_404

from readthedocs.core.utils import broadcast
from readthedocs.projects.tasks import remove_dirs
from readthedocs.builds.models import Version


def wipe_version_via_slugs(version_slug, project_slug):
"""Wipes the given version of a given project."""
def _wipe_version_helper(version_slug, project_slug):
"""
Wipes the given version of a project.

It does two things:
* Clears the `checkouts`, `envs`, and `conda` direcories (if exist).
* Removes the html files from cloud storage.
"""
version = get_object_or_404(
Version,
slug=version_slug,
Expand All @@ -23,3 +31,17 @@ def wipe_version_via_slugs(version_slug, project_slug):
]
for del_dir in del_dirs:
broadcast(type='build', task=remove_dirs, args=[(del_dir,)])

_clear_html_files_from_cloud_storage(version)


def _clear_html_files_from_cloud_storage(version):
"""Removes html files from cloud storage for a given version of a project."""

storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
storage_path = version.project.get_storage_path(
type_='html',
version_slug=version.slug,
include_file=False,
)
storage.delete_directory(storage_path)
4 changes: 2 additions & 2 deletions readthedocs/core/views/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from django.views.static import serve as static_serve

from readthedocs.builds.models import Version
from readthedocs.core.utils.general import wipe_version_via_slugs
from readthedocs.core.utils.general import _wipe_version_helper
from readthedocs.core.resolver import resolve_path
from readthedocs.core.symlink import PrivateSymlink, PublicSymlink
from readthedocs.projects.constants import PRIVATE
Expand Down Expand Up @@ -72,7 +72,7 @@ def wipe_version(request, project_slug, version_slug):
raise Http404('You must own this project to wipe it.')

if request.method == 'POST':
wipe_version_via_slugs(
_wipe_version_helper(
version_slug=version_slug,
project_slug=project_slug,
)
Expand Down
14 changes: 7 additions & 7 deletions readthedocs/rtd_tests/tests/test_core_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from readthedocs.builds.constants import LATEST
from readthedocs.builds.models import Version
from readthedocs.core.utils import slugify, trigger_build
from readthedocs.core.utils.general import wipe_version_via_slugs
from readthedocs.core.utils.general import _wipe_version_helper
from readthedocs.projects.models import Project
from readthedocs.projects.tasks import remove_dirs

Expand Down Expand Up @@ -200,8 +200,8 @@ def test_slugify(self):
)

@mock.patch('readthedocs.core.utils.general.broadcast')
def test_wipe_version_via_slug(self, mock_broadcast):
wipe_version_via_slugs(
def test_wipe_version_helper(self, mock_broadcast):
_wipe_version_helper(
version_slug=self.version.slug,
project_slug=self.version.project.slug
)
Expand All @@ -221,20 +221,20 @@ def test_wipe_version_via_slug(self, mock_broadcast):
)

@mock.patch('readthedocs.core.utils.general.broadcast')
def test_wipe_version_via_slug_wrong_param(self, mock_broadcast):
def test_wipe_version_helper_wrong_param(self, mock_broadcast):
self.assertFalse(Version.objects.filter(slug='wrong-slug').exists())
with self.assertRaises(Http404):
wipe_version_via_slugs(
_wipe_version_helper(
version_slug='wrong-slug',
project_slug=self.version.project.slug
)
mock_broadcast.assert_not_called()

@mock.patch('readthedocs.core.utils.general.broadcast')
def test_wipe_version_via_slugs_same_version_slug_with_diff_proj(self, mock_broadcast):
def test_wipe_version_helper_same_version_slug_with_diff_proj(self, mock_broadcast):
project_2 = get(Project)
version_2 = get(Version, project=project_2, slug=self.version.slug)
wipe_version_via_slugs(
_wipe_version_helper(
version_slug=version_2.slug,
project_slug=project_2.slug,
)
Expand Down