Skip to content

Remove build artifacts and repo storage after build #5646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
agjohnson opened this issue Apr 30, 2019 · 1 comment · Fixed by #5680
Closed
2 of 3 tasks

Remove build artifacts and repo storage after build #5646

agjohnson opened this issue Apr 30, 2019 · 1 comment · Fixed by #5680
Labels
Accepted Accepted issue on our roadmap Improvement Minor improvement to code
Milestone

Comments

@agjohnson
Copy link
Contributor

agjohnson commented Apr 30, 2019

One of the tests that we'd like to perform, in order to address our build disk usage, is to remove build artifacts and repo storage directly after the build process ends. This will reduce storage, remove the need for wiping environments, and will further allow us to isolate build processes. We do need to test for the performance hit of continually re-fetching repositories and pip/etc fresh each build. The trade off eventually might be more time on the build servers, but with vastly smaller storage disks.

  • Add a feature flag to remove these files after build for select projects on the feature flag
  • Add a task chain that calls for build server delete of files after the build process completes and files are synced to webs/blob storage
  • Ensure we're deleting the project build artifacts, repo checkout, and pip cache
@agjohnson agjohnson added Improvement Minor improvement to code Accepted Accepted issue on our roadmap labels Apr 30, 2019
@agjohnson agjohnson added this to the 3.5 milestone Apr 30, 2019
@humitos
Copy link
Member

humitos commented Jul 25, 2019

Ensure we're deleting the project build artifacts, repo checkout, and pip cache

One part of this item was not done: remove build artifacts, because of a concurrency problem. See https://github.com/rtfd/readthedocs.org/blob/25191ff2f6b15e01eeb2617b239d724b04c98d78/readthedocs/projects/tasks.py#L1433-L1438

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Accepted Accepted issue on our roadmap Improvement Minor improvement to code
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants