Use tar to extract cache, write by chunks and new build state
#6793
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have experimented memory leak issues when extracting a file using the Python API. We are using regular `tar` for this operation now since the test done in the server with 2.5Gb has shown better results regarding memory usage and extracting time as well.
We were loading all the file in memory and then dumping it into the disk. Now, we just read a chunk and dump that chunk into the disk.
humitos
changed the title
tar command to extract cached environmenttar to extract cache, write by chunks and new build state
humitos
commented
Mar 20, 2020
Instead of downloading the file into a temporary file (`storage.open` and then `.read`) to save the file into another temporary file (our loop reading by chunks) we just use the `storage.open` response as the input of the `tarfile.open` function and extract it from there.
This was referenced Mar 20, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have experimented memory leak issues when extracting a file using the Python API. We are using regular
tarfor this operation now since the test done in the server with 2.5Gb has shown better results regarding memory usage and extracting time as well.Besides, instead of reading the whole file into memory (
fd.read()), we are iterating over its chunks and saving these small chunks into disk.Finally, we are adding a new build status "Pulling cache" for now to communicate that the build is not in "Triggered" state, but it's actually doing something.
Related to #6763