[Backfill corrections] Restore original params if pipeline exits with an error #1763
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Modify params dirs (for convenience of working with user-provided paths) only inside the docker container so that the original user-provided params file is preserved regardless of pipeline exit status or other unintended behavior.
Changelog
Fixes
Previously modifications to dir paths were performed on the original
params.json
outside the docker container. Restoring it to its original state relied on ateardown
step that didn't run if the pipeline exited with an error.Instead of solving this by capturing and rethrowing errors, it was easier to limit params modifications to a copy of the mounted
params.json
inside the docker container. The copy is not linked to any file on the host, so changes to it are not propagated back to the originalparams.json
.I can't find any built-in support for mounting a file from the host but NOT propagating in-container changes back to the host, so
params.json
is mounted under another name and then copied manually.