Skip to content

[Backfill corrections] Restore original params if pipeline exits with an error #1763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 18, 2023

Conversation

nmdefries
Copy link
Contributor

@nmdefries nmdefries commented Jan 17, 2023

Description

Modify params dirs (for convenience of working with user-provided paths) only inside the docker container so that the original user-provided params file is preserved regardless of pipeline exit status or other unintended behavior.

Changelog

  • Makefile

Fixes

Previously modifications to dir paths were performed on the original params.json outside the docker container. Restoring it to its original state relied on a teardown step that didn't run if the pipeline exited with an error.

Instead of solving this by capturing and rethrowing errors, it was easier to limit params modifications to a copy of the mounted params.json inside the docker container. The copy is not linked to any file on the host, so changes to it are not propagated back to the original params.json.

I can't find any built-in support for mounting a file from the host but NOT propagating in-container changes back to the host, so params.json is mounted under another name and then copied manually.

@nmdefries nmdefries requested a review from krivard January 17, 2023 17:58
Copy link
Contributor

@krivard krivard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! It's always great when you can drop a target without loss of functionality.

Suggestion optional; probably works either way.

@nmdefries nmdefries marked this pull request as ready for review January 18, 2023 17:06
@nmdefries
Copy link
Contributor Author

@krivard This is ready to merge

@krivard krivard merged commit 4641604 into main Jan 18, 2023
@krivard krivard deleted the ndefries/backfill/restore-params-finally branch January 18, 2023 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants