Skip to content

Revise epix_merge to always fit in last-version-carried-forward framework, merge additional metadata #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brookslogan opened this issue Jul 6, 2022 · 1 comment
Assignees
Labels
op-semantics Operational semantics; many potentially breaking changes here P1 medium priority

Comments

@brookslogan
Copy link
Contributor

as_of and other planned epi_archive operations assume we formed an archive using last-version-carry-forwardable data. Using epix_merge without all=TRUE and locf=TRUE breaks this.

locf=TRUE can produce a result that gives inconsistent as_of results with the input archives if the user has NAs in the input archive non-key columns. We want to locf only for rows that didn't exist before, at least when interpolating versions, and always want to locf when interpolating versions. However, there are other cases, such as when one archive has more up-to-date version data than the other, where we wonder about what, if any, extrapolation we should be doing; locf should be replaced with an observed_versions_end_conflict arg that allows various options there (e.g., to raise an error if one input isn't as fresh, to insert an all-NA version immediately after the last version in the less fresh input, to extrapolate with last-observation-carried-forward, or to truncate versions from the fresher input).

Additionally, epix_merge should appropriately check input metadata and set result metadata: the DT key, geo and time types, clobberable and observed versions, etc.

@brookslogan
Copy link
Contributor Author

Closed by #101.

@brookslogan brookslogan added P1 medium priority op-semantics Operational semantics; many potentially breaking changes here labels Aug 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
op-semantics Operational semantics; many potentially breaking changes here P1 medium priority
Projects
None yet
Development

No branches or pull requests

1 participant