Backfill/claims_hosp

[jingjtang]

Description

Generate and store county level backfill file locally (On MIDAS) for claims_hosp. The newly added function will check the input file daily and store backfill file on a daily basis. Every 4 weeks, we will check the daily files and merge them into a combined file.

Changelog

Itemize code/test/documentation changes and files added/removed.

• backfill.py, run.py modify the current code, to save the county level intermediate file to backfill_dir before generating signals if there are new input raw files added by DV.
• setup.py add pyarrow as a necessary package so as to save the backfill file in the parquet format.
• tests/test_backfill.py add the unit tests for backfill related helper functions.
• params.json.template add backfill_dir, backfill_merge_day

[nmdefries]

@jingjtang Can you resolve the conflict with main? Looks like Brian changed the receiving_dir in tandem; I'm not sure which is more up to date.

Added the daily file format changes.

[jingjtang]

@jingjtang Can you resolve the conflict with main? Looks like Brian changed the receiving_dir in tandem; I'm not sure which is more up to date.

Added the daily file format changes.

Use the one that Brian uses, that one should be more up-to-date

[krivard]

👍