Keep thresholded counties for future calculations

[mariajahja]

Description

Fix bug where rows in thresholded counties are dropped in megacounty creation. When dropped, the signal backfill is incorrectly computed on 0-padded dates.

Changelog

• extract megacounties and concatenate to original data in geo_maps::county_to_megacounty
• update unit test test_geo_map::test_county_to_megacounty

Unit tests pass. Checks on state/msa/hrr match exactly to current behavior. More counties are produced (due to increased denominator from backwards padding).

@krivard Do we need to regenerate past issues of DV, up till when we switched to the geomapper? The data isn't wrong, but it is different, since it would have been produced only on larger counties, without the correct dynamic backwards padding.

[chinandrew]

Would leave a short comment explaining why the concat is being done, otherwise looks good.

[krivard]

Do we need to regenerate past issues of DV

hmm. I'd rather not, but I also don't want people using past issues for training to get a nasty surprise. How obnoxious is it to produce them?

[mariajahja]

Would take an afternoon to regenerate the files, but then I'd have to hand off to someone else to correctly insert issues into the database. I'm happy to help, but not sure if this is a priority at the moment.

[krivard]

let's hold off then -- if you finish your december okrs then pick it up, otherwise it can wait either until january, or until someone inquires, whichever comes first.

(loading them into the db is my job 🙂 )

[mariajahja]

Sounds good -- will put it on my list so I don't forget. Just a note that the automation has been updated -- so hopefully tomorrow we see better behavior!

[mariajahja]

DV from 2020-11-09 to 2020-12-13 will be reuploaded as part of a database fix.