doctor-visits reporting fewer megafips than usual

[krivard]

Actual Behavior:

Ryan identified a problem(slack thread) with doctor-visits where fewer megafips regions than usual are being reported. The effect recedes after a few days, so megafips are being produced, just fewer than (notably) before the geo refactor.

Expected behavior

Number of megafips produced by doctor-visits in fbd87a2f497 should be roughly equal to that produced by doctor-visits in 47016ab6e. Hopefully the effect is duplicated in the synthetic data file, otherwise this will be way harder to debug.

Context

Since doctor-visits receives FIPS codes from the source, and not ZIP codes, it's clear that this problem is not due to the geo utility splitting ZIPs that overlap multiple counties. There is a note in the pull request about a threshold that seems to control the number of megafips being reported; that threshold may just need adjusting.

[chinandrew]

@mariajahja we had a comment on this on the PR, but I'm still not clear on this. I'm not sure it's related to this issue but it'd still be useful to help my understand what's going on. In geo_maps.py, the previous implementation for dealing with megacounties and excluding small counts looked something like this:

        data["ToExclude"] = data["RecentVisits"] < threshold_visits
        # some seemingly unrelated code on dropping matches for fips and stuff....
        megacounty_df = (
            data[data["ToExclude"]]
            .groupby(["ServiceDate", "StateFIPS"])
            .sum()
            .reset_index()
        )
        megacounty_df["ToExclude"] = False
        megacounty_df.rename(columns={"StateFIPS": "PatCountyFIPS"}, inplace=True)
        result = pd.concat([data, megacounty_df])
        result.drop(columns=["StateFIPS", "state_id"], inplace=True)

and returns the result goroupy by fips. I don't see any uses of ToExclude anywhere else in the code outside of these uses. However, this means you could get a result df like the following:

   Covid_like  Flu_like  Mixed  Flu1  Denominator ServiceDate PatCountyFIPS  ...  RecentVisits  ToExclude
0          44        97    907   184       197274  2020-02-01         01001  ...      197274.0       True
1         214       223   1697    25        73459  2020-02-01         01003  ...       73459.0       True
2         148       454    185    37       165151  2020-02-01         01005  ...      165151.0       True
3          20       504   1007  1467        69846  2020-02-01         01007  ...       69846.0       True
4         347       538    501  1179       169301  2020-02-01         01009  ...      169301.0       True
5         650       565    695  1194        80537  2020-02-01         01011  ...       80537.0       True
6         763       303   1530   236        49264  2020-02-01         01013  ...       49264.0       True
7         678       404    978  1152       193254  2020-02-01         01015  ...      193254.0       True
8         304       412    524  1674        17635  2020-02-01         01017  ...       17635.0       True
0        3168      3500   8024  7148      1015721  2020-02-01         01000  ...     1015721.0      False

Where do the ToExclude=True FIPS codes get removed? Update sensor looks it it just takes all the unique geo_ids and runs with it.

Again, I don't know if this has anything to do with the lack of megacounties than usual, I'm just confused

[chinandrew]

Further debugging. Using the "SYNEDI_AGG_OUTPATIENT_07022020_1455CDT.csv" test dataset in the unit tests, the old implementation returns the following groupby (which I've summed)

(Pdb) out.sum()
               Covid_like  Flu_like  Mixed  Flu1  Denominator  Pat HRR ID  RecentVisits  ToExclude
PatCountyFIPS                                                                                     
01000                1951      2007   5453  4596       290741        23.0      290741.0          0
01001                  44        97    907   184       197274         0.0      197274.0          0
01003                 214       223   1697    25        73459         1.0       73459.0          1
01005                 148       454    185    37       165151         2.0      165151.0          0
01007                  20       504   1007  1467        69846         3.0       69846.0          1
01009                 347       538    501  1179       169301         4.0      169301.0          0
01011                 650       565    695  1194        80537         5.0       80537.0          1
01013                 763       303   1530   236        49264         6.0       49264.0          1
01015                 678       404    978  1152       193254         7.0      193254.0          0
01017                 304       412    524  1674        17635         8.0       17635.0          1

filtered to ToExclude=0:

               Covid_like  Flu_like  Mixed  Flu1  Denominator  Pat HRR ID  RecentVisits  ToExclude
PatCountyFIPS                                                                                     
01000                1951      2007   5453  4596       290741        23.0      290741.0          0
01001                  44        97    907   184       197274         0.0      197274.0          0
01005                 148       454    185    37       165151         2.0      165151.0          0
01009                 347       538    501  1179       169301         4.0      169301.0          0
01015                 678       404    978  1152       193254         7.0      193254.0          0

With the new implementation:

               Covid_like  Flu_like  Mixed  Flu1  Denominator  Pat HRR ID  RecentVisits
PatCountyFIPS                                                                          
01000                1951      2007   5453  4596       290741        23.0      290741.0
01001                  44        97    907   184       197274         0.0      197274.0
01005                 148       454    185    37       165151         2.0      165151.0
01009                 347       538    501  1179       169301         4.0      169301.0
01015                 678       404    978  1152       193254         7.0      193254.0

so doesnt look like a change there, unless the ToExclude rows need to stay in (or some edge case that isn't capture here)?

In both the head of dv-package and the past commit, the call to the function is the same, so thresholds aren't different.:

geo_map.county_to_megacounty(
            data, Config.MIN_RECENT_VISITS, Config.RECENT_LENGTH
        )

@krivard is there a way I can get one of the production files from the last few days to run this on?

On an unrelated note, I think there's a couple of statements reading mapping files I forgot to delete, so will check those separately.

[krivard]

@chinandrew There are some DUA formalities for getting you access to the real data; I'll send you the details offline.

[mariajahja]

Updated the bigchunk-dev machine that runs this daily to use the latest fix from #513. Next drop tonight (2020/11/19) should run it.

[chinandrew]

Appears to be resolved