Research and fix: JHU CSSE added borough support for NYC

[krivard]

The CDC meeting this week announced that JHU CSSE will be changing its reporting of cases and deaths in NYC to include all boroughs instead of just Manhattan.

• Determine whether this change will cause problems for us
• Decide how and when to fix it

[krivard]

@jsharpna?

[dshemetov]

@krivard I can take a look at this after I get a handle on the Unassigned situation.

[krivard]

@katie @jingjing Tang I’m not sure I remember who’s handling the JHU signals, but this morning, Kings County’s (7-day avg) incidence rate is ridiculous: over 8 million.

 geo_value                        signal time_value direction      issue lag   value stderr sample_size data_source
     36047 confirmed_7dav_incidence_prop 2020-09-09        NA 2020-09-10   1 8828571     NA          NA    jhu-csse

Ryan 5 minutes ago
It looks like other counties that comprise NY are giving ridiculous numbers this morning, as well.
This might have to do with the fact that JHU now started separating out counties in NY whereas before they weren’t?
cc @addison @alex Reinhart who if memory serves me right, thought about this at one time too.

[krivard]

@huisaddison is picking up a hot fix; in the meantime I'm scrubbing the affected prop entries from the DB

[huisaddison]

Issue:

• static/fips_population.csv took all of the population from non-Manhattan (36061) and put them in Manhattan. I actually left 1 person in each borough to avoid a divide by zero error, which was our undoing (an actual divide by zero may have just halted the pipeline)

EDIT: actually may have included 1 in pop to avoid 0/0 division... but I could have handled it more elegantly

Fix:

• Re-generate static/fips_population.csv leaving people in their home boroughs instead of relocating them to Manhattan

[capnrefsmmat]

Can you do that in a way that doesn't break the old JHU data? Or did they change the historical data too?

[huisaddison]

Yeah they changed historical data too, cases taken from here https://health.data.ny.gov/Health/New-York-State-Statewide-COVID-19-Testing/xdss-u53e and deaths taken from here: https://github.com/nychealth/coronavirus-data/blob/master/boro/boroughs-case-hosp-death.csv it's documented in their git issue: CSSEGISandData/COVID-19#3084

[huisaddison]

I think that we also asked Viz to implement NYC-specific visualization for counts / prop to account for this. Have we already done work on the Viz side to account for the fact that we no longer need to do special hacks? If not, it may be confusing to website users

[krivard]

@huisaddison Viz is not using JHU for anything except Puerto Rico (they use the USAFacts+JHU combo signal for cases/deaths)

I'm calling this resolved