Merge pull request #1831 from cmu-delphi/jhu-deactivation

JHU deactivation: geomap + validator

Author: minhkhul

_delphi_utils_python/data_proc/geomap/README.md

@@ -24,7 +24,7 @@ We support the following geocodes.
   - We are reserving 10001-10099 for states codes of the form 100XX where XX is the FIPS code for the state (the current smallest CBSA is 10100). In the case that the CBSA codes change then it should be verified that these are not used.
 - State codes are a series of equivalent identifiers for US state. They include the state name, the state number (state_id), and the state two-letter abbreviation (state_code). The state number is the state FIPS code. See [here](https://en.wikipedia.org/wiki/List_of_U.S._state_and_territory_abbreviations) for more.
 - The Hospital Referral Region (HRR) and the Hospital Service Area (HSA). More information [here](https://www.dartmouthatlas.org/covid-19/hrr-mapping/).
-- The JHU signal contains its own geographic identifier, labeled the UID. Documentation is provided at [their repo](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic). Its FIPS codes depart in some special cases, so we produce manual changes listed below.
+FIPS codes depart in some special cases, so we produce manual changes listed below.
 
 ## Source files
 
@@ -34,28 +34,20 @@ The source files are requested from a government URL when `geo_data_proc.py` is
 - ZIP -> HRR -> HSA crosswalk file comes from the 2018 version at the [Dartmouth Atlas Project](https://atlasdata.dartmouth.edu/static/supp_research_data).
 - FIPS -> MSA crosswalk file comes from the September 2018 version of the delineation files at the [US Census Bureau](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html).
 - State Code -> State ID -> State Name comes from the ANSI standard at the [US Census](https://www.census.gov/library/reference/code-lists/ansi.html#par_textimage_3). The first two digits of a FIPS codes should match the state code here.
-- JHU UID -> FIPS comes from [the JHU documentation](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic). We have to do some hand modifications to the JHU UID because the mapping to FIPS isn't always consistent.
+
 
 ## Derived files
 
 The rest of the crosswalk tables are derived from the mappings above. We provide crosswalk functions from granular to coarser codes, but not the other way around. This is because there is no information gained when crosswalking from coarse to granular.
 
-## JHU UID mapping changes
 
-- Dukes and Nantucket counties in Massachusets are aggregated, so we split them with population-proportional weights (approximately 2/3 Dukes and 1/3 Nantucket).
-- The same procedure is followed by Kansas City and four of its counties.
-- Kusilvak, Alaska is mapped to the FIPS code 02270.
-- Ogalala Lakota, South Dakota is mapped to the FIPS code 46113.
-- Utah reports at a territory level, so we only report it at in a state level megaFIPS 49000.
-- JHU places cases and deaths that cannot be localized to a single county into "Out of State" and "Unassigned" categories. We map these to the "megaFIPS" code XX000, where XX is the state FIPS code. This way, the data is recovered when aggregating up to the state level, but does not interfere with other counties.
 
 ## Deprecated source files
 
 - ZIP to FIPS to HRR to states: `02_20_uszips.csv` comes from a version of the table [here](https://simplemaps.com/data/us-zips) modified by Jingjing to include population weights.
   - The `02_20_uszips.csv` file is based on the newest consensus data including 5-digit zipcode, fips code, county name, state, population, HRR, HSA (I downloaded the original file from [here](https://simplemaps.com/data/us-zips). This file matches best to the most recent (2020) situation in terms of the population. But there still exist some matching problems. I manually checked and corrected those lines (~20) with [zip-codes](https://www.zip-codes.com/zip-code/58439/zip-code-58439.asp). The mapping from 5-digit zipcode to HRR is based on the file in 2017 version downloaded from [here](https://atlasdata.dartmouth.edu/static/supp_research_data).
 - ZIP -> FIPS is provided by [huduser.gov](https://www.huduser.gov/portal/datasets/usps_crosswalk.html) for zip -> fips?
 - FIPS county population data from [US Census Bureau](http://www.census.gov/programs-surveys/popest/technical-documentation/methodology.html). Details of Bedford, Virginia counting [here](https://www.census.gov/programs-surveys/geography/technical-documentation/county-changes.html).
-- JHU UID crosswalk table [here](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic)
 - CBSA -> FIPS crosswalk from [here](https://data.nber.org/data/cbsa-fips-county-crosswalk.html) (the file is `cbsatocountycrosswalk.csv`).
 - MSA tables from March 2020 [here](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html). This file seems to differ in a few fips codes from the source for the 02_20_uszip file which Jingjing constructed. There are at least 10 additional fips in 03_20_msa that are not in the uszip file, and one of the msa codes seems to be incorrect: 49020 (a google search confirms that it is incorrect in uszip and correct in the census data).
 - MSA tables from 2019 [here](https://apps.bea.gov/regional/docs/msalist.cfm)

_delphi_utils_python/data_proc/geomap/geo_data_proc.py

@@ -27,7 +27,6 @@
 ZIP_HSA_HRR_URL = "https://atlasdata.dartmouth.edu/downloads/geography/ZipHsaHrr18.csv.zip"
 ZIP_HSA_HRR_FILENAME = "ZipHsaHrr18.csv"
 FIPS_MSA_URL = "https://www2.census.gov/programs-surveys/metro-micro/geographies/reference-files/2018/delineation-files/list1_Sep_2018.xls"
-JHU_FIPS_URL = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/UID_ISO_FIPS_LookUp_Table.csv"
 STATE_CODES_URL = "http://www2.census.gov/geo/docs/reference/state.txt?#"
 FIPS_POPULATION_URL = f"https://www2.census.gov/programs-surveys/popest/datasets/2010-{YEAR}/counties/totals/co-est{YEAR}-alldata.csv"
 FIPS_PUERTO_RICO_POPULATION_URL = "https://www2.census.gov/geo/docs/maps-data/data/rel/zcta_county_rel_10.txt?"
@@ -57,7 +56,6 @@
 STATE_POPULATION_OUT_FILENAME = "state_pop.csv"
 HHS_POPULATION_OUT_FILENAME = "hhs_pop.csv"
 NATION_POPULATION_OUT_FILENAME = "nation_pop.csv"
-JHU_FIPS_OUT_FILENAME = "jhu_uid_fips_table.csv"
 
 
 def create_fips_zip_crosswalk():
@@ -111,101 +109,6 @@ def create_fips_msa_crosswalk():
     msa_df.sort_values(["fips", "msa"]).to_csv(join(OUTPUT_DIR, FIPS_MSA_OUT_FILENAME), columns=["fips", "msa"], index=False)
 
 
-def create_jhu_uid_fips_crosswalk():
-    """Build a crosswalk table from JHU UID to FIPS."""
-    # These are hand modifications that need to be made to the translation
-    # between JHU UID and FIPS. See below for the special cases information
-    # https://cmu-delphi.github.io/delphi-epidata/api/covidcast-signals/jhu-csse.html#geographical-exceptions
-    hand_additions = pd.DataFrame(
-        [
-            {
-                "jhu_uid": "84070002",
-                "fips": "25007",  # Split aggregation of Dukes and Nantucket, Massachusetts
-                "weight": 16535 / (16535 + 10172),  # Population: 16535
-            },
-            {
-                "jhu_uid": "84070002",
-                "fips": "25019",
-                "weight": 10172 / (16535 + 10172),  # Population: 10172
-            },
-            {
-                "jhu_uid": "84070003",
-                "fips": "29095",  # Kansas City, Missouri
-                "weight": 674158 / 1084897,  # Population: 674158
-            },
-            {
-                "jhu_uid": "84070003",
-                "fips": "29165",
-                "weight": 89322 / 1084897,  # Population: 89322
-            },
-            {
-                "jhu_uid": "84070003",
-                "fips": "29037",
-                "weight": 99478 / 1084897,  # Population: 99478
-            },
-            {
-                "jhu_uid": "84070003",
-                "fips": "29047",
-                "weight": 221939 / 1084897,  # Population: 221939
-            },
-            # Kusilvak, Alaska
-            {"jhu_uid": "84002158", "fips": "02270", "weight": 1.0},
-            # Oglala Lakota
-            {"jhu_uid": "84046102", "fips": "46113", "weight": 1.0},
-            # Aggregate Utah territories into a "State FIPS"
-            {"jhu_uid": "84070015", "fips": "49000", "weight": 1.0},
-            {"jhu_uid": "84070016", "fips": "49000", "weight": 1.0},
-            {"jhu_uid": "84070017", "fips": "49000", "weight": 1.0},
-            {"jhu_uid": "84070018", "fips": "49000", "weight": 1.0},
-            {"jhu_uid": "84070019", "fips": "49000", "weight": 1.0},
-            {"jhu_uid": "84070020", "fips": "49000", "weight": 1.0},
-        ]
-    )
-    # Map the Unassigned category to a custom megaFIPS XX000
-    unassigned_states = pd.DataFrame(
-        {"jhu_uid": str(x), "fips": str(x)[-2:].ljust(5, "0"), "weight": 1.0}
-        for x in range(84090001, 84090057)
-    )
-    # Map the Out of State category to a custom megaFIPS XX000
-    out_of_state = pd.DataFrame(
-        {"jhu_uid": str(x), "fips": str(x)[-2:].ljust(5, "0"), "weight": 1.0}
-        for x in range(84080001, 84080057)
-    )
-    # Map the Unassigned and Out of State categories to the cusom megaFIPS 72000
-    puerto_rico_unassigned = pd.DataFrame(
-        [
-            {"jhu_uid": "63072888", "fips": "72000", "weight": 1.0},
-            {"jhu_uid": "63072999", "fips": "72000", "weight": 1.0},
-        ]
-    )
-    cruise_ships = pd.DataFrame(
-        [
-            {"jhu_uid": "84088888", "fips": "88888", "weight": 1.0},
-            {"jhu_uid": "84099999", "fips": "99999", "weight": 1.0},
-        ]
-    )
-
-
-    jhu_df = pd.read_csv(JHU_FIPS_URL, dtype={"UID": str, "FIPS": str}).query("Country_Region == 'US'")
-    jhu_df = jhu_df.rename(columns={"UID": "jhu_uid", "FIPS": "fips"}).dropna(subset=["fips"])
-
-    # FIPS Codes that are just two digits long should be zero filled on the right.
-    # These are US state codes (XX) and the territories Guam (66), Northern Mariana Islands (69),
-    # Virgin Islands (78), and Puerto Rico (72).
-    fips_territories = jhu_df["fips"].str.len() <= 2
-    jhu_df.loc[fips_territories, "fips"] = jhu_df.loc[fips_territories, "fips"].str.ljust(5, "0")
-
-    # Drop the JHU UIDs that were hand-modified
-    manual_correction_ids = pd.concat([hand_additions, unassigned_states, out_of_state, puerto_rico_unassigned, cruise_ships])["jhu_uid"]
-    jhu_df.drop(jhu_df.index[jhu_df["jhu_uid"].isin(manual_correction_ids)], inplace=True)
-
-    # Add weights of 1.0 to everything not in hand additions, then merge in hand-additions
-    # Finally, zero fill FIPS
-    jhu_df["weight"] = 1.0
-    jhu_df = pd.concat([jhu_df, hand_additions, unassigned_states, out_of_state, puerto_rico_unassigned])
-    jhu_df["fips"] = jhu_df["fips"].astype(int).astype(str).str.zfill(5)
-    jhu_df.sort_values(["jhu_uid", "fips"]).to_csv(join(OUTPUT_DIR, JHU_FIPS_OUT_FILENAME), columns=["jhu_uid", "fips", "weight"], index=False)
-
 
 def create_state_codes_crosswalk():
     """Build a State ID -> State Name -> State code crosswalk file."""
@@ -659,7 +562,6 @@ def clear_dir(dir_path: str):
     create_fips_zip_crosswalk()
     create_zip_hsa_hrr_crosswalk()
     create_fips_msa_crosswalk()
-    create_jhu_uid_fips_crosswalk()
     create_state_codes_crosswalk()
     create_state_hhs_crosswalk()
     create_fips_population_table()