Skip to content

Refactor cdc_covidnet to use geo utils #311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 19, 2020
Merged

Refactor cdc_covidnet to use geo utils #311

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8a1112d
Refactor to Geomapper function
chinandrew Oct 14, 2020
d6f43ae
Change na behavior to be consistent with previous implementation
chinandrew Oct 15, 2020
d8c491a
Remove unused variable
chinandrew Oct 15, 2020
893aafc
Update comment on column assignment
chinandrew Oct 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion cdc_covidnet/delphi_cdc_covidnet/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@

from . import run
from . import api_config
from . import geo_maps
from . import update_sensor
from . import covidnet

Expand Down
45 changes: 0 additions & 45 deletions cdc_covidnet/delphi_cdc_covidnet/geo_maps.py
View file
Open in desktop

This file was deleted.

24 changes: 15 additions & 9 deletions cdc_covidnet/delphi_cdc_covidnet/update_sensor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,10 @@
import numpy as np
import pandas as pd

from delphi_utils import read_params
from delphi_utils import read_params, GeoMapper
import covidcast
from .api_config import APIConfig
from .covidnet import CovidNet
from .geo_maps import GeoMaps
from .constants import SIGNALS

def write_to_csv(data: pd.DataFrame, out_name: str, output_path: str):
Expand Down Expand Up @@ -49,17 +48,18 @@ def write_to_csv(data: pd.DataFrame, out_name: str, output_path: str):


def update_sensor(
state_files: List[str], mmwr_info: pd.DataFrame,
output_path: str, static_path: str,
start_date: datetime, end_date: datetime) -> pd.DataFrame:
state_files: List[str],
mmwr_info: pd.DataFrame,
output_path: str,
start_date: datetime,
end_date: datetime) -> pd.DataFrame:
"""
Generate sensor values, and write to csv format.

Args:
state_files: List of JSON files representing COVID-NET hospitalization data for each state
mmwr_info: Mappings from MMWR week to actual dates, as a pd.DataFrame
output_path: Path to write the csvs to
static_path: Path for the static geographic fiels
start_date: First sensor date (datetime.datetime)
end_date: Last sensor date (datetime.datetime)

Expand All @@ -85,9 +85,15 @@ def update_sensor(
]

# Set state id to two-letter abbreviation
geo_map = GeoMaps(static_path)
hosp_df = geo_map.state_name_to_abbr(hosp_df)

gmpr = GeoMapper()
hosp_df = gmpr.add_geocode(hosp_df,
from_col=APIConfig.STATE_COL,
from_code="state_name",
new_code="state_id",
dropna=False)
# To preserve column order, reassign original column and drop new one
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a test that fails if this line is omitted?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if hosp_df[APIConfig.STATE_COL] = hosp_df["state_id"].str.upper() is omitted, tests fail because APIConfig.STATE_COL isn't the two letter abbreviation.

if hosp_df.drop("state_id", axis=1, inplace=True) is omitted, tests fail because dimensions are wrong.

I also actually just realized column order for these fields doesn't matter since there's a hosp_df.set_index(["date", "geo_id"], inplace=True) right after, so this is actually just to avoid name collisions since there isn't currently functionality to replace a column while keeping the name the same. I've updated the comment to reflect that.

hosp_df[APIConfig.STATE_COL] = hosp_df["state_id"].str.upper()
hosp_df.drop("state_id", axis=1, inplace=True)
assert not hosp_df.duplicated(["date", "geo_id"]).any(), "Non-unique (date, geo_id) pairs"
hosp_df.set_index(["date", "geo_id"], inplace=True)

Expand Down
38 changes: 0 additions & 38 deletions cdc_covidnet/tests/test_geomaps.py
View file
Open in desktop

This file was deleted.

5 changes: 1 addition & 4 deletions cdc_covidnet/tests/test_update_sensor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,7 @@ def test_syn_update_sensor(self):
end_date = datetime(year=2020, month=3, day=17)

# Generate the csvs
hosp_df = update_sensor(
state_files, mmwr_info,
temp_dir, STATIC_DIR,
start_date, end_date)
hosp_df = update_sensor(state_files, mmwr_info, temp_dir, start_date, end_date)

# Check dataframe returned
assert hosp_df.index.nlevels == 2
Expand Down