new data source: covid hospitalization #292
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sync to delphi
sync to delphi
- nothing here is deployed in prod or runs in prod - this will be used by unit and integration tests later on - including some realistic sample data to fully exercise all the code paths - also includes readme with basic source info and plan for acquisition
- extract out all the parts that make network calls - this class will be swapped out with a mock in unit and integration tests to prevent hitting real external services during testing - tests included
- extract out all the database interactions - this will be mocked out in unit tests (but not integration tests, as they will interact with a real (local) database) - implemented as a context manager which ensures graceful cleanup on both success and failure - care is taken to only commit transaction on success - ddl included (this will require a one-time manual run on the server) - tests included
- checks for new data - bails if nothing new, otherwise fetches and stores in database - tested using mocks for database and network operations, using sample data for more realistic testing - this should be scheduled in automation, suggest 1-2 times per day `python3 -m delphi.epidata.acquisition.covid_hosp.update`
- after this commit, acquisition can be scheduled in automation
- just another data source, following all the same conventions as before - wow there are so many columns in this dataset... - tests to follow shortly, after the api clients are updated
- updates all the client libraries - coffee recompiled to js using latest version of coffeescript
- tests server-side api behavior, like handling of versioning (issues) - tests acquisition "pipeline", which uses test data to download and store a dataset
- adds new page for the `covid_hosp` endpoint - adds entry in the parent readme for this new covid-19 data source
krivard
reviewed
Nov 18, 2020
committing suggestion from @krivard Co-authored-by: Katie Mazaitis <[email protected]>
- use built-in fuzzy comparison instead of taking abs and less than
- these names aren't actually changed
krivard
suggested changes
Nov 19, 2020
dont hardcode column count Co-authored-by: Katie Mazaitis <[email protected]>
- line was too long, split up
- converted from es6 to es5 by babel
|
@krivard thanks for the review, should be good to go! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new data source to the API, start to finish.
"COVID-19 Reported Patient Impact and Hospital Capacity by State Timeseries" dataset provided by the US Department of
Health & Human Services via healthdata.gov.
https://healthdata.gov/dataset/covid-19-reported-patient-impact-and-hospital-capacity-state-timeseries
Fixes #288
See individual commits for more details on what happens at each step.
covid_hospto api serverrepo-wide unit tests: ✔ All 52 tests passed!
repo-wide integration tests: ✔ All 29 tests passed!