Skip to content

Feature: Add streaming smoothing and cumulative/incidence derivations to server #599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 22 commits into from
Closed

Feature: Add streaming smoothing and cumulative/incidence derivations to server #599

wants to merge 22 commits into from

Conversation

dshemetov
Copy link
Contributor

@dshemetov dshemetov commented Jun 16, 2021

Fixes #607
Fixes #608

Prerequisites:

  • Unless it is a documentation hotfix it should be merged against the dev branch
  • Branch is up-to-date with the branch to be merged with, i.e. dev
  • Build is successful
  • Code is cleaned up and formatted

Summary

Adds support for streaming 7-day averaging and differencing calculations to the Flask server. Relies on #564 for signal name aliasing.

The following signals are currently supported (based on the recent meta-table):

google-symptoms	ageusia_raw_search	TRUE	ageusia_smoothed_search
google-symptoms	anosmia_raw_search	TRUE	anosmia_smoothed_search
google-symptoms	sum_anosmia_ageusia_raw_search sum_anosmia_ageusia_smoothed_search

jhu-csse	confirmed_incidence_num	TRUE	confirmed_7dav_cumulative_num
jhu-csse	confirmed_incidence_num	TRUE	confirmed_7dav_incidence_num
jhu-csse	confirmed_incidence_num	TRUE	confirmed_cumulative_num
jhu-csse	deaths_incidence_num	TRUE	deaths_7dav_cumulative_num
jhu-csse	deaths_incidence_num	TRUE	deaths_7dav_incidence_num
jhu-csse	deaths_incidence_num	TRUE	deaths_cumulative_num

usa-facts	confirmed_incidence_num	TRUE	confirmed_7dav_cumulative_num
usa-facts	confirmed_incidence_num	FALSE	confirmed_7dav_incidence_num
usa-facts	confirmed_incidence_num	FALSE	confirmed_cumulative_num
usa-facts	deaths_incidence_num	TRUE	deaths_7dav_cumulative_num
usa-facts	deaths_incidence_num	FALSE	deaths_7dav_incidence_num
usa-facts	deaths_incidence_num	FALSE	deaths_cumulative_num

There may be others, but these were immediately apparent.

TODO:

  • figure out how to use the name aliasing feature and handle "source:*" requests
  • choose the row ordering as they are streamed from the database
  • make sure "issue" and "as_of" params are handled
  • reimplement derivable signal logic: add a "derivable" column to the meta-table, use signal base for the "derivable" signal, and use "is_smoothed", "is_cumulative" to determine which transformations to apply
  • change the base signals in the db / meta table from incidence to cumulative

@sgratzl sgratzl changed the base branch from dev to sgratzl/covidcast_meta_file June 17, 2021 06:26
@sgratzl sgratzl added api change affect the API and its responses enhancement labels Jun 17, 2021
@sgratzl sgratzl closed this Jul 13, 2021
@sgratzl sgratzl deleted the branch cmu-delphi:sgratzl/covidcast_meta_file July 13, 2021 14:38
@sgratzl
Copy link
Member

sgratzl commented Jul 13, 2021

@dshemetov apparently if the base branch is deleted and it is a fork GitHub will close the PR instead of changing the base branch. Please open another one

@krivard
Copy link
Contributor

krivard commented Jul 13, 2021

you have write permissions here, please use a branch instead of a fork

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change affect the API and its responses enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants