From 4a75ba3b0ea5fdc1b02505ade2e8b8a338989947 Mon Sep 17 00:00:00 2001 From: Kathryn M Mazaitis Date: Tue, 29 Sep 2020 15:58:59 -0400 Subject: [PATCH 1/8] Release under MIT license --- LICENSE | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 000000000..92a5e44d9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2018 The Delphi Group at Carnegie Mellon University + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. From 95126755eee4a03a55e0c577f2a7ce6a3a94cd8b Mon Sep 17 00:00:00 2001 From: Kathryn M Mazaitis Date: Tue, 29 Sep 2020 16:38:26 -0400 Subject: [PATCH 2/8] Split off contribution guide; add context to README --- CONTRIBUTING.md | 86 +++++++++++++++++++++++++++++++++++++++++++ README.md | 98 +++++++------------------------------------------ 2 files changed, 100 insertions(+), 84 deletions(-) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..0d6e84000 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,86 @@ +# How to contribute to this repository + +## General workflow for indicators creation and deployment + +**tl;dr** + +1. Create your new indicator branch from `main`. +2. Build it using the appropriate template, following the guidelines in the included README.md and REVIEW.md files. +3. Make some stuff! +4. When your stuff works, push your `dev-*` branch to remote for review. +5. Consult with a platform engineer for the remaining production setup needs. They will create a branch called `deploy-*` for your indicator. +6. Initiate a pull request against this new branch. +7. Following [the source documentation template](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md), create public API documentation for the source. You can submit this as a pull request against the delphi-epidata repository. +8. If your peers like the code, the documentation is ready, and Jenkins approves, deploy your changes by merging the PR. +9. An admin will propagate your successful changes to `main`. +10. Rejoice! + +### Starting out + +The `main` branch should contain up-to-date code and supporting libraries. This should be your starting point when creating a new indicator. + +```shell +# Hint +# +git checkout main +git checkout -b dev-my-feature-branch +``` + +### Creating your indicator + +Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach. + +- Consult your peers with questions! :handshake: + +Once you have something that runs locally and passes tests you set up your remote branch eventual review and production deployment. + +```shell +# Hint +# +git push -u origin dev-my-feature-branch +``` + +You can then set draft public API documentation for people who would fetch this +data from the API. Public API documentation is kept in the delphi-epidata +repository, and there is a [template Markdown +file](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md) +that outlines the features that need to be documented. You can create a pull +request to add a new file to `docs/api/covidcast-signals/` for your source. Our +goal is to have public API documentation for the data at the same time as it +becomes available to the public. + +### Setting up for review and deployment + +Once you have your branch set up you should get in touch with a platform engineer to pair up on the remaining production needs. These include: + +- Creating the corresponding `deploy-*` branch in the repo. +- Adding the necessary Jenkins scripts for your indicator. +- Preparing the runtime host with any Automation configuration necessities. +- Reviewing the workflow to make sure it meets the general guidelines and will run as expected on the runtime host. + +Once all the last mile configuration is in place you can create a pull request against the correct `deploy-*` branch to initiate the CI/CD pipeline which will build, test, and package your indicator for deployment. + +If everything looks ok, you've drafted source documentation, platform engineering has validated the last mile, and the pull request is accepted, you can merge the PR. Deployment will start automatically. + +Hopefully it'll be a full on :tada:, after that :crossed_fingers: + +If not, circle back and try again. + +## Production overview + +### Running production code + +Currently, the production indicators all live and run on the venerable and perennially useful Delphi primary server (also known generically as "the runtime host"). + +### Delivering an indicator to the production environment + +We use a branch-based git workflow coupled with [Jenkins](https://www.jenkins.io/) and [Ansible](https://www.ansible.com/) to build, test, package, and deploy each indicator individually to the runtime host. + +- Jenkins dutifully manages the whole process for us by executing several "stages" in the context of a [CI/CD pipeline](https://dzone.com/articles/learn-how-to-setup-a-cicd-pipeline-from-scratch). Each stage does something unique, building on the previous stage. The stages are: + - Environment - Sets up some environment-specific needs that the other stages depend on. + - Build - Create the Python venv on the Jenkins host. + - Test - Run linting and unit tests. + - Package - Tar and gzip the built environment. + - Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary). + +There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host. It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release. diff --git a/README.md b/README.md index a71d7352a..ff3593a71 100644 --- a/README.md +++ b/README.md @@ -1,93 +1,23 @@ # Covidcast Indicators -Pipeline code and supporting libraries for the **Real-time COVID-19 Indicators** used in the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu). +In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals from which the course of the COVID-19 pandemic might be predicted using statistical models and forecasters. Delphi's long history in forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for convenience and to provide data versioning for sources that do not track revisions themselves. -## The indicators +Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract aggregate signals useful for forecasting, and output those signals in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). -Each subdirectory contained here that is named after an indicator has specific documentation. Please review as necessary! +A subset of the resulting **Real-time COVID-19 Indicators** have visualizations in the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu). -## General workflow for indicators creation and deployment +## Organization -**tl;dr** +Utilities: +* `_delphi_utils_python` - common behaviors +* `_template_python` & `_template_r` - starting points for new data sources +* `ansible` & `jenkins` - automated testing and deployment +* `sir_complainsalot` - a Slack bot to check for failed indicator runs -1. Create your new indicator branch from `main`. -2. Build it using the appropriate template, following the guidelines in the included README.md and REVIEW.md files. -3. Make some stuff! -4. When your stuff works, push your `dev-*` branch to remote for review. -5. Consult with a platform engineer for the remaining production setup needs. They will create a branch called `deploy-*` for your indicator. -6. Initiate a pull request against this new branch. -7. Following [the source documentation template](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md), create public API documentation for the source. You can submit this as a pull request against the delphi-epidata repository. -8. If your peers like the code, the documentation is ready, and Jenkins approves, deploy your changes by merging the PR. -9. Rejoice! +Indicator pipelines: all remaining directories. -### Starting out +Each indicator pipeline includes its own documentation. -The `main` branch should contain up-to-date code and supporting libraries. This should be your starting point when creating a new indicator. - -```shell -# Hint -# -git checkout main -git checkout -b dev-my-feature-branch -``` - -### Creating your indicator - -Create a directory for your new indicator by making a copy of `_template_r` or `_template_python` depending on the programming language you intend to use. The template copies of `README.md` and `REVIEW.md` include the minimum requirements for code structure, documentation, linting, testing, and method of configuration. Beyond that, we don't have any established restrictions on implementation; you can look at other existing indicators see some examples of code layout, organization, and general approach. - -- Consult your peers with questions! :handshake: - -Once you have something that runs locally and passes tests you set up your remote branch eventual review and production deployment. - -```shell -# Hint -# -git push -u origin dev-my-feature-branch -``` - -You can then set draft public API documentation for people who would fetch this -data from the API. Public API documentation is kept in the delphi-epidata -repository, and there is a [template Markdown -file](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md) -that outlines the features that need to be documented. You can create a pull -request to add a new file to `docs/api/covidcast-signals/` for your source. Our -goal is to have public API documentation for the data at the same time as it -becomes available to the public. - -### Setting up for review and deployment - -Once you have your branch set up you should get in touch with a platform engineer to pair up on the remaining production needs. These include: - -- Creating the corresponding `deploy-*` branch in the repo. -- Adding the necessary Jenkins scripts for your indicator. -- Preparing the runtime host with any Automation configuration necessities. -- Reviewing the workflow to make sure it meets the general guidelines and will run as expected on the runtime host. - -Once all the last mile configuration is in place you can create a pull request against the correct `deploy-*` branch to initiate the CI/CD pipeline which will build, test, and package your indicator for deployment. - -If everything looks ok, you've drafted source documentation, platform engineering has validated the last mile, and the pull request is accepted, you can merge the PR. Deployment will start automatically. - -Hopefully it'll be a full on :tada:, after that :crossed_fingers: - -If not, circle back and try again. - -## Production overview - -### Running production code - -Currently, the production indicators all live and run on the venerable and perennially useful Delphi primary server (also known generically as "the runtime host"). - -- This is a virtual machine running RHEL 7.5 and living in CMU's Campus Cloud vSphere-based infrastructure environemnt. - -### Delivering an indicator to the production environment - -We use a branch-based git workflow coupled with [Jenkins](https://www.jenkins.io/) and [Ansible](https://www.ansible.com/) to build, test, package, and deploy each indicator individually to the runtime host. - -- Jenkins dutifully manages the whole process for us by executing several "stages" in the context of a [CI/CD pipeline](https://dzone.com/articles/learn-how-to-setup-a-cicd-pipeline-from-scratch). Each stage does something unique, building on the previous stage. The stages are: - - Environment - Sets up some environment-specific needs that the other stages depend on. - - Build - Create the Python venv on the Jenkins host. - - Test - Run linting and unit tests. - - Package - Tar and gzip the built environment. - - Deploy - Trigger an Ansible playbook to place the built package onto the runtime host, place any necessary production configuration, and adjust the runtime envirnemnt (if necessary). - -There are several additional Jenkins-specific files that will need to be created for each indicator, as well as some configuration additions to the runtime host. It will be important to pair with a platform engineer to prepare the necessary production environment needs, test the workflow, validate on production, and ultimately sign off on a production release. +* Consult README.md for directions to install, lint, test, and run the pipeline for that indicator. +* Consult REVIEW.md for the checklist to use for code reviews. +* Consult DETAILS.md (if present) for implementation details, including handling of corner cases. From 549840e267d489ebf98c95987df0a073cb1d3f24 Mon Sep 17 00:00:00 2001 From: Kathryn M Mazaitis Date: Tue, 29 Sep 2020 17:17:18 -0400 Subject: [PATCH 3/8] Add sections on branches, issues, and project boards --- CONTRIBUTING.md | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0d6e84000..0eec61bf3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,40 @@ -# How to contribute to this repository +# Contributing to COVIDcast indicator pipelines + +## Branches + +* `main` + +The primary/authoritative branch of this repository is called `main`, and contains up-to-date code and supporting libraries. This should be your starting point when creating a new indicator. It is protected so that only reviewed pull requests can be merged in. + +* `deploy-*` + +Each automated pipeline has a corresponding branch which automatically deploys to a runtime host which runs the pipeline at a designated time each day. New features and bugfixes are merged into this branch using a pull request, so that our CI system can run the lint and test cycles and make sure the package will run correctly on the runtime host. If an indicator does not have a branch named after it starting with `deploy-`, that means the indicator has not yet been automated, and has a designated human keeper who is responsible for making sure the indicator runs each day -- whether that is manually or using a scheduler like cron is the keeper's choice. + +* everything else + +All other branches are development branches. We don't enforce a naming policy. + +## Issues + +Issues are the main communication point when it comes to bugfixes, new features, or other possible changes. The repository has several issue templates that help to structure issues. + +If you ensure that each issue deals with a single topic (ie a single new proposed data source, or a single data quality problem), we'll all be less likely to drop subordinate tasks on the floor, but we also recognize that a lot of the people filing issues in this repository are new to large project management and not used to focusing their thoughts in this way. It's okay, we'll all learn and get better together. + +Admins will assign issues to one or more people based on balancing expediency, expertise, and team robustness. It may be faster for one person to fix something, but we can reduce the risk of having too many single points of failure if two people work on it together. + +## Project Boards + +The Delphi Engineering team uses project boards to structure its weekly calls and track active tasks. + +Immediate work is tracked on [Release Planning](https://github.com/cmu-delphi/covidcast-indicators/projects/2) + +Long-term work and modeling collaborations are tracked on [Refactoring](https://github.com/cmu-delphi/covidcast-indicators/projects/3) + ## General workflow for indicators creation and deployment +So, how does one go about developing a pipeline for a new data source? + **tl;dr** 1. Create your new indicator branch from `main`. @@ -40,7 +73,7 @@ Once you have something that runs locally and passes tests you set up your remot git push -u origin dev-my-feature-branch ``` -You can then set draft public API documentation for people who would fetch this +You can then draft public API documentation for people who would fetch this data from the API. Public API documentation is kept in the delphi-epidata repository, and there is a [template Markdown file](https://github.com/cmu-delphi/delphi-epidata/blob/main/docs/api/covidcast-signals/_source-template.md) From 58806adf34a7aabdf67147855d0959a419780b30 Mon Sep 17 00:00:00 2001 From: Kathryn M Mazaitis Date: Tue, 29 Sep 2020 17:17:29 -0400 Subject: [PATCH 4/8] Add license information --- README.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ff3593a71..03a19d7a7 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # Covidcast Indicators +[![License: MIT][mit-image]][mit-url] + In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals from which the course of the COVID-19 pandemic might be predicted using statistical models and forecasters. Delphi's long history in forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for convenience and to provide data versioning for sources that do not track revisions themselves. Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract aggregate signals useful for forecasting, and output those signals in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). @@ -12,7 +14,7 @@ Utilities: * `_delphi_utils_python` - common behaviors * `_template_python` & `_template_r` - starting points for new data sources * `ansible` & `jenkins` - automated testing and deployment -* `sir_complainsalot` - a Slack bot to check for failed indicator runs +* `sir_complainsalot` - a Slack bot to check for missing data Indicator pipelines: all remaining directories. @@ -21,3 +23,11 @@ Each indicator pipeline includes its own documentation. * Consult README.md for directions to install, lint, test, and run the pipeline for that indicator. * Consult REVIEW.md for the checklist to use for code reviews. * Consult DETAILS.md (if present) for implementation details, including handling of corner cases. + + +## License + +This repository is released under the **MIT License**. + +[mit-image]: https://img.shields.io/badge/License-MIT-yellow.svg +[mit-url]: https://opensource.org/licenses/MIT From 669d01b2063f5ac9fe7105541cd44219b3d5aa76 Mon Sep 17 00:00:00 2001 From: Kathryn M Mazaitis Date: Tue, 29 Sep 2020 17:17:59 -0400 Subject: [PATCH 5/8] Moved contribution guide to where github expects it --- CONTRIBUTING.md => .github/CONTRIBUTING.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename CONTRIBUTING.md => .github/CONTRIBUTING.md (100%) diff --git a/CONTRIBUTING.md b/.github/CONTRIBUTING.md similarity index 100% rename from CONTRIBUTING.md rename to .github/CONTRIBUTING.md From 98b1992c8b7f75701d6c38e8781b5d6a4cb1610c Mon Sep 17 00:00:00 2001 From: krivard Date: Wed, 30 Sep 2020 09:18:51 -0400 Subject: [PATCH 6/8] Update LICENSE with correct copyright date Co-authored-by: Alex Reinhart --- LICENSE | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/LICENSE b/LICENSE index 92a5e44d9..fbfb2f9bd 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2018 The Delphi Group at Carnegie Mellon University +Copyright (c) 2020 The Delphi Group at Carnegie Mellon University Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal From e42e56689167de9668fe2175a3e1a64a04df14c9 Mon Sep 17 00:00:00 2001 From: RoniRos Date: Wed, 30 Sep 2020 13:46:52 -0400 Subject: [PATCH 7/8] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 03a19d7a7..a7e02bedf 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ [![License: MIT][mit-image]][mit-url] -In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals from which the course of the COVID-19 pandemic might be predicted using statistical models and forecasters. Delphi's long history in forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for convenience and to provide data versioning for sources that do not track revisions themselves. +In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals that would track in real-time and in fine geographic granularity all facets of the COVID-19 pandemic, and from which the future course of the pandemic might be predicted using statistical models and forecasters. Delphi's long history in tracking and forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for user convenience and to provide data versioning for sources that do not track revisions themselves. -Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract aggregate signals useful for forecasting, and output those signals in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). +Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract informative aggregate signals, and output those signals in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). -A subset of the resulting **Real-time COVID-19 Indicators** have visualizations in the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu). +A subset of the resulting **Real-time COVID-19 Indicators** are visualized on the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu). ## Organization From e3e80affdbaeeb0cf53c94efb2378c0ed40ea61f Mon Sep 17 00:00:00 2001 From: ryantibs Date: Wed, 30 Sep 2020 16:32:11 -0400 Subject: [PATCH 8/8] Small edits to README --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a7e02bedf..1e2997dee 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,13 @@ [![License: MIT][mit-image]][mit-url] -In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals that would track in real-time and in fine geographic granularity all facets of the COVID-19 pandemic, and from which the future course of the pandemic might be predicted using statistical models and forecasters. Delphi's long history in tracking and forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for user convenience and to provide data versioning for sources that do not track revisions themselves. +In early April 2020, Delphi developed a uniform data schema for [a new Epidata endpoint focused on COVID-19](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). Our intent was to provide signals that would track in real-time and in fine geographic granularity all facets of the COVID-19 pandemic, aiding both nowcasting and forecasting. Delphi's long history in tracking and forecasting influenza made us uniquely situated to provide access to data streams not available anywhere else, including medical claims data, electronic medical records, lab test records, massive public surveys, and internet search trends. We also process commonly-used publicly-available data sources, both for user convenience and to provide data versioning for sources that do not track revisions themselves. -Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract informative aggregate signals, and output those signals in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). +Each data stream arrives in a different format using a different delivery technique, be it sftp, an access-controlled API, or an email attachment. The purpose of each pipeline in this repository is to fetch the raw source data, extract informative aggregate signals, and output those signals---which we call **COVID-19 indicators**---in a common format for upload to the [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html). -A subset of the resulting **Real-time COVID-19 Indicators** are visualized on the Delphi Group's [**COVIDcast** map](https://covidcast.cmu.edu). +For client access to the API, along with a variety of other utilities, see our [R](https://cmu-delphi.github.io/covidcast/covidcastR/) and [Python](https://cmu-delphi.github.io/covidcast/covidcast-py/html/) packages. + +For interactive visualizations (of a subset of the available indicators), see our [COVIDcast map](https://covidcast.cmu.edu). ## Organization