Merge branch 'dev' into Tina_doc_page_reorganization

Author: tinatownes

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 4.1.23
+current_version = 4.1.24
 commit = False
 tag = False
 

.git-blame-ignore-revs

@@ -20,3 +20,5 @@ b9ceb400d9248c8271e8342275664ac5524e335d
 07ed83e5768f717ab0f9a62a9209e4e2cffa058d
 # style(black): format wiki acquisition
 923852eafa86b8f8b182d499489249ba8f815843
+# lint: trailing whitespace changes
+81179c5f144b8f25421e799e823e18cde43c84f9

dev/local/setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = Delphi Development
-version = 4.1.23
+version = 4.1.24
 
 [options]
 packages =

docs/api/covidcast-signals/covid-act-now.md

@@ -15,11 +15,13 @@ grand_parent: COVIDcast Main Endpoint
 * **Time type:** day (see [date format docs](../covidcast_times.md))
 * **License:** [CC BY-NC](../covidcast_licensing.md#creative-commons-attribution-noncommercial)
 
-The COVID Act Now (CAN) data source provides COVID-19 testing statistics, such as positivity rates and total tests performed.
-The county-level positivity rates and test totals are pulled directly from CAN.
-While CAN provides this data potentially from multiple sources, we only use data sourced from the
+The [COVID Act Now (CAN)](https://covidactnow.org/) data source provides COVID-19 testing statistics, such as positivity rates and total tests performed.
+The county-level positivity rates and test totals are pulled directly from CAN using [their API](https://covidactnow.org/data-api).
+While CAN provides this data potentially from multiple sources, we only use data that CAN sources from the
 [CDC's COVID-19 Integrated County View](https://covid.cdc.gov/covid-data-tracker/#county-view).
 
+Delphi's mirror of the CAN data was deactivated in December 2021 (last issue 2021-12-10) in favor of the [DSEW CPR data](./dsew-cpr.md), which reports the same information under the `covid_naat_pct_positive_7dav` signal.
+
 
 | Signal                         | Description                                                    |
 |--------------------------------|----------------------------------------------------------------|
@@ -34,9 +36,9 @@ While CAN provides this data potentially from multiple sources, we only use data
 
 ## Estimation
 
-The quantities received from CAN / CDC are the county-level positivity rate and total tests, 
-which are based on the counts of PCR specimens tested.
-In particular, they are also already smoothed with a 7-day-average.
+We receive county-level positivity rate and total tests from CAN, originating from the CDC.
+These quantiles are based on the counts of PCR specimens tested.
+They are also already smoothed with a 7-day-average.
 
 For a fixed location $$i$$ and time $$t$$, let $$Y_{it}$$ denote the number of PCR specimens 
 tested that have a positive result. Let $$N_{it}$$ denote the total number of PCR specimens tested.
@@ -79,38 +81,41 @@ $$
 
 ### Smoothing
 
-No additional smoothing is done to avoid double-smoothing, since the data pulled from CAN / CDC 
+No additional smoothing is done to avoid double-smoothing, since the CAN data
 is already smoothed with a 7-day-average.
 
 ## Limitations
 
-Estimates for geographical levels beyond counties may be inaccurate due to how aggregations 
-are done on smoothed values instead of the raw values. Ideally we would aggregate raw values 
+Estimates for geographical levels beyond counties may be inaccurate because our aggregations
+are performed on smoothed values instead of the raw values.
+Ideally we would aggregate raw values
 then smooth, but the raw values are not accessible in this case.
 
-The positivity rate here should not be interpreted as the population positivity rate as 
+The reported test positivity rate should not be interpreted as the population positivity rate as
 the testing performed are typically not randomly sampled, especially for early data 
 with lower testing volumes.
 
 A few counties, most notably in California, are also not covered by this data source.
 
-Entries with zero total tests performed are also suppressed, even if it was actually the case that 
+Entries with zero total tests performed are suppressed, even if it was actually the case that
 no tests were performed for the day.
 
 ## Lag and Backfill
 
 The lag for these signals varies depending on the reporting patterns of individual counties.
 Most counties have their latest data report with a lag of 2 days, while others can take 9 days 
-or more in the case of California counties.
+or more, as is the case with California counties.
 
-These signals are also backfilled as backlogged test results could get assigned to older 7-day timeframes.
-Most recent test positivity rates do not change substantially with backfill (having a median delta of close to 0).
-However, most recent total tests performed is expected to increase in later data revisions (having a median increase of 7%).
+Revisions are sometimes made to the data. For example, backlogged test results can get assigned to past dates.
+The majority of recent test positivity rates do not change substantially with backfill (having a median delta of close to 0).
+However, the majority of recent total tests performed is expected to increase in later data revisions (having a median increase of 7%).
 Values more than 5 days in the past are expected to remain fairly static (with total tests performed 
 having a median increase of 1% of less), as most major revisions have already occurred.
 
 ## Source and Licensing
 
-County-level testing data is scraped by CAN from the 
+County-level testing data is scraped by [CAN](https://covidactnow.org/) from the
 [CDC's COVID-19 Integrated County View](https://covid.cdc.gov/covid-data-tracker/#county-view),
 and made available through [CAN's API](https://covidactnow.org/tools).
+
+The data is made available under a [CC BY-NC](../covidcast_licensing.md#creative-commons-attribution-noncommercial) license.

docs/api/covidcast-signals/hhs.md

@@ -1,6 +1,6 @@
 ---
-title: Department of Health & Human Services (inactive)
-parent: Data Sources and Signals
+title: Department of Health & Human Services
+parent: Inactive Signals
 grand_parent: COVIDcast Main Endpoint
 ---
 

docs/api/covidcast-signals/indicator-combination renamed to docs/api/covidcast-signals/indicator-combination.md

@@ -50,7 +50,7 @@ $ [sudo] make test pdb=1
 $ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition
 ```
 
-You can read the commands executed by the Makefile [here](../dev/local/Makefile).
+You can read the commands executed by the Makefile [here](https://github.com/cmu-delphi/delphi-epidata/blob/dev/dev/local/Makefile).
 
 ## Rapid Iteration and Bind Mounts
 
@@ -88,8 +88,8 @@ You can test your changes manually by:
 
 What follows is a worked demonstration based on the `fluview` endpoint. Before
 starting, make sure that you have the `delphi_database_epidata`,
-`delphi_web_epidata`, and `delphi_redis` containers running; if you don't, see
-the Makefile instructions above.
+`delphi_web_epidata`, and `delphi_redis` containers running (with `docker ps`);
+if you don't, see the Makefile instructions above.
 
 First, let's insert some fake data into the `fluview` table:
 

docs/epidata_development.md

@@ -4,17 +4,16 @@ has_children: true
 nav_order: 1
 ---
 
-# Delphi's Epidata API
+# The Epidata API
 
 Delphi's Epidata API provides real-time access to epidemiological surveillance data.
 It is built and maintained by the Carnegie Mellon University [Delphi research
 group](https://delphi.cmu.edu/). The Epidata API includes:
 
-- [COVIDcast data](api/covidcast.md), providing daily updates about COVID-19
-  activity across the United States. [API clients](api/covidcast_clients.md) for
-  quick access to COVID data are available.
-- [Data about other diseases](api/README.md), including influenza, dengue, and
-  other diseases tracked by Delphi through various data streams.
+* The [main endpoint (COVIDcast)](api/covidcast.md), providing daily updates about current COVID-19 and influenza activity across the United States.
+* A [variety of other endpoints](api/README.md), providing primarily historical data about various diseases including COVID-19, influenza, dengue fever, and norovirus in several countries.
+
+A [full-featured R client](api/client_libraries.md) is available for quick access to all data. While we continue developing a full-featured Python client, the [legacy Python client](api/client_libraries.md#python) remains available. The main endpoint can also be accessed with a [dedicated COVIDcast client](api/covidcast_clients.md).
 
 Anyone may access the Epidata API anonymously without providing any personal
 data. Anonymous API access is currently rate-limited and restricted to public

docs/index.md

@@ -2,8 +2,8 @@
 
 # standard library
 import time
+import json
 from json import JSONDecodeError
-from requests.models import Response
 from unittest.mock import MagicMock, patch
 
 # first party
@@ -49,6 +49,12 @@ def localSetUp(self):
     secrets.db.host = 'delphi_database_epidata'
     secrets.db.epi = ('user', 'pass')
 
+  @pytest.fixture(autouse=True)
+  def capsys(self, capsys):
+    """Hook capsys (stdout and stderr) into this test class."""
+
+    self.capsys = capsys
+
   def test_covidcast(self):
     """Test that the covidcast endpoint returns expected data."""
 
@@ -238,46 +244,46 @@ def raise_for_status(self): pass
 
     try:
       with self.subTest(name='test multiple GET'):
-        with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
-          get.reset_mock()
-          get.return_value = MockResponse(b'{"key": "value"}', 200)
-          Epidata._request_with_retry("test_endpoint1", params={"key1": "value1"})
-          Epidata._request_with_retry("test_endpoint2", params={"key2": "value2"})
+        get.reset_mock()
+        get.return_value = MockResponse(b'{"key": "value"}', 200)
+        Epidata._request_with_retry("test_endpoint1", params={"key1": "value1"})
+        Epidata._request_with_retry("test_endpoint2", params={"key2": "value2"})
 
-        output = logs.output
+        captured = self.capsys.readouterr()
+        output = captured.err.splitlines()
         self.assertEqual(len(output), 4) # [request, response, request, response]
         self.assertIn("Sending GET request", output[0])
-        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint1/\"", output[0])
-        self.assertIn("\"params\": {\"key1\": \"value1\"}", output[0])
+        self.assertIn("\'url\': \'http://delphi_web_epidata/epidata/test_endpoint1/\'", output[0])
+        self.assertIn("\'params\': {\'key1\': \'value1\'}", output[0])
         self.assertIn("Received response", output[1])
-        self.assertIn("\"status_code\": 200", output[1])
-        self.assertIn("\"len\": 16", output[1])
+        self.assertIn("\'status_code\': 200", output[1])
+        self.assertIn("\'len\': 16", output[1])
         self.assertIn("Sending GET request", output[2])
-        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint2/\"", output[2])
-        self.assertIn("\"params\": {\"key2\": \"value2\"}", output[2])
+        self.assertIn("\'url\': \'http://delphi_web_epidata/epidata/test_endpoint2/\'", output[2])
+        self.assertIn("\'params\': {\'key2\': \'value2\'}", output[2])
         self.assertIn("Received response", output[3])
-        self.assertIn("\"status_code\": 200", output[3])
-        self.assertIn("\"len\": 16", output[3])
+        self.assertIn("\'status_code\': 200", output[3])
+        self.assertIn("\'len\': 16", output[3])
 
       with self.subTest(name='test GET and POST'):
-        with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
-          get.reset_mock()
-          get.return_value = MockResponse(b'{"key": "value"}', 414)
-          post.reset_mock()
-          post.return_value = MockResponse(b'{"key": "value"}', 200)
-          Epidata._request_with_retry("test_endpoint3", params={"key3": "value3"})
-
-        output = logs.output
-        self.assertEqual(len(output), 3) # [request, response, request, response]
+        get.reset_mock()
+        get.return_value = MockResponse(b'{"key": "value"}', 414)
+        post.reset_mock()
+        post.return_value = MockResponse(b'{"key": "value"}', 200)
+        Epidata._request_with_retry("test_endpoint3", params={"key3": "value3"})
+
+        captured = self.capsys.readouterr()
+        output = captured.err.splitlines()
+        self.assertEqual(len(output), 3) # [request, retry, response]
         self.assertIn("Sending GET request", output[0])
-        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[0])
-        self.assertIn("\"params\": {\"key3\": \"value3\"}", output[0])
+        self.assertIn("\'url\': \'http://delphi_web_epidata/epidata/test_endpoint3/\'", output[0])
+        self.assertIn("\'params\': {\'key3\': \'value3\'}", output[0])
         self.assertIn("Received 414 response, retrying as POST request", output[1])
-        self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[1])
-        self.assertIn("\"params\": {\"key3\": \"value3\"}", output[1])
+        self.assertIn("\'url\': \'http://delphi_web_epidata/epidata/test_endpoint3/\'", output[1])
+        self.assertIn("\'params\': {\'key3\': \'value3\'}", output[1])
         self.assertIn("Received response", output[2])
-        self.assertIn("\"status_code\": 200", output[2])
-        self.assertIn("\"len\": 16", output[2])
+        self.assertIn("\'status_code\': 200", output[2])
+        self.assertIn("\'len\': 16", output[2])
     finally: # make sure this global is always reset
       Epidata.debug = False
 
@@ -288,18 +294,42 @@ def test_sandbox(self, get, post):
     Epidata.debug = True
     Epidata.sandbox = True
     try:
-      with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
-        Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234')
-      output = logs.output
+      Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234')
+      captured = self.capsys.readouterr()
+      output = captured.err.splitlines()
       self.assertEqual(len(output), 1)
       self.assertIn("Sending GET request", output[0])
-      self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/covidcast/\"", output[0])
+      self.assertIn("\'url\': \'http://delphi_web_epidata/epidata/covidcast/\'", output[0])
       get.assert_not_called()
       post.assert_not_called()
     finally: # make sure these globals are always reset
       Epidata.debug = False
       Epidata.sandbox = False
 
+  @patch('requests.get')
+  def test_version_check(self, get):
+    """Test that the _version_check() function correctly logs a version discrepancy."""
+    class MockJson:
+      def __init__(self, content, status_code):
+          self.content = content
+          self.status_code = status_code
+      def raise_for_status(self): pass
+      def json(self): return json.loads(self.content)
+    get.reset_mock()
+    get.return_value = MockJson(b'{"info": {"version": "0.0.1"}}', 200)
+    Epidata._version_check()
+    captured = self.capsys.readouterr()
+    output = captured.err.splitlines()
+    self.assertEqual(len(output), 1)
+    self.assertIn("Client version not up to date", output[0])
+    self.assertIn("\'latest_version\': \'0.0.1\'", output[0])
+
+  @patch('delphi.epidata.client.delphi_epidata.Epidata._version_check')
+  def test_version_check_once(self, version_check):
+    """Test that the _version_check() function is only called once on initial module import."""
+    from delphi.epidata.client.delphi_epidata import Epidata
+    version_check.assert_not_called()
+
   def test_geo_value(self):
     """test different variants of geo types: single, *, multi."""
 

integrations/client/test_delphi_epidata.py

@@ -11,7 +11,7 @@
 
 # first party
 import delphi.operations.secrets as secrets
-from delphi.epidata.common.logger import get_structured_logger
+from delphi_utils import get_structured_logger
 
 Columndef = namedtuple("Columndef", "csv_name sql_name dtype")
 

src/acquisition/covid_hosp/common/database.py

@@ -13,10 +13,9 @@
 import pandas as pd
 
 # first party
-from delphi_utils import Nans
+from delphi_utils import get_structured_logger, Nans
 from delphi.utils.epiweek import delta_epiweeks
 from delphi.epidata.common.covidcast_row import CovidcastRow
-from delphi.epidata.common.logger import get_structured_logger
 
 DataFrameRow = NamedTuple('DFRow', [
   ('geo_id', str),

src/acquisition/covidcast/csv_importer.py

@@ -11,7 +11,7 @@
 from delphi.epidata.acquisition.covidcast.csv_importer import CsvImporter, PathDetails
 from delphi.epidata.acquisition.covidcast.database import Database, DBLoadStateException
 from delphi.epidata.acquisition.covidcast.file_archiver import FileArchiver
-from delphi.epidata.common.logger import get_structured_logger
+from delphi_utils import get_structured_logger
 
 
 def get_argument_parser():

src/acquisition/covidcast/csv_to_database.py

@@ -14,7 +14,7 @@
 
 # first party
 import delphi.operations.secrets as secrets
-from delphi.epidata.common.logger import get_structured_logger
+from delphi_utils import get_structured_logger
 from delphi.epidata.common.covidcast_row import CovidcastRow
 
 
@@ -117,28 +117,28 @@ def insert_or_update_batch(self, cc_rows: List[CovidcastRow], batch_size=2**20,
       get_structured_logger("insert_or_update_batch").fatal(err_msg)
       raise DBLoadStateException(err_msg)
 
-    # NOTE: `value_update_timestamp` is hardcoded to "NOW" (which is appropriate) and 
+    # NOTE: `value_update_timestamp` is hardcoded to "NOW" (which is appropriate) and
     #       `is_latest_issue` is hardcoded to 1 (which is temporary and addressed later in this method)
     insert_into_loader_sql = f'''
       INSERT INTO `{self.load_table}`
         (`source`, `signal`, `time_type`, `geo_type`, `time_value`, `geo_value`,
-        `value_updated_timestamp`, `value`, `stderr`, `sample_size`, `issue`, `lag`, 
+        `value_updated_timestamp`, `value`, `stderr`, `sample_size`, `issue`, `lag`,
         `is_latest_issue`, `missing_value`, `missing_stderr`, `missing_sample_size`)
       VALUES
-        (%s, %s, %s, %s, %s, %s, 
-        UNIX_TIMESTAMP(NOW()), %s, %s, %s, %s, %s, 
+        (%s, %s, %s, %s, %s, %s,
+        UNIX_TIMESTAMP(NOW()), %s, %s, %s, %s, %s,
         1, %s, %s, %s)
     '''
 
     # all load table entries are already marked "is_latest_issue".
     # if an entry in the load table is NOT in the latest table, it is clearly now the latest value for that key (so we do nothing (thanks to INNER join)).
     # if an entry *IS* in both load and latest tables, but latest table issue is newer, unmark is_latest_issue in load.
     fix_is_latest_issue_sql = f'''
-        UPDATE 
-            `{self.load_table}` JOIN `{self.latest_view}` 
-                USING (`source`, `signal`, `geo_type`, `geo_value`, `time_type`, `time_value`) 
-            SET `{self.load_table}`.`is_latest_issue`=0 
-            WHERE `{self.load_table}`.`issue` < `{self.latest_view}`.`issue` 
+        UPDATE
+            `{self.load_table}` JOIN `{self.latest_view}`
+                USING (`source`, `signal`, `geo_type`, `geo_value`, `time_type`, `time_value`)
+            SET `{self.load_table}`.`is_latest_issue`=0
+            WHERE `{self.load_table}`.`issue` < `{self.latest_view}`.`issue`
     '''
 
     # TODO: consider handling cc_rows as a generator instead of a list

src/acquisition/covidcast/database.py

@@ -6,7 +6,7 @@
 import shutil
 
 # first party
-from delphi.epidata.common.logger import get_structured_logger
+from delphi_utils import get_structured_logger
 
 class FileArchiver:
   """Archives files by moving and compressing."""

src/acquisition/covidcast/file_archiver.py

@@ -15,7 +15,7 @@ Epidata <- (function() {
   # API base url
   BASE_URL <- getOption('epidata.url', default = 'https://api.delphi.cmu.edu/epidata/')
 
-  client_version <- '4.1.23'
+  client_version <- '4.1.24'
 
   auth <- getOption("epidata.auth", default = NA)