streaming support

[sgratzl]

closes #51

streams results back to the user instead of caching it locally and then send it in a bulk.

It is a bigger change since I introduced the notion of a printer which is responsible for printing in different formats. There is no change for clients.

Following formats are supported:

• classic = classic epidata format
• tree = the epidata format which the first element grouped by signal ... no streaming
• json = flat json array
• csv = csv file
• jsonl = jsonlines format (https://jsonlines.org/) which is a nice format for streaming

don't know why

delphi.delphi-epidata.integrations.acquisition.covidcast.test_csv_uploading.CsvUploadingTests.test_uploading: fail

is failing. I didn't touch those files

[dfarrow0]

looks like this other test fails too:

FAIL: test_caching (repos.delphi.delphi-epidata.integrations.acquisition.covidcast.test_covidcast_meta_caching.CovidcastMetaCacheTests)
Populate, query, cache, query, and verify the cache.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/app/repos/delphi/delphi-epidata/integrations/acquisition/covidcast/test_covidcast_meta_caching.py", line 112, in test_caching
    self.assertEqual(epidata2['result'], -2, json.dumps(epidata2))
AssertionError: 0 != -2 : {"result": 0, "message": "error: Expecting value: line 1 column 1 (char 0)"}

---

there's a lot going on in the pr, so i'm going to try to run the integration tests under php5 also. will update when done.

@krivard i'd love to get to php7 in prod. delphi-epidata, www-epidata, and github-deploy-repo are all ready. is automation the only thing holding this back?

[dfarrow0]

php5.4 image, streaming branch:

 delphi.delphi-epidata.integrations.acquisition.covid_hosp.state_timeseries.test_scenarios.AcquisitionTests.test_acquire_dataset: fail
 delphi.delphi-epidata.integrations.acquisition.covidcast.test_covidcast_meta_caching.CovidcastMetaCacheTests.test_caching: fail
 delphi.delphi-epidata.integrations.acquisition.covidcast.test_csv_uploading.CsvUploadingTests.test_uploading: fail
 delphi.delphi-epidata.integrations.client.test_delphi_epidata.DelphiEpidataPythonClientTests.test_covidcast: fail
 delphi.delphi-epidata.integrations.client.test_delphi_epidata.DelphiEpidataPythonClientTests.test_covidcast_meta: fail
 delphi.delphi-epidata.integrations.client.test_delphi_epidata.DelphiEpidataPythonClientTests.test_geo_value: fail
 delphi.delphi-epidata.integrations.server.test_api_analytics.ApiAnalyticsTests.test_analytics_update: fail
 delphi.delphi-epidata.integrations.server.test_covid_hosp.ServerTests.test_query_by_issue: fail
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_csv_format: fail
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_date_formats: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_fields: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_geo_value: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_location_timeline: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_location_wildcard: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_nullable_columns: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_raw_json_format: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_round_trip: error
 delphi.delphi-epidata.integrations.server.test_covidcast.CovidcastTests.test_temporal_partitioning: error
 delphi.delphi-epidata.integrations.server.test_covidcast_meta.CovidcastMetaTests.test_filter: error
 delphi.delphi-epidata.integrations.server.test_covidcast_meta.CovidcastMetaTests.test_round_trip: error
 delphi.delphi-epidata.integrations.server.test_covidcast_meta.CovidcastMetaTests.test_suppress_work_in_progress: error
 delphi.delphi-epidata.integrations.server.test_fluview.FluviewTests.test_round_trip: fail
 delphi.delphi-epidata.integrations.server.test_fluview_meta.FluviewMetaTests.test_round_trip: fail

php5.4 image, main branch from cmu-delphi:

✔ All 31 tests passed! 43% (189/435) coverage.

@sgratzl did you use any php features that aren't available in php5?

[sgratzl]

I didn't know that the server is just using php5.4. I was testing with the docker image from https://github.com/cmu-delphi/operations/blob/main/dev/docker/web/Dockerfile which is PHP 7. So, most likely with all the types and classes.

[krivard]

get to php7 in prod

agreed, will start an asana task until we figure out what work is actually required

[dfarrow0]

yeah, not your fault for using php7 -- we were planning to go to php7 many months ago and it fizzled with everything else happening. we should discuss cost+risk+timeline for switching over to php7 (and also cost+risk of staying on php5). @krivard @korlaxxalrok

---

@sgratzl on the php7 image, the failing test is FAIL: test_caching (repos.delphi.delphi-epidata.integrations.acquisition.covidcast.test_covidcast_meta_caching.CovidcastMetaCacheTests)

looks like there's a bare return when the cache is empty. no response is emitted, and the client is unable to parse. in this case, the desired behavior is result -2, message something about no data, and empty or absent epidata.

[dfarrow0]

all unit/integration tests are passing for me now. next doing some manual testing.

curl -I 'http://127.0.0.1:10080/epidata/api.php'

HTTP/1.0 500 Internal Server Error
Date: Wed, 16 Dec 2020 16:48:12 GMT
Server: Apache/2.4.38 (Debian)
X-Powered-By: PHP/7.4.13
Connection: close
Content-Type: text/html; charset=UTF-8

expected 200 and message about missing source:
curl 'https://delphi.cmu.edu/epidata/api.php'

{"result":-1,"message":"no data source specified"}

---

i tried curling the various formats with source=fluview, and the responses looked reasonable to me.

[sgratzl]

curl -I 'http://127.0.0.1:10080/epidata/api.php'

should be fixed

[sgratzl]

I didn't know that the server is just using php5.4. I was testing with the docker image from https://github.com/cmu-delphi/operations/blob/main/dev/docker/web/Dockerfile which is PHP 7. So, most likely with all the types and classes.

So, I guess the system has been upgraded to php 7?

[krivard]

Mmmmm looks like I misinterpreted the thread to mean that had been fixed. I know Brian started looking at that yesterday, let me see if we can expedite that before this deploys to production. It's currently behind at least four indicator updates so we have some time.

[krivard]

Brian recommends we roll back. Apologies for the fire drill!