ght `raw` is the same as `smoothed` since august

[dfarrow0]

Actual Behavior

The raw signal is equal to the smoothed signal, starting in August.

Expected behavior

The raw signal is generally different than the smoothed signal.

Context

https://github.com/cmu-delphi/covidcast-indicators/tree/main/google_health

[dfarrow0]

By default, only data for a single day is pulled from the api:

covidcast-indicators/google_health/delphi_google_health/run.py

Lines 53 to 61 in 8a0a2c0

	# if missing start_date, set to today (GMT) minus 5 days
	if start_date == "":
	now = datetime.datetime.now(datetime.timezone.utc)
	start_date = (now - datetime.timedelta(days=4)).strftime("%Y-%m-%d")
	# if missing end_date, set to today (GMT) minus 5 days
	if end_date == "":
	now = datetime.datetime.now(datetime.timezone.utc)
	end_date = (now - datetime.timedelta(days=4)).strftime("%Y-%m-%d")

And a 1-length timeseries results in a singular matrix, causing smoothing to fall back to the raw value:

covidcast-indicators/google_health/delphi_google_health/smooth.py

Lines 72 to 74 in 8a0a2c0

	except np.linalg.LinAlgError:
	# At idx 0, method will fail due to rank deficiency.
	t[idx] = s[idx] if impute else np.nan

[dfarrow0]

Some unorganized thoughts:

• my general advice is to avoid precomputing and storing a signal that's deterministically derived from another signal. recommendation is to serve the underlying signal and let callers do the computation on the fly. not saying that's always the best choice, as there are certainly tradeoffs to consider. but my preference is to avoid precomputing unless there's a compelling reason otherwise.
• due to the bug in this issue, the smoothed signal is equal to the raw signal for many, many rows. once this all gets sorted out, we'll need to go back and recompute the smoothed signal so that the fix is applied retroactively to affected rows, and not just to new rows.
• an attribute of the smoother, which is a feature from a theoretical perspective but a drawback from a practical perspective, is that it has infinite support. meaning that the smoothing isn't done over a fixed window, but over the entire history of the signal. using the full history is problematic because it requires fetching the whole time-series (either from GHT or from API or from cache), which adds cost and complexity. it would be simpler to just fetch, say, the last 10 days, and then smooth over a sliding window. unfortunately, that would lead to ever-so-slightly different values of smoothed for each of the 10 days that it's computed. which dirties the CSV cache on those days, and causes us to store not 2x the data, but 10x the data. i.e. we would store the original raw signal, the smoothed signal that's as_of a day with 10 days of data, the smoothed signal that's as_of a day with 9 days of data, and so on.

do we have any idea of whether the smoothed signal is actively used? if it's not, i recommend just removing the smoothed signal from the api altogether.

[jingjtang]

The cache file used for automation is surprisingly differently from the ones generated locally. Coordinate with @korlaxxalrok to make sure whether this is a problem with cache files. Currently, if run locally, everything seems fine to me. The smoothed_search is different from raw_search, test_smooth.py passed.

Other problems noticed with unit tests:

• Not a dry-run mode, ght-key is needed for test_run.py
• mocked or faked aws_credentials and bucket should be set up for ./tests/params.json which is not there currently.
• test_match_old_raw_output and test_match_old_smoothed_output failed.

============================================ short test summary info ============================================
FAILED test_run.py::TestRunModule::test_match_old_raw_output - AssertionError: DataFrame.iloc[:, 1] (column na...
FAILED test_run.py::TestRunModule::test_match_old_smoothed_output - AssertionError: DataFrame.iloc[:, 1] (colu...

[krivard]

GHT hitting the network during tests is a known problem; it may be easier to fix that first and then address the cache problem?

[jingjtang]

@krivard A simple fix here. We can fix that problem with unit tests in another PR.

[dshemetov]

@dfarrow0 FWIW, I changed the support to a finite value (about a month) in the refactored smoother utility. Speed-wise the smoothing could move to the client side (smoothing a 1000-length signal takes ~3.5 ms), but I don't have a good sense of other involved dependencies.

EDIT: A couple other thoughts: smoothing in the client would have to add logic for grabbing extra data do the smoothing (e.g. an extra week in the past). There are also some indicators, like Quidel, which rely on geographical pooling for smoothing when sample thresholds are too low; the API may have to fetch extra data or may need to do those more involved computations locally (probably not slow, just more complex logic).