testing forecasters on simple datasets

[dsweber2]

addresses #23 and #20. I ended up not actually using targets for the simple data tests b/c it ended up too awkward and distributed. I should probably do a little cleanup of the repo and check for edge cases before we actually merge this.

[dsweber2]

I also added the flatline forecaster and made the necessary changes to get that working. Should be ready for review (pending the tests passing on the remote).

[dsweber2]

To be clear, the tests this adds are:

• 2 states with different constants (technically normally distributed with sd=1e-5, to avoid problems with linear_reg)
• single state with white noise (sd=2 while the mean is 25)
• 2 states, one of which has latency varying between 0-3 days (poisson mean 0.5)
• linearly increasing by 1 each day (the flatline is exempt from passing this test)

[nmdefries]

Comments are mostly for formatting/structure, still looking at this for content.

There are a lot of objs being printed out, I assume for debugging. Test code should be removed. I marked some of them, probably not all.

Generally, it'd be good to have more descriptive "test_that" names and/or comments about what the goal of a particular test is.

[nmdefries]

Confused about the reasoning behind a couple of the checks.

Other tests that might be useful to include:

• Which dates predicted values are available for/making sure that preds are on days where we have enough data
• Right now, looks like predictions are only for dates that are included in the input dataset. Might be useful to check behavior when making true forecasts (again, that output dates are what we expect, etc)

[nmdefries]

question: Why are we normalizing by days ahead?

[dsweber2]

iirc the variance should be increasing linearly with the number of days into the future? It's probably a complication we don't need to include, since ahead=1 for all of these tests.

[nmdefries]

the variance should be increasing linearly with the number of days into the future

Oh, I see what you mean. A prediction for ahead_i will have error within synth_sd around the prediction for ahead_i - 1, which also has error. So the error accumulates and is ~bounded by synth_sd * days ahead for each ahead. Sound approximately correct? 😄

[nmdefries]

maybe leave this in case we want add more aheads later, but add a comment about the normalization? But no strong feelings on my part.

[dsweber2]

output date correctness is checked here in the "basics" section, rather than the "data" section:

exploration-tooling/tests/testthat/test-forecasters-basics.R

Lines 18 to 21 in b60b500

	expect_true(all(
	res$target_end_date ==
	as.Date("2022-01-01")
	))

as well as somewhat indirectly in the latency adjusting specific functions:

exploration-tooling/tests/testthat/test-latency_adjusting.R

Line 3 in 98c1370

expect_no_error(epidataAhead <- extend_ahead(case_death_rate_subset, 1))

[dshemetov]

Looks good for a first pass.

[nmdefries]

Looks good!