Covidcast table refactor for data versioning performance and quantile support

[krivard]

Proposed new table design:

• Invariant parameters: (id, source, signal, time_type, geo_type, geo_value, time_value, most_recent_fk)
• Versioned indicator values: (id, invariants_fk, value, std, sample_size, direction, ts1, ts2, issue, lag)
• Versioned forecast values: (id, invariants_fk, value, std, sample_size, direction, ts1, ts2, forecastdate, lag)
• Forecast parameters: future work but probably something like (id, invariants_fk, quantile, forecast_value_fk)

[krivard]

Migration could be complicated. Options:

• Load all the data from scratch
• Shift data from another live table (could need tons of memory)

Currently exploring loading/transferring data in batches

If all else fails could just bring it up on a separate VM and transfer DNS over once the data is all loaded. Would need to handle DNS TTL carefully in advance; coordinate with CMU on that.

Meet to discuss details next week.

[krivard]

Performance issues arise!

[krivard]

Working on removing the direction stuff so we can stop contorting ourselves to work around it.

[krivard]

Mystery test gremlins conquered, PR imminent