[Backfill corrections] Use dplyr joins in add_7dav fn

[nmdefries]

Description

Switch to dplyr join functions in place of base::merge, for speed.

Changelog

• preprocessing.R
• NAMESPACE

Note: Row order seems to impact model fit. This PR changes the final row order compared to using merge, and it has differences in some model coefficients versus earlier versions. Most changes are small (coefficients are approximately equal, e.g. within 10e-8). Some models also have the same coef value but the associated predictor switched (e.g. from Wed_issue_coef = 0 and Sat_ref_coef = 0.683, to Wed_issue_coef = 0.683 and Sat_ref_coef = 0).

As a result, some predicted values are different. Avg WIS is slightly different as well.

That said, all this seems inconsequential; the original merge result was not sorted in any particular way. Technically merge sorts by the by columns, but the sort algorithm it uses appears to not be stable, so the rows get un-ordered WRT other non-by fields like lag and time_value.

[nmdefries]

I verified that using the old Reduce-merge approach also results in slightly different model fits/predictions if the sort order of the data is changed. The intermediate dfs of the old and new merge step are also identical once sorted, so it's only the sorting that changes the final result.

[nmdefries]

Options on what to do with this code chunk:

• Leave implementation as-is.
  • + no change in prediction/model output
  • - slowest option
  • - sort order is pseudo-random and arbitrary. Any other code change that changes the sort order will result in the output changing.
• Use new join-based implementation
  • + fast(est)
  • + df is not arbitrarily sorted
  • - results don't exactly match merge-based implementation
• Use new join-based implementation, but sort it to match old implementation
  • + faster than old code
  • + results exactly match merge-based implementation
  • - not entirely sure how to recreate sort order from merge implementation. I guess I can take the sort code they're using?
  • - sort slows code down and is arbitrary. Any other code change that changes the sort order will result in the output changing.

[nmdefries]

My vote is for the second option. I don't really see any reason to try to make sure a new implementation maintains the same sort order and thus model/prediction output. The original sort order was arbitrary and unintentional. The results are very similar using different sort orders.

[jingjtang]

My vote is for the second option. I don't really see any reason to try to make sure a new implementation maintains the same sort order and thus model/prediction output. The original sort order was arbitrary and unintentional. The results are very similar using different sort orders.

Agree. Vote for the second one as well!

[nmdefries]

Output before and after the included code changes. The R script loads the files, and finds and compares differing columns.

20230320_new_reciving.tar.gz
20230320_old_reciving.tar.gz
comparing_differing_output.R.txt (This is an R script, but GH wouldn't accept it with that file extension.)

[nmdefries]

@jingjtang This is ready for review.

[jingjtang]

LGTM

[nmdefries]

@krivard This is ready to merge