DRAFT: 285 rfm model

[bdeck8317]

Meet to discuss integration into pipeline. Will need a var passed into create_scores.py which is a str(query_date)

Updates the rfm_scores table in postgres database.

Adds a number of module dependencies to requirements.txt

[c-simpson]

I'm getting this error running create_scores():

  File "C:\Projects\paws-data-pipeline\src\server\rfm_funcs\create_scores.py", line 73, in create_scores
    grouped_past_year['recency_score'] = pd.cut(grouped_past_year[('days_since','min')], bins= recency_bins, labels=recency_labels, include_lowest = True)
  File "C:\Python38\lib\site-packages\pandas\core\reshape\tile.py", line 262, in cut
    raise ValueError("bins must increase monotonically.")
ValueError: bins must increase monotonically.

with rfm_edges as:

{
"r": {"5": 0, "4": 262, "3": 1097, "2": 1910, "1": 2851}, 
"f": {"1": 0, "2": 1, "3": 2, "4": 3, "5": 4}, 
"m": {"1": 0.0, "2": 50.0, "3": 75.0, "4": 100.0, "5": 210.0}
}

It looks like the issue is with the 'r' set. Do the values need to be reversed wrt the keys? 

[bdeck8317]

@c-simpson looks like a pandas issue with handling numpy.diff. Could try updating pandas to 1.3.2

Apparently, the bug is fixed in that version as I haven't had this issue. I also ran the script using the API and it seemed okay....

I'll keep digging around on this issue, but my temporary solution is to just update pandas to 1.3.2

pandas-dev/pandas#40969

[c-simpson]

I'm still getting it with 1.3.2

[c-simpson]

We poked in this while you weren't around and noticed that

paws-data-pipeline/src/server/rfm_funcs/create_scores.py

Line 71 in 5367b6e

recency_bins.append(grouped_past_year[('days_since', 'min')].max())

adds 43 to the end of recency_bins, which causes the 'monotonic' issue. In the debugger, if I put 43 in the proper sort position, we don't get the monotonic complaint. We didn't know exactly what you were doing so we stopped there.

[c-simpson]

@bdeck8317 I created a branch ( 285-sort-after-append ) off yours that does nothing more than sort recency_bins after you append. It runs but only creates scores for 754 matching_ids. So I'll stop messing with things I don't understand and will let you carry on!

[bdeck8317]

@c-simpson @sposerina ready for review and initial integration. There is one conflict. Not sure how to address it.

[c-simpson]

The 'append 43' is making me think about robustness:

• What happens if date_differences() gets a badly-formatted date (shouldn't happen)? A future date(could) ? Can callers handle?
• In create_scores(), what should happen if read_rfm_edges() returns None?
• It might be cleaner to put the r,f,m processing in a separate function for each and put a try/catch in each. What to return if a failure?
• If one of those fails, what should we do? We don't want to push bad data. @kfettich Any thoughts?

[c-simpson]

Spoke to BD (who is swamped) - I'll make the above changes this week.

[kfettich]

Sorry, I missed the mention! I think we discussed briefly, but my thoughts if any of the 3 failures happens:

• What happens if date_differences() gets a badly-formatted date (shouldn't happen)? A future date(could) ? Can callers handle? -> future dates should be filtered out; meanwhile, a user who has donation data with badly formatted dates should display some kind of warning (either an "error" label instead of an RFM label, or something else to notify the user that there was a problem calculating the score)

• In create_scores(), what should happen if read_rfm_edges() returns None? -> revert to the previous set of edges that worked, and alert dev team

• It might be cleaner to put the r,f,m processing in a separate function for each and put a try/catch in each. What to return if a failure? ->return last RFM score for person, but with some indication that it is not the most recent score

If one of those fails, what should we do? We don't want to push bad data. @kfettich Any thoughts? -> see above; I don't expect there to be dramatic changes from one cycle to the other, so using outdated RFM scores from the previous cycle should be ok; doing it more than once will be problematic though