[$150] Investigate and fix performance

[jmgasper]

@atelomycterus - We now have Tideways deployed and working in prod, thanks! Can you have a look and see if there are any quick fixes we can do to help with performance? I'm happy to adjust the payment up if we run into tricky issues, thanks.

[jmgasper]

Challenge https://www.topcoder.com/challenges/9fffbcc9-590f-4890-abde-800c9ce3c826 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

[jmgasper]

Challenge https://www.topcoder.com/challenges/9fffbcc9-590f-4890-abde-800c9ce3c826 has been assigned to obog.

This is an automated message for ghostar via Topcoder X

[atelomycterus]

@jmgasper Yes, sure. Keep you updated.

[atelomycterus]

@jmgasper Please see attached Tideways Report. While waiting for your feedback, I am working on #475. Let me know which issues from the report should be fixed first. Thanks!

[jmgasper]

@atelomycterus - I think the easiest place to start would be to see if we can update the memcache without first invalidating the data set. That way there's always data in the memcache, even if it may be slightly out of date.

[atelomycterus]

@jmgasper Yes, sure. We can do the following:

• Topcoder user data (Topcoder userId, handle, roles, pic url, rating). We can set 1 week or another value for it and update it from jwt token when user signs in. The Topcoder rating is only missing in jwt payload. The rating is recalculated after the end of the challenge, the expiration time can be set as the average duration of the challenge. Please correct me if I am wrong.
  Users who do not visit the Vanilla site, have not posted any content and are not members of groups will be removed from the cache automatically.
  Jwt payload:
  

• Role resources. Set default Memcached expiration time (3hours) or 1 day. I think that it is not updated often. Please correct me if I am wrong.

• Challenge role resources. Topcoder handles with challenge roles are displayed for Challenge forums. When
  /group/XXX/unarchive => add Challenge role resources in cache, set expiration time to endDate or endDate +1 day,
  If it is requested again then set a default Memcached expiration time.

Let me know your feedback.

[jmgasper]

1. Yes, this is fine. We can have the rating lag slightly without issue, I think, if it helps with performance.
2. Yeah, roles shouldn't be updated often, and if this becomes a problem we can design something in the API to update specific users manually
3. Yep, this sounds good.

[atelomycterus]

@jmgasper I am going to work on it tomorrow.

[atelomycterus]

@jmgasper Please apply PR-topcoder-platform/forums-plugins#88. Thanks! Sdgun will verify some tickets tomorrow. So I'll have a look and compare performance on DEV before and after changes. Keep you updated.

Please let me know which next tasks have higher priority.

[sdgun]

@atelomycterus Is there any impact of the performance fixes to any functionality? Just checking so I can focus on those before I do a sanity test. Thanks

[atelomycterus]

@sdgun From a functional point of view, no. This is more cache tuning, which data should be in the cache longer, etc. I'll verify it by analyzing requests with Tideways. So you can do a sanity test.

This morning there were a bunch of different suspicious requests. If you notice any strange behavior in Vanilla let me know. Thanks a lot!

[atelomycterus]

@jmgasper I've tested caching Topcoder data on DEV. It looks good. Let me know when the next production release is planned.
It would be nice to see the results when Vanilla has been working without deployment for several days.

Testing (DEV)

I flushed all memcached data. There were Topcoder API calls for the first several requests:

Group pages: Challenge roles were cached for challengeID. The second request didn’t load data from Topcoder API:

Discussion pages:

After 2 hours, no Topcoder API requests:

PROD

After deploying, I'll verify it:

• discussions.topcoder.com/group/896 - 50+ members
  https://app.tideways.io/o/Topcoder/Discussions/trace/8Mesy3gBhnQ2FlZ7D3PK - 70 Topcoder API requests -
• discussions.topcoder.com/group/883 - 50+ members
  https://app.tideways.io/o/Topcoder/Discussions/trace/qSe1y3gBmm2oVpuHN_w9 -22 Topcoder API requests
• https://discussions.topcoder.com/discussion/3596 20+ comments

[jmgasper]

Payment task has been updated: https://www.topcoder.com/challenges/9fffbcc9-590f-4890-abde-800c9ce3c826
Payments Complete
Winner: obog
Copilot: ghostar
Challenge 9fffbcc9-590f-4890-abde-800c9ce3c826 has been paid and closed.

This is an automated message for ghostar via Topcoder X