V4 to V5 Sync Issue: V4 Challenge Phase updates not seen in V5

[SathyaJayabal]

topcoder-platform/community-app#4569

cc @lakshmiathreya

[SathyaJayabal]

Looks good now @rootelement

[SathyaJayabal]

@rootelement , Phases were synced on 25th morning IST after your update. Any phase updates that happened from the 25th to 26th IST, are not synced to V5.

Example:
http://api.topcoder.com/v5/challenges?legacyId=30130329
http://api.topcoder.com/v4/challenges/30130329

cc @lakshmiathreya

[rootelement]

@SathyaJayabal Please close this here if this is fixed.

[SathyaJayabal]

@rootelement this issue is not fixed. The phases line up only when the migration script is run and not in real time as expected (with 1-5 mins lag).

When there is any change in V4 (when review phase ends - from Online Review, Registration phase opens etc) it must reflect in real time in V5. We have been going back and forth over this many times. I think we have a misunderstanding. The previous examples which I have given works now because the migration script is run. What is not working is the real time sync between v4 and v5 and vice versa.

Please see example below:
https://www.topcoder.com/challenges/30131013
https://beta-community-app.topcoder.com/challenges/4d8767f9-de20-4e67-b022-44badf9a7c9f

[SathyaJayabal]

cc @lakshmiathreya @mtwomey

[lakshmiathreya]

@rootelement @mtwomey - what @SathyaJayabal is trying to emphasize is the need for v4 and v5 to be in real time sync ... this would become critical once v5 goes to Prod as legacy Apps such as Direct and OR would continue to read off v4 ... we'd continue to depend on legacy Apps for Challenge Creation and Updates even after v5 goes to Prod ... cc @sushilshinde

[lakshmiathreya]

Related - #204

[rootelement]

The sync is not going to be "real time". It's about 1 to 3 minutes. All the challenges you list here are in sync now.
http://api.topcoder.com/v4/challenges/30131013
http://api.topcoder.com/v5/challenges/4d8767f9-de20-4e67-b022-44badf9a7c9f

Those phases match, everything is closed. Beyond that, i'm not sure what we can do. I can crank up the periodicity of the sync, but that's going to put a strange on v4, v5 apis and informix as it cross compares everything.

[SathyaJayabal]

@rootelement , I don't mean the 1 to 3 minutes delay. There are some phase transitions in v4 that are not synced in v5.
For example, when the review phase completes and appeals begin, this is not synced to v5. The examples contests I provide get completed by the time you check on them.

Please look at any contest in the ongoing bucket in prod which is in appeals phase. And compare the same contest in v5.

https://www.topcoder.com/challenges?bucket=ongoing
https://beta-community-app.topcoder.com/challenges?bucket=ongoing

Another example you can find is: Some challenges in production (v4) are in registration/submission phases, but are in stalled phase in V5(beta). Please check the listing pages for examples

Let us sync up on slack and I can show you real time examples if that works for you.

[SathyaJayabal]

cc @lakshmiathreya

[rootelement]

Here's the functionality order of operations:

1. User creates a challenge in v5. Legacy processor puts this into informix via v4 api. This should take < 1min
2. Autopilot performs an action on a challenge, like reading the time on a phase and updating the status. This happens in informix. The challenge "updated" property is set to the time of that action.
3. v4 Elastic Search feeder pics up that change in informix and loads that change into Elastic Search. This could take ~5min
4. v5 Sync queue service loops through every 1 min and picks up challenges edited in the last 10 minutes to compare. If the modified date we have logged on our side doesn't match the updated date on the v4 ES side, it queues that challenge for update
5. v5 Sync processor loops through every 2 minutes (every minute divisible by 2) and pulls the challenge properties over and adds any missing resources.

So this process can take upwards of 7 or 8 minutes to perform. We can ratchet down the sync window times, but that will have to be a team effort, because I'm not positive of the system-wide impact of this.

The fact that you say by the time I see the issue, it's fixed, means that the sync is working, because i'm doing nothing other than letting the sync system run. So if the Appeals phase moves to closed that quickly, there is a chance the whole system may miss that process. This is a drawback to the v4/v5 sync method and we must focus on moving autopilot to v5 systems.

The only thing I'm not sure of is the Stalled challenge status. I'm looking into that part.

[SathyaJayabal]

@rootelement, please also look at the review to appeals phase changes. It doesn't sync up to more than 24 hours, not just 7-8 mins, You can check the screenshots above. Sync happens only after appeals response completes and contest ends. We can never see a contest in appeals or appeals response phases in V5(beta). The phase is always in review (when in actuality the appeals and appeals response phases are going on) and then it goes to completed (thats a delay of 24+12 = 36 hours). Same contest goes through appeals and appeals response phases in v4(prod).

cc @lakshmiathreya

[lakshmiathreya]

@rootelement there are other examples too with some other phases too -

• pls see the one below:

[SathyaJayabal]

@mtwomey @rootelement , we still see a lot of challenges with phases out fo sync.
cc @lakshmiathreya

[mtwomey]

James is testing fix now in dev.

[SathyaJayabal]

@rootelement , phases sync looks good so far. Will observe for a couple fo days.

cc @lakshmiathreya

[SathyaJayabal]

Phase sync looks good.

[SathyaJayabal]

Phases seem out of sync again. cc @rootelement @lakshmiathreya

[SathyaJayabal]

phases are good now