Retries ignored concurrency limit reached and failed status not reported to GitHub

[chrisburr]

Details

• Read the Docs project URL: https://readthedocs.org/projects/dirac/
• Build URL (if applicable): https://readthedocs.org/projects/dirac/builds/16750115/

I've seen this on several builds in the dirac project over today and yesterday.

Expected Result

I suspect this might be two different bugs but it's a little hard guess based on the information I have available:

• The build is retried until the maximum number of retries is reached.
• When the maximumn number of retries is reached GitHub sees the status as failed

Actual Result

• Build shows as Concurrency limit reached (2), retrying in 5 minutes.
• Our builds take ~10 minutes so I expect it to start after about 15
• In practice the Concurrency limit reached message persists and the build shows as Build failed
• I'm not sure when the Build failed status is assigned
• The status on GitHub stays in pending forever: [sweep:integration] Implement time machine DIRACGrid/DIRAC#6040

Debugging

• This might be a duplicate Builds not retrying after concurrency limit reached #9014
• If it is a duplicate the GitHub status not updating might be a legitimate bug?
• The build state being marked as failed might also be a bug Builds not retrying after concurrency limit reached #9014 (comment)
• I've seen issues with retries getting lost in recent versions of celery which might be the underlying cause auto retry fails when expires is set celery/celery#7091

[stsewd]

Hi, this seems to match #9014, and it looks like your build was triggered before we deployed that fix. If you see this again, please let us know.

But there is another case that will cause this behavior, we retry till 25 time max, if the build exceeds that number, the build will be marked as failed and will keep the Concurrency limit reached (2), retrying in 5 minutes. message, we should update our code to display something else in those cases.

[stsewd]

I opened #9148, I'm closing this, but let us know if you see this happening again (because reaching the retry limit should be rare I think).