Provide Github (etc) commit status for builds on branches / tags

[Cadair]

This is not in relation to PR builds.

It would be really nice if after a build is triggered on a commit (i.e. a PR is merged or a tag pushed) if RTD sent a commit status back to the VCS site that triggered it. This would allow you to see the status of the RTD build along side all the other CI checks on that commit.

A bonus would be supporting the GitHub checks API to allow restarting the build etc.

[ericholscher]

I think this is useful, and I was just wanting it today 👍 Putting it in Q1 roadmap, since I don't think it will be a huge amount of work, since we already have this code for PR's.

[benjaoming]

UPDATE These cases are elaborated here: #10320 (comment)

I think the explanation for why this wasn't immediately added is the following:

We have a very long sequence of calls and celery tasks. An incoming webhook is received and a new build is triggered. The build isn't supplied any information about the commit value from the webhook call, even though the information is available. So we just build a version at whatever commit that's the latest in the branch at checkout/build time.

• Major change: We need to start passing the commit hash from the incoming webhook call to the entire chain of calls that initiate a build task. If we apply the commit hash, we will also change build behavior. UPDATE: This is now implemented in Build: Handle commit hash on incoming webhooks and send commit status on all builds #10320.

• If we receive many push events for individual commits, we have an auto cancellation feature. But I'm not sure how it will work once we start to use explicit commit hashes. It could potentially stop working. Will investigate. UPDATE: I still need to understand this a bit better, since we were passing commit IDs on Pull Request events, and build cancellation did seem to work well here.

• Need to understand behavior when webhooks are received for commits that are overwritten before the build is started (force-pushes). We should figure out if we want to just silently ignore commits that do not exist and auto-cancel a build. UPDATE: This doesn't appear to be an issue, the build from an overwritten commit can just fail, status update goes to the same commit ID which doesn't exist, hence that's not a problem.

[benjaoming]

Oh yeah: In case someone wonders "why not just send back a status using the current pattern to whatever commit is the latest after checkout?"

This is because the "pending" status is sent before the build is even started. And a checkout happens during the build, so in this case we cannot set the "pending" status without checking out the repo. Which is not nice. If that was to be fixed, we'd have to implement more changes than what is outlined above.

We also don't want to send subsequent failed/succeeded statuses to a different commit id, so we should track the same commit id from triggering the build to the build process.

[benjaoming]

Another point: A push can contain several commits. We always build the last commit.

UPDATE: All webhooks list which is the latest commit and we can guarantee that it's the commit id we are passing to the build processes.

[benjaoming]

New implementation looking good!

Test with a build failure:

Test with build success:

[humitos]

I'm de-prioritizing this issue for now because the work required here is not trivial.

[humitos]

I'm going to close this issue because we weren't able to prioritize in the past year and we are not planning to work on it soon. We can revisit in the future when working on expanding GitHub features.