build hangs when conf.py blocks for TTY input; cannot kill hung build

[iandennismiller]

I had a misconfigured conf.py that blocked for TTY input (it invoked git with less). When I committed this and RTD began building the documentation, the builds started hanging. Now I have 3 builds that started ~45 minutes ago and they are still hanging. I'm concerned that I am clogging up the build queue with these 3 hung builds.

I have fixed the issue with my project so that it will not block for TTY input anymore. However, I want to kill the old builds so I can free up the queue. I cannot find a way to do this. Worse, it seems like the builds are not being automatically killed even though they've been running for ~45 minutes.

One solution is to expose a UI element in the admin interface that would cause a build to be killed. Another solution is to automatically kill builds after a shorter amount of time.

[gregmuellegger]

Hi thanks for the report! I agree we should make it possible to kill builds. But we need to decide where and when this will happen.

1. We could make a button in the admin panel like you suggested
2. We could kill already running builds if a new build starts for the very same version (since the old build will get superseded in short anyways)
3. Auto-kill all long running builds (that take more than ~30 Minutes!?)

@iandennismiller what project did you have this problem with? Maybe we need to cleanup that by hand for now.

[agjohnson]

• @iandennismiller was kind enough to alert us when his build was taking a long time, but I don't know that an admin button would get much use normally. If we go through the trouble of tracking build <-> task relationship, this would be a trivial addition -- if we decided to include it
• I'd be -1 on killing superseding builds, I feel like that would be unexpected behavior on our end
• We should probably have more protection against hung builds in code. I'm not sure if this is code that would live in the build process itself -- or if it would have to be orchestrated by an outside process.

#1524 improves container virtualization to a point that I would be comfortable deploying with it, and it will also address issues timing out long running builds or builds using excessive memory. However, there should probably be some in-code fail safe mechanism for killing builds that run long as well.

Additionally, to address this issue directly, we should at least be killing STDIN on forked processes.

[agjohnson]

RTD.org is now using containers for build encapsulation, which indirectly solves this problem. I don't think there is much we can do to limit things like processes waiting on STDIN anyways. Calling this resolved.