Epic: Improve reliability of prebuilds and prebuild logs

[jldec]

Summary
Users have reported sporadic issues with prebuilds not happening, or prebuild logs not showing up. The goal of this epic is to identify specific root causes in separate issues, and mitigate those one by one.

• [bridge] Ensure we do not override "failed" instance states (affects prebuilds!) #8596
• [dashboard, (server?)] Inconsistent derivation of prebuild status #8197
• Prebuilds failing with "failed OTS download" #8595
• Show prebuild configuration state on the project prebuilds page  #7010
• Prebuilds are reported as "FAILED" in the projects page, but cannot be retriggered because they are "still running"
• Prebuilds marking as failed but seem to have succeeded #7819
• make imagebuild logs independent of cluster locality #7874
• Incremental Prebuilds can sometimes execute an outdated config #7475
• Remove messagebus shovel #7468

[purkhusid]

I would like to add an issue that I've seen happen for my repo:

1. Be in the Web UI and click Open Workspace on a branch that has an active prebuild
2. Wait until the prebuild finishes and the workspace is automatically opened
3. Notice that the it seems like the prebuild was not presisted and the prebuild steps have been triggered inside the workspace on workspace open

[shaal]

@purkhusid I've seen that as well.
When I disable all automatic prebuild in Github - I no longer see that.
After a commit, if I didn't manually trigger a prebuild, I'll see the init tasks running when I open worksapce.
If I trigger a prebuild - I'll see the init tasks in prebuild, and after that every new workspace will not run again the init tasks.

[geropl]

In the context of this issue we found that failing prebuild instances would sometimes:
1.) fail on first start
2.) be re-started (why?)
3.) eventually stop, but without(!) an error: overriding the bad state with a"positive" signal in the DB ⚠️
This could be a root cause for the perceived unreliability, of course.

Note that this seems to occur most with private repos, so not sure if this is a general issue or not.
/cc @jldec @JanKoehnlein

[jldec]

I would like to include #8446 as part of this epic once we can identify / extract actionable sub-tasks.
#8637 may be a good candidate for an incremental prebuild UI improvement.

[geropl]

@easyCZ If you have a spare minute or two, and as you're the one with the most context loaded in your prefrontal cortex 😉 : Could you go through the issues listed here and see if they are still relevant? If not clear, please ask the original author for clarification.

Especially these maybe can be closed, because they are too broadly scoped, and/or the core issues have already been resolved with one of your PRs:

• [dashboard, (server?)] Inconsistent derivation of prebuild status #8197
• "Prebuilds are reported as "FAILED" in the projects page, but cannot be retriggered because they are "still running""

[easyCZ]

@geropl Thanks, I've already closed a couple yesterday and keeping an eye on these.

For #8197, there are still 2 more PRs that need to go in to fully normalize the status, will make sure I close this once they land.