CI false-negatives on conditional jobs

[anatoliykmetyuk]

The following was reported on Slack today:

It looks like GitHub action decided to report the progress sbt-test , sbt-java11 , publish_nightly and nightly_documentation on each PR even though they do not run. This reports false failures on each new PR.

I wrote the following to GH Actions support:

Hi,

We have some jobs in our workflow that run conditionally according to the jobs.<job_name>.if flag. Starting today, these jobs appear in each workflow run regardless of the "if" condition value. In case "if" condition is true, the jobs run as usual. In case it is false, the jobs fail without running. The only thing in the log output is "This check failed". This situation is quite annoying since it marks all our PRs as failing even though they pass.

I've minimized the problem: https://github.com/lampepfl/dotty/actions/runs/69871975
Above, we have two jobs, "good" and "bad". If you look at the workflow file, you'll see that "bad" is conditional. If you look at the logs of the "good" job, you'll find that it prints out the value of the "bad"'s condition, which is "false". You will also see that "bad" job is present in the CI workflow run and contains nothing in its log output but "This check failed".

If I remove the condition from the "bad" task, it runs successfully: https://github.com/lampepfl/dotty/actions/runs/69879643

I've also tried to reproduce the issue on a dummy repo, but it seems to be specific only to the lampepfl/dotty repo: https://github.com/anatoliykmetyuk/gh-actions-test/runs/558042216?check_suite_focus=true

Can you help us figure out what's going on?

Regards,
Anatolii

GH Actions Support Ticket ID: 630276

It doesn't seem like we can do much about it until they get back to us.

[nicolasstucki]

It looks like this is fixed. See #8684. Now it marks the skipped jobs explicitly

[anatoliykmetyuk]

The same on #8657. Looks like it was something on GH Actions side.