Increase parallelism in Travis builds #2047

tautschnig · 2018-04-12T11:32:08Z

Rarely used configuration options (aka preprocessor defines) that are not
generally used in production settings can safely be tested at cron-defined
intervals only rather than upon every PR/change to a PR.

The remaining non-cron builds are: {GCC, Clang} x {Linux, OS X} +
CMake/GCC/Linux.

The changes proposed here break up large build+test tasks into separate steps so that as much work as possible can be done in parallel, resulting in failures being reported as soon as possible.

thk123 · 2018-04-12T12:58:35Z

IMO this is not better than the random scattering of options (e.g. per the matrix suggested in #1706). I believe this changes not knowing precisely which variant flag causes the problem (around ~6 options) to not knowing precisely which PR caused the problem (~3/day).

However, I think it is easier to try different options (easy to turn on flags locally on your PR, easy to compile with make rather than cmake) than check out each of the PRs.

Further, this allows broken stuff to get merged. There then ends up being a lag in getting it fixed, and for that duration more PRs can break the configuration that doesn't work.

tautschnig · 2018-04-12T13:09:05Z

@smowton @thk123 would you mind (possibly offline) reaching an agreement on cron/no-cron. Either way I'm starting to think that 4e46d01 might be one of the bigger steps forward. So much that I'm thinking we might not have to resort to cron at all?

smowton · 2018-04-12T13:43:26Z

.travis.yml

@@ -173,12 +203,19 @@ jobs:
            - libstdc++-5-dev
            - libubsan0
      before_install:
+        - |
+          if [[ "${TRAVIS_EVENT_TYPE}" != "cron" ]]


We can do better, not starting the job at all, like https://docs.travis-ci.com/user/conditional-builds-stages-jobs/#Conditional-Builds

smowton · 2018-04-12T13:49:02Z

@thk123 my take is the matrix is actually quite large: roughly 2 OSes x 2 compilers x 2 build systems x 2^5 (or so) flag combinations.

By scattering the flags around the different configs we effectively randomly sample the matrix, testing <linux, g++, !USE_DSTRING>, <osx, clang, SUB_IS_LIST>, <linux, g++, USE_GLUCOSE>, ...

Contrast the solution proposed by this PR: we use all the most common entries in the matrix at PR time.

In either case we're missing some matrix cells which are untested and so may break. There's not much getting around that unless you're willing to do 2^8 or so builds per PR. The question is, what configurations do we most care about not breaking? I'd say surely vanilla configs, as these are most likely to be used by new / inexperienced users.

As to tracking down how we broke it: git bisect is efficient and easy to use. With only a week (~21 PRs) to bisect in you should track the culprit down in 5 builds.

smowton · 2018-04-12T13:51:11Z

Incidentally if you want to include other particular configs in the every-PR build then fair enough -- but this allows the weekly job to check <osx, clang, !USE_DSTRING, NDEBUG, SUB_IS_LIST>, which is obviously never tested at present.

tautschnig · 2018-04-12T15:05:56Z

Does anyone know how many jobs Travis will run concurrently within a stage? Assuming all the builds of various configurations take roughly the same amount of time we could run a lot of them, as long as they execute concurrently.

Edit: I think we are wasting a lot of available time in the first stage (actually n-1 slots, all waiting on this one job). Let me try to work on this.

smowton · 2018-04-12T15:14:36Z

This is bounded by our willingness to burn cash I suspect ;)

tautschnig · 2018-06-01T09:17:41Z

Out-of-band conversation: it seems 5 or 10 concurrent builds should be within the current limits. I'll tweak the configuration accordingly.

tautschnig · 2018-06-05T11:38:06Z

This should now be ready for further discussion and/or another review. Note that I have updated the description of the pull request at the top - it used to be about doing (more) cron builds, which now isn't being changed; instead, more parallelism is introduced.

Tests are run, otherwise using the very same configuration as the primary build, in the second stage. As all tasks within one stage run in parallel, the ~20mins that test execution takes will pass while also doing other work rather than blocking progress to the next stage. Via ccache we will avoid spending time for re-building.

Test configuration options in parallel rather than combining them to make diagnosing problems easier. Don't squash the build matrix as diffblue#1706 suggested earlier. Fixes: diffblue#1706

We can run up to 10 jobs in parallel - make use of that!

…g tests

chrisr-diffblue · 2018-06-21T13:03:37Z

.travis.yml

      before_install:
-        - mkdir bin ; ln -s /usr/bin/gcc-5 bin/gcc
+        - HOMEBREW_NO_AUTO_UPDATE=1 brew install ccache parallel


Don't need to install parallel here, as that's only used for tests.

chrisr-diffblue · 2018-06-21T13:12:10Z

.travis.yml

      # env: COMPILER=clang++-3.7 SAN_FLAGS="-fsanitize=undefined -fno-sanitize-recover=undefined,integer -fno-omit-frame-pointer"
      env:
        - COMPILER="ccache /usr/bin/clang++-3.7"
        - EXTRA_CXXFLAGS="-Qunused-arguments -fcolor-diagnostics -DNDEBUG"
        - CCACHE_CPP2=yes

-    # Ubuntu Linux with glibc using clang++-3.7, debug mode, disable USE_DSTRING
+    # Ubuntu Linux with glibc using clang++-3.7, debug mode
    - stage: Test different OS/CXX/Flags


Should these 'build only' tests (i.e. the debug builds) be either moved into the first stage (i.e. builds), or positioned into a third 'debug builds' stage? I don't have a strong opinion, but thought it was worth us making an explicit decision.

chrisr-diffblue · 2018-06-21T13:14:18Z

.travis.yml

+          packages:
+            - libwww-perl
+            - g++-5
+            - libubsan0


Is libubsan0 actually needed for this build, is it a hangover from when we used to have sanitizer builds? (ditto for elsewhere in this file as well)

chrisr-diffblue · 2018-06-21T13:19:59Z

.travis.yml

+      env:
+        - NAME="USE_STD_STRING"
+        - COMPILER="ccache /usr/bin/g++-5"
+        - EXTRA_CXXFLAGS="-D_GLIBCXX_DEBUG -DUSE_STD_STRING"


Previously, the USE_STD_STRING builds were not running tests, with this patch hunk they will now be running tests (which appear to fail) - is this an intentional change? If so, we need the tests and/or USE_STD_STRING fixed before merging. :-)

tautschnig · 2020-11-07T00:06:10Z

Closing as most CI tasks are nowadays performed via GitHub actions.

tautschnig requested review from forejtv, peterschrammel and thk123 as code owners April 12, 2018 11:32

tautschnig mentioned this pull request Apr 12, 2018

Repair HASH_CODE support and enable it in Travis #2044

Closed

tautschnig force-pushed the travis-cron branch from 42e64da to 4e46d01 Compare April 12, 2018 12:21

smowton reviewed Apr 12, 2018

View reviewed changes

tautschnig force-pushed the travis-cron branch from 4e46d01 to 73579ed Compare April 12, 2018 14:55

tautschnig force-pushed the travis-cron branch from 73579ed to d282ba4 Compare April 12, 2018 18:29

tautschnig assigned smowton, thk123, peterschrammel and forejtv Apr 13, 2018

tautschnig mentioned this pull request May 22, 2018

Post-jbmc move cleanup #2215

Open

3 tasks

tautschnig assigned tautschnig and unassigned smowton, thk123, peterschrammel and forejtv Jun 1, 2018

tautschnig changed the title ~~Move more Travis jobs to cron-only mode~~ Increase parallelism in Travis builds Jun 5, 2018

tautschnig force-pushed the travis-cron branch 2 times, most recently from 11c5c24 to a190e47 Compare June 5, 2018 10:19

tautschnig assigned smowton and unassigned tautschnig Jun 5, 2018

tautschnig assigned thk123, peterschrammel, forejtv and chrisr-diffblue Jun 5, 2018

tautschnig added 6 commits June 11, 2018 17:14

Travis: Perform the CMake build (but not the tests) in the first stage

d0fd2a9

Do not mix configuration options

2caa469

Test configuration options in parallel rather than combining them to make diagnosing problems easier. Don't squash the build matrix as diffblue#1706 suggested earlier. Fixes: diffblue#1706

Travis: Move the clang (incl OSX) build to the first stage as well

8d37bc4

We can run up to 10 jobs in parallel - make use of that!

Remove build and preparation steps that are only required when runnin…

210782e

…g tests

Remove redundant setting of variable

219b430

tautschnig force-pushed the travis-cron branch from a190e47 to 219b430 Compare June 11, 2018 17:15

tautschnig assigned tautschnig and unassigned smowton, thk123, peterschrammel, forejtv and chrisr-diffblue Jun 12, 2018

chrisr-diffblue reviewed Jun 21, 2018

View reviewed changes

tautschnig closed this Nov 7, 2020

tautschnig deleted the travis-cron branch November 7, 2020 00:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increase parallelism in Travis builds #2047

Increase parallelism in Travis builds #2047

tautschnig commented Apr 12, 2018 •

edited

Loading

thk123 commented Apr 12, 2018

tautschnig commented Apr 12, 2018

smowton Apr 12, 2018

smowton commented Apr 12, 2018 •

edited

Loading

smowton commented Apr 12, 2018

tautschnig commented Apr 12, 2018 •

edited

Loading

smowton commented Apr 12, 2018

tautschnig commented Jun 1, 2018

tautschnig commented Jun 5, 2018

chrisr-diffblue Jun 21, 2018

chrisr-diffblue Jun 21, 2018

chrisr-diffblue Jun 21, 2018

chrisr-diffblue Jun 21, 2018

tautschnig commented Nov 7, 2020

Increase parallelism in Travis builds #2047

Increase parallelism in Travis builds #2047

Conversation

tautschnig commented Apr 12, 2018 • edited Loading

thk123 commented Apr 12, 2018

tautschnig commented Apr 12, 2018

smowton Apr 12, 2018

Choose a reason for hiding this comment

smowton commented Apr 12, 2018 • edited Loading

smowton commented Apr 12, 2018

tautschnig commented Apr 12, 2018 • edited Loading

smowton commented Apr 12, 2018

tautschnig commented Jun 1, 2018

tautschnig commented Jun 5, 2018

chrisr-diffblue Jun 21, 2018

Choose a reason for hiding this comment

chrisr-diffblue Jun 21, 2018

Choose a reason for hiding this comment

chrisr-diffblue Jun 21, 2018

Choose a reason for hiding this comment

chrisr-diffblue Jun 21, 2018

Choose a reason for hiding this comment

tautschnig commented Nov 7, 2020

tautschnig commented Apr 12, 2018 •

edited

Loading

smowton commented Apr 12, 2018 •

edited

Loading

tautschnig commented Apr 12, 2018 •

edited

Loading