2.12.x-release-package-unix fails with disk space errors

[SethTisue]

this has been happening every night at https://scala-ci.typesafe.com/job/scala-2.12.x-release-package-unix/

excerpt from most recent failure:

scala-2.12.1-46382e7-nightly/api/scala-library/scala/collection/mutable/LinkedHashMap$MappedValues.html
tar: /tmp/sbt_a08cf5bd/scala-docs-2.12.1-46382e7-nightly.tar: Wrote only 2048 of 10240 bytes
tar: Error is not recoverable: exiting now
 (deflated 88%)
  adding: scala-2.12.1-46382e7-nightly/api/scala-library/scala/Enumeration$ValueSet$.html
zip I/O error: No space left on devicetar: /tmp/sbt_86d86b5c/scala-docs-2.12.1-46382e7-nightly.tar: Wrote only 8192 of 10240 bytes
tar: Error is not recoverable: exiting now

zip error: Output file write failure (unexpected error on zip file)

but there is no disk space problem on jenkins-worker-ubuntu-publish that I can see:

% ssh jenkins-worker-ubuntu-publish
[...]
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.4G     0  7.4G   0% /dev
tmpfs           1.5G  8.7M  1.5G   1% /run
/dev/xvda1      7.8G  2.8G  4.6G  38% /
tmpfs           7.4G     0  7.4G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.4G     0  7.4G   0% /sys/fs/cgroup
/dev/xvdj        50G   12G   35G  25% /home/jenkins
tmpfs           1.5G     0  1.5G   0% /run/user/1001

We might need to observe disk space usage as the job actually runs.

[SethTisue]

yup! as the job runs:

jenkins@ip-172-31-15-209:/$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.2G  216M  98% /
jenkins@ip-172-31-15-209:/$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.2G  136M  99% /
jenkins@ip-172-31-15-209:/$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.4G  1.1M 100% /

[SethTisue]

@adriaanm can the size of the root partition on jenkins-worker-ubuntu-publish easily be increased? sbt-native-packager is using sbt.IO.temporaryDirectory which is just System.getProperty("java.io.tmpdir"), and on the worker /tmp is simply part of the root partition.

another fix route would be modify scripts/jobs/release/package/generic in the scala-dist repo to set java.io.tmpdir to something in the current directory

[SethTisue]

a third fix route would be to keep /tmp in a different partition so the root partition can be left alone to hold mostly static system files

[adriaanm]

yeah, the easiest thing would be to move tmp (attributes/worker.rb already has some logic to do this on windows, could extend this to ubuntu-publish and use /home/jenkins/tmp?)

[SethTisue]

I see, so you're suggesting a different fix path where we set java.io.tmpdir but do it in the scala-jenkins-infra repo, not by modifying the scala-dist repo.

unfortunately the situation w/r/t JVM options in scala-jenkins-infra is rather confusing. you suggested looking at worker.rb; I tried adding this there:

    default["jenkinsHomes"]["/home/jenkins"]["jvm_options"] = publisher ? "-Djava.io.tmpdir=/home/jenkins/tmp" : ""

but it didn't seem to work. I think because jvm_options is for the JVM that runs Jenkins itself, not the forked JVM that runs our script. note that we currently only set jvm_options for jenkins-master and it has -Dhudson... type stuff in it, so I'm pretty sure there.

my next try was scala/scala-jenkins-infra@7afc293, but it didn't seem to have any effect and I'm not sure why... maybe we want to set JAVA_OPTS instead of _JAVA_OPTIONS, via workerJavaOpts? I'll try that next... (one hopes that ANT_OPTS and MAVEN_OPTS aren't being used anymore...) scripts/jobs/release/package/generic in the scala-dist repo does explicitly use JAVA_OPTS

[adriaanm]

Ah, my bad -- the jvm_options stuff was for the worker agent launched by Jenkins. We could try JAVA_OPTS instead.

[adriaanm]

Could just start by changing this in https://scala-ci.typesafe.com/computer/jenkins-worker-ubuntu-publish/configure

[SethTisue]

• for the record, what I tried before with _JAVA_OPTIONS may have failed simply because I only ran chef-client on jenkins-master, not because the solution approach was wrong. regardless, I'm going to press forward with JAVA_OPTS instead
• changing JAVA_OPTS definitely requires running chef-client on the worker, not only on jenkins-master (verified by experiment, also confirmed by Adriaan)
• a test run done by manually editing JAVA_OPTS at https://scala-ci.typesafe.com/computer/jenkins-worker-ubuntu-publish/configure failed because /home/jenkins/tmp doesn't actually exist on the workers. I'll need to have Chef make it

[SethTisue]

I couldn't run chef-client on the worker because my ~/.ssh/config had User jenkins (which does not have sudo) instead of User ubuntu (which does). I will update the doc with this change.

[SethTisue]

it also turned out that the automatic runs of chef-client had been failing on the ubuntu publish worker for 4 months because (iirc the details) chef-vault wasn't in that node's run list, which Adriaan fixed in the Chef web UI. we manually inspected the diffs of the first chef-client run and made sure there wasn't anything alarming looking. (the change to the sbt launch script that Adriaan made a while back went through on this node only now; if something scala-ish breaks, that could be the cause)