Merge pull request #195 from SethTisue/disk-space-advice

document free-disk-space maintenance

Author: adriaanm

doc/maintenance.md

@@ -14,9 +14,39 @@ For troubleshooting Jenkins in particular it's often to helpful to ssh
 in to the worker nodes and poke around the workspaces, see what
 processes are running, that kind of thing.
 
-You can also delete files if we've run out of disk space, though our
-first line of defense on this is the "Clear Workspace" button in the
-Jenkins UI.
+## Monitoring disk space
+
+A member of the Scala team should check the free disk space numbers
+on all nodes about every 3 to 4 days.  We should always have at
+least 10G free on all nodes.
+
+A quick way to keep an eye on disk space is by visiting
+https://scala-ci.typesafe.com/computer/.  You'll need to launch
+any offline nodes in order to get a disk space number.
+
+All of the nodes have a tendency to gradually run out of space, and
+it's also possible to have a temporary space issue, e.g. if a
+community build is run on a lot of different Scala versions.
+
+Tips for addressing gradually dwindling free space:
+
+* on jenkins-worker-windows-publish, the usual culprit is
+  `/home/tmp`.  ssh in and just blow away everything in there
+  (but not while a job is running!).
+* on jenkins-master, the culprit may vary.  Usually the problem
+  is under `/var/lib/jenkins`. Poke around with commands like
+  `find` and `du` and see where the space is going.
+  (A frequent culprit prior to fall 2016 was multi-gigabyte log
+  files generated by failed PR validation jobs, but we think
+  this was fixed by https://github.com/scala/scala-jenkins-infra/pull/183.)
+
+Tips for addressing a temporary free-space issue on the behemoths:
+
+* Usually the problem is in `/home/jenkins`.
+* You can use the "Clear Workspace" button in the Jenkins UI.
+* Alternatively, on the command line, in `/home/jenkins/workspace`,
+  you can blow away `*/{dbuild,clones,target}-*`.  (Not while a
+  community build job is running!)
 
 ## Updating the scala-jenkins-infra cookbook