Syncers (pull from builds into webs) fail randomly

[humitos]

I found this error is triggered when there were some files that couldn't be transferred for some reason (partial transfer). The proper error code is 23 but I found that os.system changes it to 5888: https://bugs.launchpad.net/backintime/+bug/1265499/comments/2

(from Python's documentation: "On Unix, the return value is the exit status of the process encoded in the format specified for wait()." https://docs.python.org/2/library/os.html#os.system) --maybe a good moment to change to subprocess: https://docs.python.org/2/library/subprocess.html#subprocess-replacements

(from rsync's man page: "23 Partial transfer due to error")

Real example from /var/log/readthedocs/celery.log in web04:

Jul 25 06:28:04 web04 supervisord: celery-web04 readthedocs.builds.syncers:129[17025]: INFO Remote Pull /home/docs/checkouts/readthedocs.org/user_builds/fmriprep/artifacts/latest/sphinx/ to 
/home/docs/checkouts/readthedocs.org/user_builds/fmriprep/rtd-builds/latest

...

Jul 25 06:28:04 web04 supervisord: celery-web04 rsync: stat "/home/docs/checkouts/readthedocs.org/user_builds/fmriprep/rtd-builds/latest/.workflows-9.png.cJ23Qu" failed: No such file or directory (2)
Jul 25 06:28:04 web04 supervisord: celery-web04 rsync: rename "/home/docs/checkouts/readthedocs.org/user_builds/fmriprep/rtd-builds/latest/.workflows-9.png.cJ23Qu" -> "workflows-9.png": No such file or directory (2)

...

Jul 25 06:28:13 web04 supervisord: celery-web04 WARNING: api/index-1.pdf failed verification -- update discarded (will try again).

...

Jul 25 06:28:14 web04 supervisord: celery-web04 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [generator=3.1.1]
Jul 25 06:28:14 web04 supervisord: celery-web04 readthedocs.builds.syncers:144[17020]: ERROR Copy error to app servers. Command: [rsync -e 'ssh -T' -av --delete docs@build02:/home/docs/check
outs/readthedocs.org/user_builds/fmriprep/artifacts/latest/sphinx/ /home/docs/checkouts/readthedocs.org/user_builds/fmriprep/rtd-builds/latest] Return: [5888]

Sentry log: https://sentry.io/read-the-docs/readthedocs-org/issues/533076930/

Copy error to app servers. Command: [rsync -e 'ssh -T' -av --delete docs@build02:/home/docs/checkouts/readthedocs.org/user_builds/openpnm/artifacts/dev/sphinx/ /home/docs/checkouts/readthedocs.org/user_builds/openpnm/rtd-builds/dev] Return: [5888]

This problem may be causing issues like "Intermittent 404 pages" randomly: #2943

[humitos]

Most of the errors are because rsync is trying to delete weird files (same name than the original but starting with . and ending with random letters --looks like a temp file created by rsync itself)

Jul 25 15:24:59 web04 supervisord: celery-web04 deleting Scheme_Tutorials/Basics/.index.html.ZYTllh
Jul 25 15:24:59 web04 supervisord: celery-web04 rsync: stat "/home/docs/checkouts/readthedocs.org/user_builds/meep/rtd-builds/master/Scheme_Tutorials/Basics/.index.html.ZYTllh" failed: No such file or directory (2)
Jul 25 15:24:59 web04 supervisord: celery-web04 rsync: rename "/home/docs/checkouts/readthedocs.org/user_builds/meep/rtd-builds/master/Scheme_Tutorials/Basics/.index.html.ZYTllh" -> "Scheme_Tutorials/Basics/index.html": No such file or directory (2)

I'm not sure, but maybe this option is the one that it's bothering us:

--delete                delete extraneous files from dest dirs

On the other hand, we could ignore deletion errors with:

--ignore-errors         delete even if there are I/O errors

[dognie]

Details

• Project URL https://smartraiden.readthedocs.io/en/v0.3/
• Build URL https://readthedocs.org/projects/smartraiden/builds/7549646/

View raw
Build #7549646 log:

Read the Docs build information
Build id: 7549646
Project: smartraiden
Version: v0.3
Commit: d9f1294451f5efaa3cdfb52bd25b3d022512faae
Date: 2018-07-26T21:34:07.210229
State: finished
Success: True


[rtd-command-info] start-time: 2018-07-27T02:34:08.063969, end-time: 2018-07-27T02:34:10.430839, duration: 2, exit-code: 0
git clone --recursive https://github.com/SmartMeshFoundation/SmartRaiden.git .
Cloning into '.'...

[rtd-command-info] start-time: 2018-07-27T02:34:10.584782, end-time: 2018-07-27T02:34:10.887090, duration: 0, exit-code: 0
git checkout --force origin/v0.3
Note: checking out 'origin/v0.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at d9f1294... :memo: update Token Swaps

[rtd-command-info] start-time: 2018-07-27T02:34:10.986829, end-time: 2018-07-27T02:34:11.010040, duration: 0, exit-code: 0
git clean -d -f -f


[rtd-command-info] start-time: 2018-07-27T02:34:11.126146, end-time: 2018-07-27T02:34:11.135787, duration: 0, exit-code: 0
git branch -r
origin/HEAD -> origin/master
  origin/master
  origin/v0.3
  origin/v1.0

[rtd-command-info] start-time: 2018-07-27T02:34:12.149314, end-time: 2018-07-27T02:34:14.798504, duration: 2, exit-code: 0
python2.7 -mvirtualenv --no-site-packages --no-download /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/envs/v0.3
New python executable in /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/envs/v0.3/bin/python2.7
Also creating executable in /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/envs/v0.3/bin/python
Installing setuptools, pip, wheel...done.

[rtd-command-info] start-time: 2018-07-27T02:34:14.882711, end-time: 2018-07-27T02:34:24.459099, duration: 9, exit-code: 0
python /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/envs/v0.3/bin/pip install --upgrade --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/.cache/pip Pygments==2.2.0 setuptools<40 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 mkdocs==0.17.3
Collecting Pygments==2.2.0
  Using cached https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl
Collecting setuptools<40
  Using cached https://files.pythonhosted.org/packages/7f/e1/820d941153923aac1d49d7fc37e17b6e73bfbd2904959fffbad77900cf92/setuptools-39.2.0-py2.py3-none-any.whl
Collecting docutils==0.13.1
  Using cached https://files.pythonhosted.org/packages/5f/6d/e864b3c61b81eec57386ac62082fccfe694c7c3046d8723258a37da6d5fc/docutils-0.13.1-py2-none-any.whl
Collecting mock==1.0.1
Collecting pillow==2.6.1
Collecting alabaster!=0.7.5,<0.8,>=0.7
  Using cached https://files.pythonhosted.org/packages/6e/71/c3648cc2f675063dbe2d669004a59e4a5120172713a1de3c3b14144d4b31/alabaster-0.7.11-py2.py3-none-any.whl
Collecting commonmark==0.5.4
Collecting recommonmark==0.4.0
  Using cached https://files.pythonhosted.org/packages/df/a5/8ee4b84af7f997dfdba71254a88008cfc19c49df98983c9a4919e798f8ce/recommonmark-0.4.0-py2.py3-none-any.whl
Collecting mkdocs==0.17.3
  Using cached https://files.pythonhosted.org/packages/79/52/f395c87a3ec799586a8cdf8f360a96ccbbf7f119200204d17166cc4fb097/mkdocs-0.17.3-py2.py3-none-any.whl
Collecting livereload>=2.5.1 (from mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/dd/b4/213daced3ff1b4e02a1f700748e20e9a7481f5bfef57d11ae9babfd4aa2f/livereload-2.5.2-py2.py3-none-any.whl
Collecting PyYAML>=3.10 (from mkdocs==0.17.3)
Collecting Jinja2>=2.7.1 (from mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting Markdown>=2.3.1 (from mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/6d/7d/488b90f470b96531a3f5788cf12a93332f543dbab13c423a5e7ce96a0493/Markdown-2.6.11-py2.py3-none-any.whl
Collecting click>=3.3 (from mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl
Collecting tornado<5.0,>=4.1 (from mkdocs==0.17.3)
Collecting six (from livereload>=2.5.1->mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.7.1->mkdocs==0.17.3)
Collecting singledispatch (from tornado<5.0,>=4.1->mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting certifi (from tornado<5.0,>=4.1->mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Collecting backports-abc>=0.4 (from tornado<5.0,>=4.1->mkdocs==0.17.3)
  Using cached https://files.pythonhosted.org/packages/7d/56/6f3ac1b816d0cd8994e83d0c4e55bc64567532f7dc543378bd87f81cebc7/backports_abc-0.5-py2.py3-none-any.whl
Installing collected packages: Pygments, setuptools, docutils, mock, pillow, alabaster, commonmark, recommonmark, six, singledispatch, certifi, backports-abc, tornado, livereload, PyYAML, MarkupSafe, Jinja2, Markdown, click, mkdocs
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed Jinja2-2.10 Markdown-2.6.11 MarkupSafe-1.0 PyYAML-3.13 Pygments-2.2.0 alabaster-0.7.11 backports-abc-0.5 certifi-2018.4.16 click-6.7 commonmark-0.5.4 docutils-0.13.1 livereload-2.5.2 mkdocs-0.17.3 mock-1.0.1 pillow-2.6.1 recommonmark-0.4.0 setuptools-39.2.0 singledispatch-3.4.0.3 six-1.11.0 tornado-4.5.3
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[rtd-command-info] start-time: 2018-07-27T02:34:24.762354, end-time: 2018-07-27T02:34:24.863583, duration: 0, exit-code: 0
cat mkdocs.yml
docs_dir: /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/checkouts/v0.3/docs
extra_css: ['https://media.readthedocs.org/css/badge_only.css', 'https://media.readthedocs.org/css/readthedocs-doc-embed.css']
extra_javascript: [readthedocs-data.js, 'https://media.readthedocs.org/static/core/js/readthedocs-doc-embed.js',
  'https://media.readthedocs.org/javascript/readthedocs-analytics.js']
google_analytics: null
site_name: SmartRaiden
theme_dir: /home/docs/checkouts/readthedocs.org/readthedocs/templates/mkdocs/readthedocs

[rtd-command-info] start-time: 2018-07-27T02:34:24.965710, end-time: 2018-07-27T02:34:25.890826, duration: 0, exit-code: 0
python /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/envs/v0.3/bin/mkdocs build --clean --site-dir _build/html --config-file /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/checkouts/v0.3/mkdocs.yml --theme readthedocs
WARNING -  Config value: 'theme_dir'. Warning: The configuration option {0} has been deprecated and will be removed in a future relase of MkDocs. 
INFO    -  Cleaning site directory 
INFO    -  Building documentation to directory: /home/docs/checkouts/readthedocs.org/user_builds/smartraiden/checkouts/v0.3/_build/html

Actual Result

I have three branches：master, v0.3,v1.0.

• latest:master build successfully
• stable: v1.0 build successfully
• v0.3 : failed got this problem "Intermittent 404 pages" randomly: #2943

[humitos]

@dognie thanks for your report. The links of your project and build don't exist. Did you delete the project?

[dognie]

@humitos Thank you for your reply.Yesterday, I deleted the project that failed to build. Today, this problem still reappears.

• Project URL https://smartraiden.readthedocs.io/en/latest/
• Build URL https://readthedocs.org/projects/smartraiden/builds/7554960/
• latest:master build failed

Before I had the 404 problem in the 0.3 branch, I moved the document to master, and I thought it would be normal to build, and I didn't think of the same failure, but it's strange that master was normally built before.

[humitos]

I'm closing this issue because we decide that we will be pushing the HTML output to a cold storage soon. So, there is no need to spend time fixing this issue.