Git LFS support?

[jonathansick]

We're using Git LFS to store images for our docs. I tried building a test doc on RTD with a Git LFS-managed image, and it looks as though the git-lfs client isn't install on the RTD's build servers. Is it true that Git LFS isn't currently setup on readthedocs.org? Or am I messing something up somehow?

I think there's a good use case for letting people manage static binary assets with Git LFS, and installing git-lfs (see https://git-lfs.github.com) client is fairly easy.

[jonathansick]

To be concrete, I have a branch of my docs (https://github.com/lsst-sqre/lsst_stack_docs/tree/tickets/DM-4511) that includes a Git LFS managed image (https://github.com/lsst-sqre/lsst_stack_docs/tree/tickets/DM-4511/_static/development/docs)

The RTD build seems to be failing https://readthedocs.org/projects/lsst-stack-docs/builds/3548182/ at the LaTeX build stage because it can't read and covert the JPEG image. I'm guessing that the git-lfs client isn't installed?

[agjohnson]

So no, the LFS client is not currently part of our server provisioning. We'll want to spend some time with it before it goes into production and we call it supported.

Without much time to look into this, it wasn't immediately clear if LFS makes the files available on cloning. It sets up git hooks, which wouldn't be present on clone, requiring extra commands for us to run. Additionally, the docs are not clear about how the file is presented on the filesystem once installed. Is the link file replaced with the actual file? Or is the LFS client expected to be an intermediary somehow?

The build above is failing because it's getting the link file, which doesn't have a JPEG header. The HTML output will likely just show a broken file.

Marking this as community support for now, our roadmap is very full right now, and I'm not sure how far out support for this might be. If it comes down to just being a server provisioning change, that's a lot less daunting.

[jonathansick]

I think it's 'just' a server provisioning change. Basically you can get the Go binary from https://git-lfs.github.com and run

git lfs install

once on the server to setup LFS in the ~/.gitconfig files.

Then the git clone and git checkout commands work exactly as you'd expect.(*) The needed files are pulled down from the server as needed. This should actually save you bandwidth since only the current copy of LFS-managed binary files are pulled down rather than all copies in Git history.

---

(*) Though GitHub Git LFS users should be easy to support, I just realized that since we're using our own Git LFS server at the Large Synoptic Survey Telescope, rather than GitHub's, there's a kludgy aspect of being prompted for usernames and passwords (which can be blank) on clone; so I don't think our Git LFS will work with a RTD build in the foreseeable future. See http://docs.lsst.codes/en/latest/development/git_lfs.html#setting-up-git-lfs-authentication (@jmatt any opinions?)

That said, most people will be using GitHub's LFS server that works perfectly and might benefit from everything 'just working' on RTD.

[agjohnson]

Well, there is still the issue of the git lfs install command needing to be run on each project repository clone, which also requires a setting in RTD to enable this feature. This makes the feature a bit more than a server provisioning change. We'd likely rather build this into https://github.com/rtfd/readthedocs-build, where it can live as a configurable step via a YAML config.

[psi29a]

Yes please! Multiple projects I'm working on make use of LFS. Two of which are for hosting images/graphs for our documentation.

[swistakm]

For anyone interested in simple workaround: you can easily use git-lfs Python package to get all LFS-stored files during RTD build. See example: https://github.com/swistakm/pyimgui/blob/master/doc/source/conf.py#L36-L37

[AlexMAS]

Does anyone know about this issue? I try to repeat @swistakm but it isn't work for me. May be I do something wrong. Can you help me to configure my project to force RTD to work with git lfs?

[AlexMAS]

Yes, @swistakm is right. His solution works fine! Thank you! When I had moved the invocation of git_lfs to the top of conf.py it began to work.

https://github.com/InfinniPlatform/InfinniPlatform.readthedocs.org.en/blob/master/docs/source/conf.py#L18-L31

Also I added git-lfs dependency to my pip.txt.

https://github.com/InfinniPlatform/InfinniPlatform.readthedocs.org.en/blob/master/requirements/pip.txt

But anyway I hope that git lfs will have native support.

[agjohnson]

Implementing this in RTD will be non-trivial, but the suggested workaround would make a great guide topic in our docs/guides/ documentation. I think we can at least start there, but adding another build step and configuration/etc for this is a sizeable amount of work.

[ljmf00]

Trying the LFS hack by @swistakm and @AlexMAS and got an error.

Fetching files with git_lfs
Can't checkout into a bare repo, please provide a valid checkout_dir
    raise ConfigError(CONFIG_EXIT_ERROR)
ConfigError: The configuration file (or one of the modules it imports) called sys.exit()

Configuration error:
The configuration file (or one of the modules it imports) called sys.exit()

---

Reference:

• [1] https://readthedocs.org/projects/aurorafoss-learn/builds/8327656/
• [2] https://github.com/aurorafossorg/learn

[ljmf00]

The problem in this issue with comment id of 450197752 was solved. I used os.path.dirname(os.path.abspath(__file__)) on fetch from git_lfs pip package.

@swistakm @AlexMAS, the workarroud work on last version using just the current working directory of conf.py .

---

Reference:

• [1] Git LFS support? #1846 (comment)
• [2] aurorafossorg/learn@9306ea8

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

Hi everybody!

I'm re-opening and re-purposing this issue to have some actionable things.

I read the whole conversation and it seems we need different things at this point:

• documentation guide with a workaround solution to point users while this gets implemented
• research a little more about what are the proper steps (and in which order) to follow when the repository uses git-lfs

This is getting more common on VCS services and I think we should start thinking on supporting this for real

I created a small example using a file that is tracked via git lfs at https://test-builds.readthedocs.io/en/git-lfs/ and the image is not shown. The plain text content is displayed:

Installing git-lfs and getting the real files locally, I can see the image properly:

[john-arcus]

Thanks for re-opening this @humitos. I'm looking forward to seeing how it develops. :)

[humitos]

I should have mentioned that the conf.py file shown in my example has 5 calls to os.system that do the trick to support git-lfs in a hacky way that you can use while we implement this properly.

# docs/conf.py

# Workaround to install and execute git-lfs on Read the Docs
import os
if not os.path.exists('./git-lfs'):
    os.system('wget https://github.com/git-lfs/git-lfs/releases/download/v2.7.1/git-lfs-linux-amd64-v2.7.1.tar.gz')
    os.system('tar xvfz git-lfs-linux-amd64-v2.7.1.tar.gz')
    os.system('./git-lfs install')  # make lfs available in current repository
    os.system('./git-lfs fetch')  # download content from remote
    os.system('./git-lfs checkout')  # make local files to have the real content on them

EDIT: added the check os.path.exists to avoid re-downloading and installing the file if it already exist

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

We do not plan to support this soon. We can keep this issue closed for now.

[forikal-uk]

Hi, thanks for letting us know.

Is there a published, documented, workaround available for those who have git-lfs installed.

I am wondering how to go about setting the git-lfs tracking configuration to, say, let everything with a certain file suffix stay as binary instead of being tracked under git-lfs.

Or maybe, some method to make all items inside the _static folder that are tracked under git-lfs to, now, not be tracked under git-lfs.

If there is no workaround, I will attempt to find one and update once discovered.

[JamesRandom]

I have tried the process described by @humitos but I get an authentication error:

git-lfs
install.sh
Updated git hooks.
Git LFS initialized.
fetch: Fetching reference HEAD
batch request: Host key verification failed.: exit status 255
error: failed to fetch some objects from 'https://github.com/JamesRandom/xxxxx.git/info/lfs'

Both git and git-lfs seem to be using the same endpoint:

$ git remote -v
origin	https://github.com/JamesRandom/xxxxxxx.git (fetch)
...
$ git lfs env
git-lfs/2.10.0 (GitHub; darwin amd64; go 1.13.8)
git version 2.27.0

Endpoint=https://github.com/JamesRandom/xxxxxxx.git/info/lfs (auth=basic)
...

[Borda]

Hi, I am facing a similar issue, so curious if there will any official support for LFS?
well, it seems to be working, so maybe just mention it in the docs... 🐰

[astrojuanlu]

Hi @Borda - when you say "it seems to be working", do you refer to @humitos hack here? #1846 (comment)

If that's the case, it's still a hack, so in my opinion we should not add docs for it.

[Borda]

Hi @Borda - when you say "it seems to be working", do you refer to @humitos hack here? #1846 (comment)
If that's the case, it's still a hack, so in my opinion we should not add docs for it.

in fact none of that above, we are linking this repo which has enabled ipynb as LFS to this repo as a submodule and all notebooks seem to be properly rendered - which would not happen if LFS or submodule won't work...

[dokempf]

FWIW, I have packaged the workaround described in this issue in a currently working state into a Sphinx extension: https://github.com/ssciwr/sphinx_lfs_content

[JamesRandom]

Thanks for letting me know (and doing this!) I will try this out. It will certainly clean things up a bit.

…

On 5 Jul 2021, at 21:49, Dominic Kempf ***@***.***> wrote: FWIW, I have packaged the workaround described in this issue in a currently working state into a Sphinx extension: https://github.com/ssciwr/sphinx_lfs_content <https://github.com/ssciwr/sphinx_lfs_content> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1846 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2SZP775W4R5H27KYN3M6TTWIECZANCNFSM4BVWIDUQ>.

[sjalloq]

@JamesRandom Did you ever sort out your authentication issue? We're using BitBucket and having just installed the sphinx_lfs_content extension am getting a similar Host key verification failed error.

[JamesRandom]

I had to add username and a GitHub access token to the Git URL: https://username:[email protected]/JamesRandom/xxxxxx.git
Not ideal, but it seems to be the only way to get it to work

[astrojuanlu]

I couldn't find any projects using @dokempf extension https://libraries.io/pypi/sphinx-lfs-content/ on GitHub, Google, or libraries.io. @Borda @JamesRandom @sjalloq @forikal-uk could you please give sphinx-lfs-content a try and report back if it worked? We would be happy to promote it more in our FAQ if we gather some social proof.

[sjalloq]

@astrojuanlu I couldn't get it working but I think that was an auth issue rather than as issue with the extension. I don't have enough time to try and make it work so backed out the use of LFS.

[dokempf]

@astrojuanlu I use this in some of my personal projects, although I have recently reduced my LFS usage due to the low bandwidth quota by GitHub. I have never faced the auth issue described here, but I would welcome an issue on the extension repo https://github.com/ssciwr/sphinx_lfs_content that gives a bit of context when exactly that happens.

[JamesRandom]

I tried it but it didn't work properly. Can't remember exactly what went wrong, but it seemed to behave as if git-lfs hadn't't been installed — despite the fact the output showed it was installed. Almost like some sort of race condition. I didn't have time to look into it further at the time. I'll try to have another look after our pending sw release

[benjaoming]

Update: Read the Docs supports Git LFS by extending the build process.

It's described in the docs: https://docs.readthedocs.io/en/stable/build-customization.html#support-git-lfs-large-file-storage