Optional use of shallow clone for git repos

[NickAb]

It would be great if there was an option to enable shallow clone for repositories. If set, then RtD should use shallow clone (--depth 1 option of git clone) instead of full clone if possible.
This will result in huge disk space saving for projects with large history.

E.g., for one of our projects /www/readthedocs.org/user_builds/{project}/checkouts/latest folder takes up to 400 Mb, when cloned with --depth 1, it takes up only 12 Mb.

[agjohnson]

This would definitely be worth looking into, this might be able to take a chunk of storage. I haven't looked into this at all, be the --depth option would need to be supported by git fetch, as we don't actually run a git clone except on initial import, afaik.

[ericholscher]

Looks like we can now support this with the --no-single-branch option on git clone. There doesn't appear to be a similar option for git fetch, though, so we might not be able to reliably update the list of branches while keeping a shallow clone on disk.

[ericholscher]

If someone could test that this is possible, it would be a pretty sizable win in terms of disk space needed.

[NickAb]

What exactly do you mean by update the list of branches?

If it is simple obtaining a list of all branches and tags, then it can be done using git ls-remote command.

git ls-remote --heads --tags https://github.com/rtfd/readthedocs.org.git

You don't even need to clone repository to do so.

See https://www.kernel.org/pub/software/scm/git/docs/git-ls-remote.html

[ericholscher]

travis uses --depth=50

[stsewd]

We have the depth option in the fetch command too https://git-scm.com/docs/git-fetch/2.17.0#git-fetch---depthltdepthgt, I'm testing it, looks like is something that could work for every project. Any reason to not do that on every project? Also, depth=1 looks fine.

[ericholscher]

Do we still get branch/tag info with a shallow clone?

[stsewd]

Yeah, we do, actually I just have a POC for this. So, we could add a feature flag to test and force everyone to do a shallow clone, or make it an option for the users. The only thing I can see is where a custom extension needs info from other commits I guess?