clone_from progress is broken in newer version

[Dreamsorcerer]

To use a workaround for another bug, I had to first update GitPython. After updating with pip3 install --upgrade gitpython, which installed 2.1.3, the progress feature no longer functions correctly.

With this code:

import git
def u(*args):
 print(args, flush=True)
git.Repo.clone_from("[email protected]:....", ".", u)

While running it on a fairly large repository, I get the first few updates, upto:
(10, 16118.0, 16118.0, '') (which means compressing has completed)

But, then the next stage when it is downloading everything, which can take about 15 mins, there is no feedback until it has completed that stage, then all the missing feedback is sent at once, as if the feedback has gathered in a buffer and not sent until the current stage is complete.

[Dreamsorcerer]

OK, I've narrowed it down through trial and error.
Working correctly: 2.0.8
Broken: 2.0.9

[Byron]

Thanks for narrowing it down!
I had a closer look and found this commit which simplified the code to just-in-time parse lines from stdout and/or stderr.

To me the code looks good - the main thread blocks while lines are read from streams as they become available. Apparently this line iteration doesn't work as expected for you.

Can you provide more information about your environment as it may relate to the issue:

• the exact python version
• the OS

Thanks a lot

[Dreamsorcerer]

Python 3.5.2
Ubuntu 16.04

[Dreamsorcerer]

Any progress on this? Have you been able to reproduce it? It should be trivial to test in a vagrant box or similar.

This is causing us some fairly bad UX issues in our project at the moment.

Full steps to reproduce:
Install Vagrant.
Save the attached Vagrantfile into a new folder (you'll need to remove the .txt extension thanks to Github's dumb mimetype detection).
Vagrantfile.txt

Run vagrant up && vagrant ssh
Inside the vagrant box:

sudo apt update
sudo apt install python3-pip -y
pip3 install gitpython
ssh-keyscan github.com > ~/.ssh/known_hosts
python3

import git
def u(*args):
 print(args, flush=True)

git.Repo.clone_from("https://github.com/gitpython-developers/GitPython.git", "test", u)

[RenanMsV]

Stills broken. Works in 2.0.8.
I'm using standard Progress class.

class Progress(RemoteProgress):
    def update(self, op_code, cur_count, max_count=None, message=''):
        print('Downloading: (==== {} ====)\r'.format(message))

and then calling it

Repo.clone_from("git://github.com/{}".format(link.split('.com/')[1]), PATH_NAME, progress=Progress())

Python 3.4.4.1Qt5 from WinPython
Windows 8.1 64bits

[Byron]

Could you try it again with the latest version?

[RenanMsV]

kind of working 2.1.7
working
print('Downloading: (==== {} ====)'.format(message)
not working
print('Downloading: (==== {} ====)'.format(message), end='\r')

[Dreamsorcerer]

Nope, no change. Followed my exact steps mentioned a couple of comments ago, and got exactly the same problem. GitPython==2.1.7
#604 (comment)

[Minipada]

I have exactly the same, seems similar to #444. Using 2.1.7

[Dreamsorcerer]

Confirmed, this is the same problem, and is fixed by following @mrozekma's comment:
I fixed it locally by changing git.remote.add_progress to also set kwargs['universal_newlines'] = True, and changing git.base.Repo._clone to pass decode_streams=False to handle_process_output

[Dreamsorcerer]

I made a tweak to those instructions to fix build issues in the referenced pull request, though there are still a couple of build errors which mean nothing to me.

[Dreamsorcerer]

Actually, it looks like the remaining build issues also exist in the main branch. So, please review and merge.

[Byron]

@Dreamsorcerer I am closing this issue as its fix will be contained in the upcoming release.

[jjloesch]

Still broken in 2.1.8
Reverting to last known working version 2.0.8

[Dreamsorcerer]

Are you following my exact instructions in my previous comment #604 (comment)? I've just tested it again following those exact instructions, and it it is working correctly. If you are encountering an issue in a different situation, then it is a different bug, please file a new one.