Push tags=True blocks on github

[kylegibson]

As of last week, we could push tags using gitpython without any issue. Now it blocks forever. Neither git nor gitpython has been upgraded. This problem is occurring for multiple developers.

git versions: 1.8.5.2, 1.7.0.4

    remote.push(tags=True)
  File "/home/kyle/.python/local/lib/python2.7/site-packages/git/remote.py", line 627, in push
    return self._get_push_info(proc, progress or RemoteProgress())
  File "/home/kyle/.python/local/lib/python2.7/site-packages/git/remote.py", line 552, in _get_push_info
    digest_process_messages(proc.stderr, progress)
  File "/home/kyle/.python/local/lib/python2.7/site-packages/git/remote.py", line 48, in digest_process_messages
    char = fh.read(1)
  File "/home/kyle/.python/local/lib/python2.7/site-packages/async/__init__.py", line 21, in thread_interrupt_handler
    prev_handler(signum, frame)
KeyboardInterrupt

[jlward]

I did some digging.

Basically behind the scenes when you call remote.push(tags=True) the command that is hanging is this guy:

proc = Popen(
    [
        'git',
        'push',
        '--porcelain',
        '--tags',
        'origin',
    ],
    cwd='/path/to/head/',
    stdin=None,
    stderr=PIPE,
    stdout=PIPE,
)

Basically in our repo if you go into a shell an call that command followed by proc.wait() it will never finish. However if you drop the PIPEs then it does finish. So the command would look something like this:

proc = Popen(
    [
        'git',
        'push',
        '--porcelain',
        '--tags',
        'origin',
    ],
    cwd='/path/to/head/',
    stdin=None,
    #stderr=PIPE,
    #stdout=PIPE,
)

A few things to note. I tried running push(tags=True) from a few different repos to see if it was repo specific.

1. Repo with no tags - works
2. Repo with few tags (less than 10) - works
3. Repo with a lot of tags (over 1200) - does not work (hangs)

This only recently started happening. We currently have 1254 tags and it started sometime in the last 10-15ish tags.

[jlward]

It should also be noted that the tags do actually get pushed, however the push(tags=True) hangs.

[winhamwr]

I think the issue is that with lots of tags, it fills up the buffer, so wait will never return. See this SO question for a reference.

Here is another relevant solution.

[Byron]

Thanks for the digging.

Looking at the latest 0.3@f573b78 at this spot, it seems the default code path will use proc.communicate(), which in theory will work correctly.
The alternative code path will clearly be prone to blocking if there is a lot of output in stderr.

Checking the communicate() code of python 2.7.6 revealed that different implementations exist for windows and posix, both posix versions use either select() or poll(), which is assured to be absolutely non-blocking.

In this particular issue however, we are looking at git/remote.py obviously first processes stderr output for progress, taking on the actual output later. If stdout fills while progress is being read from stderr, the git process is locking up.

Citing from the comment right above git/remote.py:552:

        # read progress information from stderr
        # we hope stdout can hold all the data, it should ...
        # read the lines manually as it will use carriage returns between the messages
        # to override the previous one. This is why we read the bytes manually
        digest_process_messages(proc.stderr, progress)

That programmer with the multiple personalty issue was me, as a git blame revealed :).

To my mind, the entire push and pull code needs revision for this reason and others. For example, output parsing will fail if there are unexpected tokens.

[msabramo]

#184 seems to help for me with a simple sample program that pushes a lot of tags.

[msabramo]

Can others verify whether #184 fixed the problem for them?

[Byron]

I will implement poll()-based parsing of stderr/stdout in case of fetch/pull/push now. The previous fixed was merged into a now obsolete branch, and would cost the real-time progress information that I was keen to have.

[Byron]

The chosen implementation uses poll() if possible, but resorts to brute-force threads for 'pumping' lines otherwise. The only reason threads are less desirable is that they have more synchronisation overhead in general, and are note reused.
Using a threadpool could help alleviate this overhead.

Please also note that to my surprise, poll() is not available on py2 on OSX either, but is available on py3 on OSX.
On windows, there is no poll() either, and the existing select() call only works on sockets. Therefore using threads was the only viable alternate code-path that would work everywhere.
Also note that I managed to have the thread implementation occasionally fail on me, which could hint at some sync issue still present.