File handles left open

[afgane]

Hello,
It seems that as soon as I access any of the fields within r.head.commit, a set of file handles open and they don't close until the parent process exits. Specifically for me, this prevents me from unmounting a file system.

Here's an example of what I'm doing:

from git import Repo
r = Repo('/mnt/galaxy/galaxy-app')
r.head.commit.hexsha

As soon as a commit object field is accessed, the following happens on the file system:

$ sudo lsof +D /mnt/galaxy
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
ipython 29619 root  mem    REG 202,80 54593196 33598181 /mnt/galaxy/galaxy-app/.git/objects/pack/pack-c5d4b5abfd9295af7aea34651359d8b056c6a1af.pack
ipython 29619 root  mem    REG 202,80  4668532 33598182 /mnt/galaxy/galaxy-app/.git/objects/pack/pack-c5d4b5abfd9295af7aea34651359d8b056c6a1af.idx
ipython 29619 root    7r   REG 202,80 54593196 33598181 /mnt/galaxy/galaxy-app/.git/objects/pack/pack-c5d4b5abfd9295af7aea34651359d8b056c6a1af.pack
ipython 29619 root    8r   REG 202,80  4668532 33598182 /mnt/galaxy/galaxy-app/.git/objects/pack/pack-c5d4b5abfd9295af7aea34651359d8b056c6a1af.idx

If there's a way around this, I'd appreciate some advice.
Thanks.

This is on Ubuntu 14.04 with GitPython 1.0.1 and git 1.9.1.

[Byron]

Thank you. Which Python version are you using?
In python 3, destructors are usually not called, even if the Repo instance is deleted or goes out of scope.

[afgane]

This is on Python 2.7.6

[Byron]

I didn't reproduce it yet, but can imagine this truly is the case.

What happens if you delete your repo instance, e.g. with del repo? This should work on python 2.7 (but I didn't try it yet).
You could also try repo.odb.release() to forcefully free all handles.

If in any of the tests above the handles remain opened, this is indeed a critical and possibly easily reproducible bug.

[afgane]

del repo unfortunately does nothing. This was the first thing I tried yesterday.

repo.odb.release() does not exist? The following are the only members odb object shows:

In [31]: r.odb.
r.odb.LooseDBCls                   r.odb.databases                    r.odb.loose_dir                    r.odb.root_path                    r.odb.store
r.odb.PackDBCls                    r.odb.db_path                      r.odb.ostream                      r.odb.set_ostream                  r.odb.stream
r.odb.ReferenceDBCls               r.odb.has_object                   r.odb.packs_dir                    r.odb.sha_iter                     r.odb.update_cache
r.odb.alternates_dir               r.odb.info                         r.odb.partial_to_complete_sha_hex  r.odb.size

I've explored those but no progress.

Thanks for looking into this.

[Byron]

Apparently I was wrong about the release() method on the pure-python implementation of the Git database. However, there is a git-command based implementation which uses git cat-file to obtain object information. Whenever such a database is destroyed, the persistent commands will be shut down.

To use it, just instantiate your Repo instance as follows:

import git
r = git.Repo(odbt=git.GitCmdObjectDB)
r.commit().message
# now lsof shows some pack files
del r
# now they are gone

It's a bit sad that there seems to be no way to trigger this kind of resource release in the pure-python implementation, and considering that GitPython is in maintenance mode, I believe this is not going to improve.

Please let me know if the suggested workaround did indeed work for you.
Thanks

[afgane]

This worked from the command prompt. I'll try it in the app a bit later.

Thank you.

[Byron]

Great to hear ! Feel free to close this issue if it worked in the app as well.

[Byron]

You can watch the development stream on youtube.

GitPython #1 [investigate resource leak]

[afgane]

It's working in the larger context as well - thank you again for tracking this down and providing a workaround. (And thanks for capturing the process in the video.)