Skip to content

some files are locked until python interpreter exits #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
topic2k opened this issue Feb 21, 2016 · 4 comments
Closed

some files are locked until python interpreter exits #596

topic2k opened this issue Feb 21, 2016 · 4 comments

Comments

@topic2k
Copy link

topic2k commented Feb 21, 2016

Hello,
i make a temporary bare repository, add a remote and fetch references. Then i want to delete the temporary folder, but it fails because of locked files. I've attached the following script to demonstrate what i mean:

# coding=utf-8

import tempfile
import pygit2
import shutil
import os
import stat


def GetBranches(repoUrl):
    repo = pygit2.init_repository(tmpDir, bare=True, origin_url=repoUrl)
    remote = repo.remotes.create('temprepo', repoUrl)
    remote.fetch()
    refs = repo.listall_references()
    return refs[:]

def DeleteDir(path):
    def remove_readonly(func, path, excinfo):
        os.chmod(path, stat.S_IWRITE)
        func(path)
    if os.path.exists(path):
        shutil.rmtree(path, ignore_errors=False, onerror=remove_readonly)


tmpDir = tempfile.mkdtemp()
print "temp dir:", tmpDir
print "references:", GetBranches('https://github.com/libgit2/pygit2.git')
DeleteDir(tmpDir)
print "temp dir deleted"

filelocktest.zip

@carlosmn
Copy link
Member

The repository's object db keeps a few files open (mapped into memory) so it doesn't have to keep opening and closing files whenever it needs to look up an object. But this is not tied to the python interpreter, but to the lifetime of the repository. If you force collecting the repository, those file handles will be closed.

You haven't said which files are the problem, what OS you're on or what the specific error message says, so I'm assuming you're on Windows as that is the OS where we would lock files by keeping them open. If this is not the case, the please provide that information.

@topic2k
Copy link
Author

topic2k commented Feb 24, 2016

Sorrry for not being detailed enough. I'm on Windows10 64Bit with Python 3.5 (32bit) or Python 2.7 Stackless (32bit).

The files are in tempdir\objects\pack\, the filenames start with pack- and have the extensions .pack and .idx.

If you force collecting the repository, those file handles will be closed.

I'm new to git/pygit2, can you explain me what do you mean by force collecting the repository?

@carlosmn
Copy link
Member

I'm new to git/pygit2, can you explain me what do you mean by force collecting the repository?

That's a python/garbage-collection thing. The repository object isn't necessarily freed immediately when you stop using it. The interpreter may decide to do it at a later time if it doesn't think there is a lot of memory pressure. del repo should be enough in most cases to get rid of the object. But as always with garbage-collected languages, you can only make sure by going into the language's GC structures.

@topic2k
Copy link
Author

topic2k commented Feb 25, 2016

Ok, thank you for taken the time and the explanation.

@topic2k topic2k closed this as completed Feb 25, 2016
n-dusan added a commit to openlawlibrary/taf that referenced this issue Jul 18, 2022
* Basing on #243, we are having some difficulty with pygit2 Repository file
  handlers being open. in `pygit.py` we instantiate the Repository
  class, which holds references to `pack` and `.idx` git objects and
  keeps files open. If taf and cleanup is attempted from same subprocess,
  Repository handler won't be deleted, and cleanup error occurs. Calling
  manual garbage collection did not help. This
  seems to be a known issue throughout the git community, see [1] and
  [2].

  [1] - libgit2/pygit2#596
  [2] - gitpython-developers/GitPython#546

  Several workarounds were proposed by the community, but it seems that
  we can't fully replicate those workarounds. In [2] they attempt to
  workaround the issue by deleting git cache. That works for them
  because their git implementation has file handlers open in pure
  python, while we only do subprocess git calls.

  Our TODO: if these  windows specific error issues keep persisting, figure out how to
  garbage collect pygit2 Repository handler correctly.

  For now settle on error handling, as this issue is
  specific to a subprocess call that calls taf/updater. Though we should
  keep track of progress of these known bugs, so we can fix them if they
  get addressed by pygit2 or gitpython
n-dusan added a commit to openlawlibrary/taf that referenced this issue Jul 18, 2022
* Basing on #243, we are having some difficulty with pygit2 Repository file
  handlers being open. in `pygit.py` we instantiate the Repository
  class, which holds references to `pack` and `.idx` git objects and
  keeps files open. If taf and cleanup is attempted from same subprocess,
  Repository handler won't be deleted, and cleanup error occurs. Calling
  manual garbage collection did not help. This
  seems to be a known issue throughout the git community, see [1] and
  [2].

  [1] - libgit2/pygit2#596
  [2] - gitpython-developers/GitPython#546

  Several workarounds were proposed by the community, but it seems that
  we can't fully replicate those workarounds. In [2] they attempt to
  workaround the issue by deleting git cache. That works for them
  because their git implementation has file handlers open in pure
  python, while we only do subprocess git calls.

  Our TODO: if these  windows specific error issues keep persisting, figure out how to
  garbage collect pygit2 Repository handler correctly.

  For now settle on error handling, as this issue is
  specific to a subprocess call that calls taf/updater. Though we should
  keep track of progress of these known bugs, so we can fix them if they
  get addressed by pygit2 or gitpython
n-dusan added a commit to openlawlibrary/taf that referenced this issue Jul 19, 2022
* Basing on #243, we are having some difficulty with pygit2 Repository file
  handlers being open. in `pygit.py` we instantiate the Repository
  class, which holds references to `pack` and `.idx` git objects and
  keeps files open. If taf and cleanup is attempted from same subprocess,
  Repository handler won't be deleted, and cleanup error occurs. Calling
  manual garbage collection did not help. This
  seems to be a known issue throughout the git community, see [1] and
  [2].

  [1] - libgit2/pygit2#596
  [2] - gitpython-developers/GitPython#546

  Several workarounds were proposed by the community, but it seems that
  we can't fully replicate those workarounds. In [2] they attempt to
  workaround the issue by deleting git cache. That works for them
  because their git implementation has file handlers open in pure
  python, while we only do subprocess git calls.

  Our TODO: if these  windows specific error issues keep persisting, figure out how to
  garbage collect pygit2 Repository handler correctly.

  For now settle on error handling, as this issue is
  specific to a subprocess call that calls taf/updater. Though we should
  keep track of progress of these known bugs, so we can fix them if they
  get addressed by pygit2 or gitpython

chore: changelog
n-dusan added a commit to openlawlibrary/taf that referenced this issue Jul 19, 2022
* Basing on #243, we are having some difficulty with pygit2 Repository file
  handlers being open. in `pygit.py` we instantiate the Repository
  class, which holds references to `pack` and `.idx` git objects and
  keeps files open. If taf and cleanup is attempted from same subprocess,
  Repository handler won't be deleted, and cleanup error occurs. Calling
  manual garbage collection did not help. This
  seems to be a known issue throughout the git community, see [1] and
  [2].

  [1] - libgit2/pygit2#596
  [2] - gitpython-developers/GitPython#546

  Several workarounds were proposed by the community, but it seems that
  we can't fully replicate those workarounds. In [2] they attempt to
  workaround the issue by deleting git cache. That works for them
  because their git implementation has file handlers open in pure
  python, while we only do subprocess git calls.

  Our TODO: if these  windows specific error issues keep persisting, figure out how to
  garbage collect pygit2 Repository handler correctly.

  For now settle on error handling, as this issue is
  specific to a subprocess call that calls taf/updater. Though we should
  keep track of progress of these known bugs, so we can fix them if they
  get addressed by pygit2 or gitpython

chore: changelog
renatav pushed a commit to openlawlibrary/taf that referenced this issue Jul 20, 2022
* fix: reraise original exception and windows specific cleanup errors

* Basing on #243, we are having some difficulty with pygit2 Repository file
  handlers being open. in `pygit.py` we instantiate the Repository
  class, which holds references to `pack` and `.idx` git objects and
  keeps files open. If taf and cleanup is attempted from same subprocess,
  Repository handler won't be deleted, and cleanup error occurs. Calling
  manual garbage collection did not help. This
  seems to be a known issue throughout the git community, see [1] and
  [2].

  [1] - libgit2/pygit2#596
  [2] - gitpython-developers/GitPython#546

  Several workarounds were proposed by the community, but it seems that
  we can't fully replicate those workarounds. In [2] they attempt to
  workaround the issue by deleting git cache. That works for them
  because their git implementation has file handlers open in pure
  python, while we only do subprocess git calls.

  Our TODO: if these  windows specific error issues keep persisting, figure out how to
  garbage collect pygit2 Repository handler correctly.

  For now settle on error handling, as this issue is
  specific to a subprocess call that calls taf/updater. Though we should
  keep track of progress of these known bugs, so we can fix them if they
  get addressed by pygit2 or gitpython

chore: changelog

* fix: move clean up handling to `on_rm_error`

* Cause of Windows Error permissions is in `on_rm_error`, which can
  trigger and fail in multiple places in updater.py. Since we only put
  out a warning for that, log the warning when calling os.unlink
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants