Python 3.5 failing with AttributeError: module 'os' has no attribute 'PathLike'

[c4xuxo]

This morning our builds started to fail as soon we moved to 3.1.15.

I seems that 3.1.15 makes use of os.PathLike that was introduced in python 3.6.

From README.md seems that Python 3.5 is still supported.

REQUIREMENTS
GitPython needs the git executable to be installed on the system and available in your PATH for most operations. If it is not in your PATH, you can help GitPython find it by setting the GIT_PYTHON_GIT_EXECUTABLE=<path/to/git> environment variable.

Git (1.7.x or newer)
Python >= 3.5

class os.PathLike
An abstract base class for objects representing a file system path, e.g. pathlib.PurePath.

New in version 3.6.

[Byron]

Thanks for reporting.

That's interesting as I would have thought that our CI can pick this up.
Can it be that PathLike was backported and a more recent version of python 3.5 supports it?

In any case, since python 3.5 is EOL and this PR exists I would probably prefer to cancel python 3.5 support rather than undoing the use of PathLike.

In order to get this fixed, I could yank the offending version. However, first I would love to know why our CI seems to be fine with it.

Thanks for your help

[c4xuxo]

We are using default in Azure Pipelines for python3, that seems to be 3.5 for us.

We just patched to use python 3.6, so we are no longer impacted by 3.5.

[Byron]

Good to hear.

Since there is no further information on how to reproduce this issue, as well as no error messages for others to find, I think it's ok to close this issue until further information is presented to help understanding what's going on.

Thanks for your understanding.

[muggenhor]

FYI: I think this issue needs to be re-opened. It's still a problem with Python 3.6 too, just a different exception. (See below for details).

That's interesting as I would have thought that our CI can pick this up.
Can it be that PathLike was backported and a more recent version of python 3.5 supports it?

Doesn't appear to be the case:

$ docker run -it --rm python:3.5-slim
Python 3.5.10 (default, Sep 10 2020, 18:40:42) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from os import PathLike
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'PathLike'
>>> 

In any case, since python 3.5 is EOL and this PR exists I would probably prefer to cancel python 3.5 support rather than undoing the use of PathLike.

This would seem like something unexpected to do in a patch release. Also, I find this to be broken for more than 3.5...

In order to get this fixed, I could yank the offending version. However, first I would love to know why our CI seems to be fine with it.

I think it's because of this in git/types.py:

PathLike = Union[str, 'os.PathLike[str]']

For some reason that was written as a forward reference instead of a direct reference. I don't see any need for that here. This would cause evaluation of the type expression to be delayed until it's evaluated:

$ docker run -it --rm python:3.5-slim
Python 3.5.10 (default, Sep 10 2020, 18:40:42) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Union
>>> import os

>>> PathLike = Union[str, os.PathLike[str]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'os' has no attribute 'PathLike'

>>> PathLike = Union[str, 'os.PathLike[str]']

>>> PathLike.__args__[1]._eval_type(globals(), locals())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/typing.py", line 245, in _eval_type
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
AttributeError: module 'os' has no attribute 'PathLike'
>>> 

As you can see this fails immediately without using a forward reference (string version of the type expression) but waits to fail until evaluated by typing.py when using a forward reference.

With Python 3.6 we get some improvement, as it's no longer complaining about PathLike not existing. Instead it complains about that not being subscriptable... (I'm skipping the forward reference here for brevity, but it's the same when still using that, it just hides the problem):

$ docker run -it --rm python:3.6-slim
Python 3.6.12 (default, Dec 11 2020, 15:17:39) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Union
>>> import os
>>> PathLike = Union[str, os.PathLike[str]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'ABCMeta' object is not subscriptable

This exact error continues up to and including 3.8, it's only starting from 3.9 that both forms succeed:

$ docker run -it --rm python:3.9-slim
Python 3.9.1 (default, Jan 12 2021, 16:56:42) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing import Union
>>> import os
>>> PathLike = Union[str, os.PathLike[str]]
>>> PathLike = Union[str, 'os.PathLike[str]']
>>> PathLike.__args__[1]._evaluate(globals(), locals(), frozenset())
os.PathLike[str]
>>> 

The reason why none of this fails is because forward references will only get evaluated by tools that actually use the type annotations. And while I see a config file for mypy, which is such a tool, I cannot find any place where mypy actually gets called from CI...

[muggenhor]

Summary:

• for 3.5 it's the non-existence of os.PathLike that's a problem.
• for 3.6, 3.7 and 3.8 it's the fact that it's not subscriptable (with str) as os.PathLike[str].

Unfortunately I cannot find any documentation about the subscript form for that type, so I wouldn't know directly how to fix it for 3.6 onwards, let alone 3.5...

I'm not even sure exactly what the intent of @Yobmod is with using the subscript form... I suggest poking him to look at this and fix it, and if that takes too long replacing the 'os.PathLike[str]' expression with TBD (with a comment) in the interim.

[c4xuxo]

In git/compat.py
from git.types import TBD

in git/types.py

from typing import Union, Any


TBD = Any
PathLike = Union[str, 'os.PathLike[str]']

os.PathLike does not exist in python 3.5..

So importing types should create this error in 3.5.

[Byron]

Thanks a lot, @c4xuxo and @muggenhor. I think this issue neatly shows the outrageous level of incompetency I have reached maintaining this project which by now relies on blind trust in CI and luck 🤦‍♂️.

Without digging into the reasons which to me are very clear, you may read this comment as a hint at my openness to pass on the torch of maintenance.

In the mean time, things like these might keep happening unless I pace myself and prefer follow my very own rules over incorporating contributions.

[Yobmod]

Hi all,

Thanks for the fix of my mistake. I've been away on work, so not been able to get online until now.
I had expected the CI to catch import problems like that, with any test that called any function using pathlike triggering it. My fault for putting it as a forward refererence maybe.

This solution is better than i would have done, which is just change os.pathlike to pathlib.Path (introduced 3.5) and see if mypy agreed.
Using the subscriptable form from 3.9+ seemed like the correct type, but i'm not sure the subscript really adds much value (it was suggested by a static checker). So it could be just os.Pathlike for 3.6+

I'm hesitant to suggest adding mypy to the CI, as it would force all contributers to type everything or add type ignore comments when mypy cannot check it.

Thanks again!

[Byron]

Thanks for chiming in @Yobmod.

The continuation of this conversation is here: #1223 (comment)