Celery: ZoneInfoNotFoundError on development instance

[agjohnson]

I'm getting the following error on my local environment, but didn't get too deep in debugging it yet. This started after a rebuild of my local environment build image.

build_1        | Traceback (most recent call last):
build_1        |   File "/usr/lib/python3.10/zoneinfo/_common.py", line 12, in load_tzdata
build_1        |     return importlib.resources.open_binary(package_name, resource_name)
build_1        |   File "/usr/lib/python3.10/importlib/resources.py", line 43, in open_binary
build_1        |     package = _common.get_package(package)
build_1        |   File "/usr/lib/python3.10/importlib/_common.py", line 66, in get_package
build_1        |     resolved = resolve(package)
build_1        |   File "/usr/lib/python3.10/importlib/_common.py", line 57, in resolve
build_1        |     return cand if isinstance(cand, types.ModuleType) else importlib.import_module(cand)
build_1        |   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
build_1        |     return _bootstrap._gcd_import(name[level:], package, level)
build_1        |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
build_1        |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
build_1        |   File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
build_1        |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
build_1        |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
build_1        |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
build_1        |   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
build_1        | ModuleNotFoundError: No module named 'tzdata'
build_1        | 
build_1        | During handling of the above exception, another exception occurred:
build_1        | 
build_1        | Traceback (most recent call last):
build_1        |   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
build_1        |     return _run_code(code, main_globals, None,
build_1        |   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
build_1        |     exec(code, run_globals)
build_1        |   File "/usr/local/lib/python3.10/dist-packages/celery/__main__.py", line 19, in <module>
build_1        |     main()
build_1        |   File "/usr/local/lib/python3.10/dist-packages/celery/__main__.py", line 14, in main
build_1        |     from celery.bin.celery import main as _main
build_1        |   File "/usr/local/lib/python3.10/dist-packages/celery/bin/celery.py", line 34, in <module>
build_1        |     from celery.bin.worker import worker
build_1        |   File "/usr/local/lib/python3.10/dist-packages/celery/bin/worker.py", line 14, in <module>
build_1        |     from celery.concurrency.base import BasePool
build_1        |   File "/usr/local/lib/python3.10/dist-packages/celery/concurrency/base.py", line 12, in <module>
build_1        |     from celery.utils import timer2
build_1        |   File "/usr/local/lib/python3.10/dist-packages/celery/utils/timer2.py", line 14, in <module>
build_1        |     from kombu.asynchronous.timer import Entry
build_1        |   File "/usr/local/lib/python3.10/dist-packages/kombu/asynchronous/__init__.py", line 7, in <module>
build_1        |     from .hub import Hub, get_event_loop, set_event_loop
build_1        |   File "/usr/local/lib/python3.10/dist-packages/kombu/asynchronous/hub.py", line 19, in <module>
build_1        |     from .timer import Timer
build_1        |   File "/usr/local/lib/python3.10/dist-packages/kombu/asynchronous/timer.py", line 32, in <module>
build_1        |     EPOCH = datetime.utcfromtimestamp(0).replace(tzinfo=ZoneInfo("UTC"))
build_1        |   File "/usr/lib/python3.10/zoneinfo/_common.py", line 24, in load_tzdata
build_1        |     raise ZoneInfoNotFoundError(f"No time zone found with key {key}")
build_1        | zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key UTC'

Looks like either a missing package or a different TZ is needed in our settings.

[agjohnson]

I tested by installing tzdata at common.sh entrypoint and it did get rid of that error.

[humitos]

I had the same issue and I fixed locally in the same way. I don't understand why this doesn't happen in production, tho. Maybe we have a dependency there that installs tzdata?

[benjaoming]

I don't understand why this doesn't happen in production, tho.

Could there be differences between the production and docker images? Maybe tzdata is available or maybe the fallback that fails in docker works in production?

The fallback w/ ZoneInfo can (I believe) use host system tz info.

Try this in production?

# see if tzdata exists
import tzdata
print (tzdata.__file__)

# Try the fallback
from zoneinfo import ZoneInfo
ZoneInfo("UTC")

tzdata is not in any requirements. It would seem like a pretty basic package to include.

tzdata isn't part of web nor celery for me, neither:

inv docker.shell -c celery  
root@df8283fa9391:/usr/src/app/checkouts/readthedocs.org# python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import celery
>>> import tzdata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tzdata'

>>> # Try the fallback
>>> from zoneinfo import ZoneInfo
>>> ZoneInfo("UTC")
Traceback (most recent call last):
  File "/usr/lib/python3.10/zoneinfo/_common.py", line 12, in load_tzdata
    return importlib.resources.open_binary(package_name, resource_name)
  File "/usr/lib/python3.10/importlib/resources.py", line 43, in open_binary
    package = _common.get_package(package)
  File "/usr/lib/python3.10/importlib/_common.py", line 66, in get_package
    resolved = resolve(package)
  File "/usr/lib/python3.10/importlib/_common.py", line 57, in resolve
    return cand if isinstance(cand, types.ModuleType) else importlib.import_module(cand)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tzdata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/zoneinfo/_common.py", line 24, in load_tzdata
    raise ZoneInfoNotFoundError(f"No time zone found with key {key}")
zoneinfo._common.ZoneInfoNotFoundError: 'No time zone found with key UTC'

[humitos]

Try this in production?

In [1]: import tzdata
   ...: print (tzdata.__file__)
   ...: 
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 1
----> 1 import tzdata
      2 print (tzdata.__file__)

ModuleNotFoundError: No module named 'tzdata'

In [2]: from zoneinfo import ZoneInfo
   ...: ZoneInfo("UTC")
Out[2]: zoneinfo.ZoneInfo(key='UTC')

Just to note that I have the same behavior locally inside Docker.

[benjaoming]

@humitos it seems like the fallback behavior works in production.. so that's why we've never seen this error ✔️ In production, Python's tz data is provided by the system itself.

But we should be able to make development and production behave the same by adding tzdata to our requirements 👍

[humitos]

@benjaoming I'm confused. Why it doesn't work on development?

[humitos]

Oh, I see. There is a system dependency that's not installed in Docker.

[benjaoming]

@humitos yes, I think that Docker images have a tendency to be very slim... for regular distros, I can see that for instance Python in Ubuntu has tzdata as "suggested", but I read that it's "required" in the Debian package... or well.. it seems that there isn't a common consensus out there about whether tzdata is supplied as a system stdlib package or not. Which probably caused this.

[humitos]

@stsewd I tried the new invoke task and it didn't worked 🙃

$ inv requirements.update -p tzdata
/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/pip/_internal/models/link.py:391: PipDeprecationWarning: DEPRECATION: git+https://github.com/mozilla/unicode-slugify@b696c37#egg=unicode-slugify==0.1.5 contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at https://github.com/pypa/pip/issues/11617
  deprecated(
Traceback (most recent call last):
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/piptools/scripts/compile.py", line 589, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/humitos/rtfd/code/readthedocs.org/.direnv/python-3.10.8/lib/python3.10/site-packages/piptools/resolver.py", line 543, in resolve
    wheel_cache = WheelCache(
TypeError: WheelCache.__init__() takes 2 positional arguments but 3 were given

[benjaoming]

Love the first line of your traceback, containing site-packages/_distutils_hack 😈

You might wanna check if this is somehow your Python environment? The error doesn't happen for me. Not sure if -p can be used to add a package, I think it's just to select which package to update.

[stsewd]

Not sure if -p can be used to add a package, I think it's just to select which package to update.

@humitos yeah, you need to add the package to the pip.in file, and then run that command.

[humitos]

I got the same error without using -p, tho.

[ericholscher]

I just hit this locally when trying to build images for deploy 🙃

I was able to fix it with pip install tzdata in common.sh

[stsewd]

Looks like we need to update our version of celery celery/kombu#1748 (comment)

[benjaoming]

@stsewd interesting - we had some other issues with the latest Celery about picking exception types... made it seem complicated: #10445

[benjaoming]

I think we definitely want timezone information in our runtime environments. It can probably even lead to more issues handling datetime objects in the future. It should be a safe bet to add it as a requirement.

[humitos]

I opened #10482 to check if tests are going to pass or not with a full upgrade 🤷🏼