deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) #337

chalmerlowe · 2025-04-17T19:49:39Z

This branch is generated automagically via AI as an experiment.
Requires a comprehensive human review and comment.
Do not merge until approved by an authorized reviewer.

Removes support for Python 3.7 and 3.8, establishing Python 3.9 as the new minimum supported version.

This change involves:

Updating python_requires and classifiers in setup.py.
Modifying Python versions in noxfile.py (default, unit tests, system tests) and ensuring constraint file logic remains correct.
Updating the GitHub Actions workflow (unittest.yml) matrix, runner, and coverage job version.
Deleting constraint files for Python 3.7 and 3.8 (testing/constraints-3.7.txt, testing/constraints-3.8.txt).
Removing Kokoro sample configuration directories (.kokoro/samples/python3.7/, .kokoro/samples/python3.8/).
Updating supported version mentions in README.rst.
Removing 3.7 and 3.8 from the ALL_VERSIONS list in samples/snippets/noxfile.py.

Removes support for Python 3.7 and 3.8, establishing Python 3.9 as the new minimum supported version. This change involves: - Updating `python_requires` and classifiers in `setup.py`. - Modifying Python versions in `noxfile.py` (default, unit tests, system tests) and ensuring constraint file logic remains correct. - Updating the GitHub Actions workflow (`unittest.yml`) matrix, runner, and coverage job version. - Deleting constraint files for Python 3.7 and 3.8 (`testing/constraints-3.7.txt`, `testing/constraints-3.8.txt`). - Removing Kokoro sample configuration directories (`.kokoro/samples/python3.7/`, `.kokoro/samples/python3.8/`). - Updating supported version mentions in `README.rst`. - Removing 3.7 and 3.8 from the `ALL_VERSIONS` list in `samples/snippets/noxfile.py`.

Linchin · 2025-04-23T17:06:42Z

Just a suggestion, maybe use deps!: so this commit shows up in release notes? (I'm not 100% sure if a breaking change "chore" will appear in the release, though)

chalmerlowe · 2025-04-23T21:13:14Z

Just a suggestion, maybe use deps!: so this commit shows up in release notes? (I'm not 100% sure if a breaking change "chore" will appear in the release, though)

Conventional Commit points at this page as reference for prefixes:

'build',
'chore',
'ci',
'docs',
'feat',
'fix',
'perf',
'refactor',
'revert',
'style',
'test'

If Google also uses deps, then sure.

Linchin · 2025-04-23T21:48:19Z

Wow, I suggested it because we used it in the commit for python-bigquery. Apparently it's not part of convetional commits, but it worked.

Edit:
It seems conventional commits doesn't enforce a whitelist of commit types

chalmerlowe · 2025-04-23T22:18:32Z

Done.

tswast · 2025-04-24T14:19:06Z

db_dtypes/__init__.py

+# Inside db_dtypes/__init__.py (TEMPORARY DEBUG)
+print(f"DEBUG: Inside __init__, JSONArray type: {type(JSONArray)}, value: {repr(JSONArray)}")
+print(f"DEBUG: Inside __init__, JSONDtype type: {type(JSONDtype)}, value: {repr(JSONDtype)}")


We'll want to remove these before merging.

Suggested change

# Inside db_dtypes/__init__.py (TEMPORARY DEBUG)

print(f"DEBUG: Inside __init__, JSONArray type: {type(JSONArray)}, value: {repr(JSONArray)}")

print(f"DEBUG: Inside __init__, JSONDtype type: {type(JSONDtype)}, value: {repr(JSONDtype)}")

tswast · 2025-04-24T14:20:30Z

db_dtypes/__init__.py

+    if sys_major == 3 and sys_minor in (7, 8):
+        warnings.warn(
+            "The python-bigquery library as well as the python-db-dtypes-pandas library no "
+            "longer supports Python 3.7 and Python 3.8. "
+            f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
+            "recommend that you update soon to ensure ongoing support. For "
+            "more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
+            FutureWarning,
+            stacklevel=2, # Point warning to the caller of __init__
+        )


If you use python_requires in setup.py, I'm having a hard time seeing how someone could end up in this state.

Do we want to issue this in a separate PR and release so folks see it before fully dropping?

I can do that.

tswast · 2025-04-24T14:21:01Z

db_dtypes/__init__.py

+# sys_major, sys_minor, sys_micro = _versions_helpers.extract_runtime_version()
+# if sys_major == 3 and sys_minor in (7, 8):
+#     warnings.warn(
+#         "The python-bigquery library as well as the python-db-dtypes-pandas library no "
+#         "longer supports Python 3.7 and Python 3.8. "
+#         f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
+#         "recommend that you update soon to ensure ongoing support. For "
+#         "more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
+#         FutureWarning,
+#     )
+


Let's remove this commented out code.

Suggested change

# sys_major, sys_minor, sys_micro = _versions_helpers.extract_runtime_version()

# if sys_major == 3 and sys_minor in (7, 8):

# warnings.warn(

# "The python-bigquery library as well as the python-db-dtypes-pandas library no "

# "longer supports Python 3.7 and Python 3.8. "

# f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "

# "recommend that you update soon to ensure ongoing support. For "

# "more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",

# FutureWarning,

# )

tswast · 2025-04-24T14:22:04Z

db_dtypes/__init__.py

+_check_python_version()
+
+# Assign __all__ by calling the function
+__all__ = _determine_all(JSONArray, JSONDtype)


This seems a bit too "magic" to me, but I suppose it could make testing easier?

This code was originally just in the file with no function def. We added _check_python_version...

To test against supported and unsupported versions we attempted to mock the version but timing became an issue, partly because dependencies also rely on version checks so our mock ended up preventing error-free installs of some dependencies.

We tried half a dozen ways to alter when the mock happened but none hit that sweet spot of reliably allowing the dependencies to do their thing and slipping in to allow our test to run.

Similarly, we ran into a mess of problems with import timing and mocking whether JSONArray JSONDtype did or did not import as expected and so ...

putting them in functions made things much easier in the testing arena.

tswast · 2025-04-24T14:22:51Z

db_dtypes/json.py

+    # Type 'dbjson' might already be registered if the module is reloaded,
+    # which is okay.


Seems like this could be a separate PR since it's a fix for an issue unrelated to 3.7/3.8 support.

This was direct result of coverage blowing up.
I tried to fix a few things to restore coverage and it all went downhill from there, hence our convo in chat this AM.

tswast · 2025-04-24T14:23:56Z

testing/constraints-3.7.txt

Make sure to update constraints-3.9.txt with our minimum supported package versions from setup.py.

tswast · 2025-04-24T14:24:41Z

tests/compliance/date/test_date_compliance_1_5.py

@@ -24,7 +24,7 @@
 import pytest

 # NDArrayBacked2DTests suite added in https://github.com/pandas-dev/pandas/pull/44974
-pytest.importorskip("pandas", minversion="1.5.0dev")
+pytest.importorskip("pandas")


pandas is a strict dependency. I don't think we need this line. We were just using it for the min version before.

Suggested change

pytest.importorskip("pandas")

tswast · 2025-04-24T14:24:52Z

tests/compliance/time/test_time_compliance_1_5.py

@@ -24,7 +24,7 @@
 import pytest

 # NDArrayBacked2DTests suite added in https://github.com/pandas-dev/pandas/pull/44974
-pytest.importorskip("pandas", minversion="1.5.0dev")
+pytest.importorskip("pandas")


Same here

Suggested change

pytest.importorskip("pandas")

tswast · 2025-04-24T14:25:05Z

tests/unit/test__init__.py

@@ -0,0 +1,137 @@
+import sys


Missing license header.

tswast · 2025-04-24T14:27:11Z

tests/unit/test__init__.py

+import pytest
+import types
+import warnings
+from unittest import mock
+import pyarrow as pa


Imports should be grouped in the following order:

Standard library imports.

Related third party imports.

Local application/library specific imports.

You should put a blank line between each group of imports.

https://peps.python.org/pep-0008/#imports

Suggested change

import pytest

import types

import warnings

from unittest import mock

import pyarrow as pa

import types

import warnings

from unittest import mock

import pyarrow as pa

import pytest

chalmerlowe added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 17, 2025

chalmerlowe self-assigned this Apr 17, 2025

chalmerlowe requested review from a team as code owners April 17, 2025 19:49

chalmerlowe requested review from gericdong and Linchin April 17, 2025 19:49

product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-db-dtypes-pandas API. samples Issues that are directly related to samples. labels Apr 17, 2025

chalmerlowe added 6 commits April 18, 2025 09:46

Updates python version in lint.yml

dccef48

Updates owlbot, removing reference to 3.8

f81c738

Updates CONTRIBUTING.rst

5cb30ca

updates pytest warnings

7f44424

Removes test_samples-impl ref to older virtualenv package

3e4e6f6

Removes references to pandas older than 1.5.0

7beea0d

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Apr 18, 2025

chalmerlowe added 6 commits April 18, 2025 12:42

Removes pandas older than 1.5 and misc changes

32d7b8f

updates pandas in setup.py

e871d73

more updates related to pandas

2bdf3ef

still broken

d902e31

Updates FutureWarning tests to account for unittest coverage

a0543b9

Updates json array type tests to account for unittest coverage

6a5d943

updates python version checks to ensure coverage

1bbd5f7

chalmerlowe changed the title ~~chore!: Drop support for Python 3.7 and 3.8 (AI Experiment)~~ deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) Apr 23, 2025

chalmerlowe added 2 commits April 23, 2025 22:19

update json test for unittest coverage

96a5e97

Update pandas_backports unittests to ensure coverage

8af9bee

tswast reviewed Apr 24, 2025

View reviewed changes

Updates per review comments

78574c0

chalmerlowe mentioned this pull request Apr 24, 2025

deps: Updates deprecation warning to FutureWarning re: 3.7 and 3.8 #338

Merged

Merge branch 'main' into remove-python-37-38

9ca34b8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) #337

deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) #337

chalmerlowe commented Apr 17, 2025

Linchin commented Apr 23, 2025

chalmerlowe commented Apr 23, 2025

Linchin commented Apr 23, 2025 •

edited

Loading

chalmerlowe commented Apr 23, 2025

tswast Apr 24, 2025

tswast Apr 24, 2025

chalmerlowe Apr 24, 2025

tswast Apr 24, 2025

tswast Apr 24, 2025

chalmerlowe Apr 24, 2025

tswast Apr 24, 2025

chalmerlowe Apr 24, 2025

tswast Apr 24, 2025

tswast Apr 24, 2025

tswast Apr 24, 2025

tswast Apr 24, 2025

tswast Apr 24, 2025

	# Inside db_dtypes/__init__.py (TEMPORARY DEBUG)
	print(f"DEBUG: Inside __init__, JSONArray type: {type(JSONArray)}, value: {repr(JSONArray)}")
	print(f"DEBUG: Inside __init__, JSONDtype type: {type(JSONDtype)}, value: {repr(JSONDtype)}")

		# Type 'dbjson' might already be registered if the module is reloaded,
		# which is okay.

deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) #337

Are you sure you want to change the base?

deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) #337

Conversation

chalmerlowe commented Apr 17, 2025

Linchin commented Apr 23, 2025

chalmerlowe commented Apr 23, 2025

Linchin commented Apr 23, 2025 • edited Loading

chalmerlowe commented Apr 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linchin commented Apr 23, 2025 •

edited

Loading