[REQ][python] get user's mypy check passing with warn_redundant_casts turned on #246

nponsard · 2023-09-29T14:35:47Z

Is your feature request related to a problem? Please describe.

I tried to run mypy on code generated by this project and got around 30 errors. The same happens when running mypy on the provided samples. There was an issue about that one year ago #1, is this a regression ?

Some errors I get on my project :

nethsm/client/components/schema/provision_request_data.py:76: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:83: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/paths/keys_generate/post/responses/response_201/__init__.py:19: error: Attributes without a default cannot follow attributes with one  [misc]
nethsm/client/paths/keys/post/responses/response_201/__init__.py:19: error: Attributes without a default cannot follow attributes with one  [misc]
nethsm/client/components/schema/public_key.py:78: error: Incompatible types in assignment (expression has type "str", variable has type "Union[Mapping[str, Union[dict[Any, Any], immutabledict[Any, Any], Mapping[str, object], list[Any], tuple[Any, ...], float, int, str, date, datetime, UUID, bool, None, bytes, io.FileIO, BufferedReader, nethsm.client.schemas.schema.FileIO]], KeyPublicDataDict, Unset]")  [assignment]

Describe the solution you'd like

That the generated code doesn't emit any type error with mypy.

Describe alternatives you've considered

I disabled type checking for the folder containing the generated code in my pyproject.toml :

[[tool.mypy.overrides]]
module = "nethsm.client.*"
ignore_errors = true

The text was updated successfully, but these errors were encountered:

spacether · 2023-09-29T15:50:28Z

Can you please provide your openapi document and the 30 errors? Mypy is successfully passing on the petstore sample in CI.

jans23 · 2023-09-29T15:52:22Z

The OpenAPI is here.

spacether · 2023-10-05T17:34:29Z

What are the 30 errors you get on your project? I only get 6.
How are you invoking mypy?

python3 -m venv venv
source venv/bin/activate
python -m pip install .
pip install mypy
# I passed in --additional-properties packageName=some_api
mypy src/some_api

-->

src/some_api/paths/keys_generate/post/responses/response_201/__init__.py:19: error: Attributes without a default cannot follow attributes with one  [misc]
src/some_api/paths/keys/post/responses/response_201/__init__.py:19: error: Attributes without a default cannot follow attributes with one  [misc]
src/some_api/components/schema/public_key.py:78: error: Incompatible types in assignment (expression has type "str", variable has type "Union[Mapping[str, Union[Dict[Any, Any], immutabledict[Any, Any], Mapping[str, object], List[Any], Tuple[Any, ...], float, int, str, date, datetime, UUID, bool, None, bytes, io.FileIO, BufferedReader, some_api.schemas.schema.FileIO]], KeyPublicDataDict, Unset]")  [assignment]
src/some_api/components/schema/public_key.py:83: error: Invalid index type "Union[Mapping[str, Union[Dict[Any, Any], immutabledict[Any, Any], Mapping[str, object], List[Any], Tuple[Any, ...], float, int, str, date, datetime, UUID, bool, None, bytes, io.FileIO, BufferedReader, some_api.schemas.schema.FileIO]], KeyPublicDataDict, Unset]" for "Dict[str, Any]"; expected type "str"  [index]
src/some_api/components/schema/private_key.py:73: error: Incompatible types in assignment (expression has type "str", variable has type "Union[Mapping[str, Union[Dict[Any, Any], immutabledict[Any, Any], Mapping[str, object], List[Any], Tuple[Any, ...], float, int, str, date, datetime, UUID, bool, None, bytes, io.FileIO, BufferedReader, some_api.schemas.schema.FileIO]], KeyPrivateDataDict]")  [assignment]
src/some_api/components/schema/private_key.py:78: error: Invalid index type "Union[Mapping[str, Union[Dict[Any, Any], immutabledict[Any, Any], Mapping[str, object], List[Any], Tuple[Any, ...], float, int, str, date, datetime, UUID, bool, None, bytes, io.FileIO, BufferedReader, some_api.schemas.schema.FileIO]], KeyPrivateDataDict]" for "Dict[str, Any]"; expected type "str"  [index]
Found 6 errors in 4 files (checked 1004 source files)

spacether · 2023-10-05T19:56:01Z

@nponsard my merged PR should resolve this issue. Can you please verify that?
Local verification with your client passed. Petstore with added component response and schema with the same root causes as yours passsed mypy check in CI.

nponsard · 2023-10-05T20:25:22Z

I still have some errors :

nethsm/client/servers/server_0.py:74: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/servers/server_0.py:81: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/time_config.py:59: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/system_update_data.py:56: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/random_request_data.py:67: error: Redundant cast to "int"  [redundant-cast]
nethsm/client/components/schema/network_config.py:66: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/network_config.py:73: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/network_config.py:80: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/info_data.py:61: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/info_data.py:68: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:109: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:119: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:129: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:139: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:149: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:159: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/distinguished_name.py:169: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/tls_key_generate_request_data.py:98: error: Redundant cast to "int"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:69: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:76: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:83: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/public_key.py:107: error: Redundant cast to "int"  [redundant-cast]

It's weird that you don't have these ones reported.

spacether · 2023-10-05T20:29:56Z

Please provide the steps to reproduce your output.
Also, what python version are you using?
My steps in the root of the generated client are:

python3 -m venv venv
source venv/bin/activate
python -m pip install .
pip install mypy
# I passed in --additional-properties packageName=some_api
mypy src/some_api

nponsard · 2023-10-05T21:12:43Z

Here's what I've done

git clone [email protected]:Nitrokey/nethsm-sdk-py.git
cd nethsm-sdk-py
make init
source venv/bin/activate
#regenerate the client
make nethsm-client
mypy nethsm

spacether · 2023-10-06T18:20:04Z

This is happening because you have:

[tool.mypy]
warn_redundant_casts = true

turned on in your pyproject.toml which is defaulted to false if unset
and I am able to reproduce warnings by turning it on
If you remove that line mypy checks will succeed

spacether · 2023-10-09T20:27:38Z

@nponsard I just merged in a commit that should eliminate your redundant casts issue. Can you verify it on your side? Locally it eliminated all redundant casts issues when mypy was run against the petstore client with warn_redundant_casts turned on.

robin-nitrokey · 2023-10-12T11:38:21Z

@spacether Thanks for the fix!

To provide some context on why we set warn_redundant_casts: nethsm-sdk-py was extracted from a bigger library, pynitrokey. In this library, we wanted to enable some basic checks for legacy code and strict checks for new code. Unfortunately, mypy’s strict option only works globally. So we manually set the options for the new modules that strict would set, see Nitrokey/pynitrokey#444. warn_unused_configs and warn_redundant_casts are the only strict options that can only be set globally, not on a per-module level.

Ideally, the OpenAPI generated code would also pass strict checks, but passing warn_unused_configs and warn_redundant_casts would already be a big improvement because it would allow us to manually enable the strict lints for other modules.

Now back to this issue: I’ve re-generated the code with the latest Docker image (sha256:efb6f7530bd2e6aaf1d6a6ece9a5bf0e045a47581e9e2d36a70eaf99195ead40). I still get some redundant-cast errors for the generated code:

nethsm/client/components/schema/time_config.py:59: error: Redundant cast to "str"  [redundant-cast]                                         
nethsm/client/components/schema/random_request_data.py:67: error: Redundant cast to "int"  [redundant-cast]                                 
nethsm/client/components/schema/tls_key_generate_request_data.py:98: error: Redundant cast to "int"  [redundant-cast]                       nethsm/client/components/schema/provision_request_data.py:69: error: Redundant cast to "str"  [redundant-cast]                              
nethsm/client/components/schema/provision_request_data.py:76: error: Redundant cast to "str"  [redundant-cast]                              
nethsm/client/components/schema/provision_request_data.py:83: error: Redundant cast to "str"  [redundant-cast]                              
nethsm/client/components/schema/public_key.py:107: error: Redundant cast to "int"  [redundant-cast]

Unfortunately it looks like the changes cause a different problem. Consider this snippet from nethsm/__init__.py:503-504:

response = self.get_api().users_get()
return [str(item["user"]) for item in response.body]

response is correctly typed as a client.paths.users.get.responses.response_200.ApiResponse. But response.body is not typed as application_json_schema.user_list.UserListTuple but the generic Union[Unset, OUTPUT_BASE_TYPES] from api_response.ApiResponse. I see this problem both with mypy 1.4.1 and mypy 1.6. Did you run into similar problems? Do you have an idea how to fix it?

spacether · 2023-10-12T16:44:07Z

You're welcome for the two PRs.

I can't tell what the root cause is from the trace that you posted. Does your repo include the latest update? It does not look like it does. Can you please post a repo or branch that demonstrates the current problems that I can look at? I have not run into similar problems.

spacether · 2023-10-12T17:46:06Z

Based on the additional context that you provided, you are expected strict mode options to pass when running mypy. mypy strict mode checks are not yet working on the client. When one runs mypy with the default options, not the additional ones that you specify in your toml file, it passed. Because of this, I am changing this to a feature request to get mypy passing with the warn_redundant_casts turned on.

robin-nitrokey · 2023-10-12T18:19:02Z

I’ve created a draft PR with the updated client code that causes both the redundant-cast warnings and the response body issue: Nitrokey/nethsm-sdk-py#42

spacether · 2023-10-12T19:36:34Z

Thanks, I am also working on this in: spacether/nethsm-sdk-py#1 but I am running into issues where the ApiResponse.body types are not correctly seen by mypy. I am seeing errors like:

nethsm/__init__.py:722: error: Value of type "Union[Unset, OUTPUT_BASE_TYPES]" is not indexable  [index]
nethsm/__init__.py:722: error: No overload variant of "__getitem__" of "tuple" matches argument type "str"  [call-overload]
nethsm/__init__.py:722: note: Possible overload variants:
nethsm/__init__.py:722: note:     def __getitem__(self, SupportsIndex, /) -> OUTPUT_BASE_TYPES
nethsm/__init__.py:722: note:     def __getitem__(self, slice, /) -> tuple[OUTPUT_BASE_TYPES, ...]

robin-nitrokey · 2023-10-12T20:02:23Z

I think this is the same problem that I mentioned here (in the last paragraph). Here’s the full list of errors that I get with our setup: https://github.com/Nitrokey/nethsm-sdk-py/actions/runs/6499415750/job/17652723428?pr=42

spacether · 2023-10-12T20:52:51Z

So I just converted the ApiResponse classes back to dataclasses and removed default values in all except ApiResponseWithoutDeserialization. Then I rebuilt your client with it here:
spacether/nethsm-sdk-py#1
In it, mypy runs without issue:

venv/bin/python3 -m mypy nethsm/
Success: no issues found in 1092 source files

Please verify that my new code solves your mypy issues. I have not been able to replicate your lingering redundant_cast issues though I see in time_config.py TimeConfigDict.time that there is a redundant cast happening.
Why would my above mypy invocation not be generating the same errors? We are both in the same working dir using a python 3.9 venv.

robin-nitrokey · 2023-10-13T10:29:20Z

I’ve updated the draft PR Nitrokey/nethsm-sdk-py#42 using d2d1d8b. The request body issues are now fixed, but I still get the redundant cast warnings:

nethsm/client/components/schema/time_config.py:59: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/random_request_data.py:67: error: Redundant cast to "int"  [redundant-cast]
nethsm/client/components/schema/tls_key_generate_request_data.py:98: error: Redundant cast to "int"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:69: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:76: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:83: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/public_key.py:107: error: Redundant cast to "int"  [redundant-cast]

I’m using Python 3.9.2 and mypy 1.6.0. It also fails in the CI, so it is not limited to my environment: https://github.com/Nitrokey/nethsm-sdk-py/actions/runs/6507023981/job/17673671662. And it also happens with Python 3.10.13: https://github.com/Nitrokey/nethsm-sdk-py/actions/runs/6507070669/job/17673794274 What mypy version do you use? What happens if you run mypy on my branch?

Until upstream issue #246 [0] is fixed, pinning openapi-json-schme-generator-cli to 3.1.0 makes sure that the CI pases and that the generated code works with our existing code. [0] openapi-json-schema-tools/openapi-json-schema-generator#246

spacether · 2023-10-13T18:10:46Z

So I grabbed your branch and ran it locally. In it, I needed to pin to python 3.9 because my system default python is 3.8.
I did that in this branch: https://github.com/spacether/nethsm-sdk-py/tree/update-openapi_py39anchor
And when I ran it locally I got the redundant cast errors:

(venv) justinblack@Justins-MacBook-Air nethsm-sdk-py % mypy nethsm             
nethsm/client/components/schema/time_config.py:59: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/random_request_data.py:67: error: Redundant cast to "int"  [redundant-cast]
nethsm/client/components/schema/tls_key_generate_request_data.py:98: error: Redundant cast to "int"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:69: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:76: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/provision_request_data.py:83: error: Redundant cast to "str"  [redundant-cast]
nethsm/client/components/schema/public_key.py:107: error: Redundant cast to "int"  [redundant-cast]
Found 7 errors in 5 files (checked 1092 source files)

My python is:

Python 3.9.18 (main, Aug 24 2023, 21:37:44) 
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin

Any my mypy is: 1.6.0

However your branch is anchored on my 3.1.0 release so it will always have that issue.

I am not understanding why my new branch is not generating the redundant casts issue:
https://github.com/spacether/nethsm-sdk-py/tree/feat_jblack_python_fix
mypy passes on it

(venv) justinblack@Justins-MacBook-Air nethsm-sdk-py % mypy nethsm       
Success: no issues found in 1092 source files

Can you please please pull my branch feat_jblack_python_fix and run mypy in a new venv without regenerating the client? Do you still get the redundant casts issue?

robin-nitrokey · 2023-10-13T20:37:21Z

For the Nitrokey/nethsm-sdk-py#42 branch, I built the Docker image locally from Git (d2d1d8b) and re-generated the client code so it was up to date and not from 3.1.0.

Your branch on spacether/nethsm-sdk-py@ea8e891 still has ignore_errors = true for nethsm.client.* in pyproject.toml. If I remove that, I only get one error:

nethsm/client/servers/server_0.py:132: error: Incompatible types in assignment (expression has type "VariablesDict", base class "ServerWithVariables" defined the type as "immutabledict[str, str]")  [assignment]

I’ve updated my Nitrokey/nethsm-sdk-py#42 branch using 24c9e46 and I think all issues are fixed now. 🎉

Edit: CI agrees!

spacether · 2023-10-13T20:51:36Z

Glad to hear it. I will add some tests to #252 before I merge it in. Happy to hear that it solved your issues.
Note: if you want the object output classes type hints to only contain values from known properties, you must set additionalProperties to false, or a schema value. By default, unset additionalProperties allows any type in, which is why most output object based classes will inherit from: schemas.immutabledict[str, schemas.OUTPUT_BASE_TYPES]

spacether · 2023-10-16T19:38:33Z

@robin-nitrokey today I released v3.1.1 which includes the fixes needed by this issue. Please use it to resolve this warn_redundant_casts mypy issue. Thank you for reporting this issue and providing more info and context on the causes.

robin-nitrokey · 2023-10-17T20:28:13Z

@spacether Great! Thank you for working on this issue and for publishing the fix! I’ve update my PR to use the released version. Nitrokey/nethsm-sdk-py#42

spacether · 2023-10-17T21:12:59Z

You're welcome 🙂

robin-nitrokey · 2023-10-17T21:23:47Z

I’ve now enabled strict checks for our own code and it works nicely with the generated code. :) Only one small issue remains: ApiClient.close is untyped. But I can just ignore that error.

spacether · 2023-10-17T21:38:32Z

Feel free to submit PRs to the generator to fix any issues that you find. Contributions are welcome.

nponsard mentioned this issue Sep 29, 2023

Enable type checks for nethsm code Nitrokey/nethsm-sdk-py#32

Closed

spacether changed the title ~~[REQ][python] fix typing errors~~ [REQ][python] mypy errors for user openapi document Sep 29, 2023

spacether linked a pull request Oct 5, 2023 that will close this issue

Fixes mypy errors in user document #250

Merged

3 tasks

spacether closed this as completed in #250 Oct 5, 2023

spacether reopened this Oct 5, 2023

spacether linked a pull request Oct 9, 2023 that will close this issue

Fixes mypy redundant casts issue #251

Merged

3 tasks

spacether closed this as completed in #251 Oct 9, 2023

jans23 mentioned this issue Oct 10, 2023

update OpenAPI generator Nitrokey/nethsm-sdk-py#39

Closed

spacether reopened this Oct 12, 2023

spacether changed the title ~~[REQ][python] mypy errors for user openapi document~~ [REQ][python] get user's client mypy check passing with warn_redundant_casts turned on Oct 12, 2023

spacether linked a pull request Oct 12, 2023 that will close this issue

Issue 246 fix, type hints for object schema dict output class values improved #252

Merged

3 tasks

spacether changed the title ~~[REQ][python] get user's client mypy check passing with warn_redundant_casts turned on~~ [REQ][python] get user's mypy check passing with warn_redundant_casts turned on Oct 13, 2023

spacether closed this as completed in #252 Oct 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REQ][python] get user's mypy check passing with warn_redundant_casts turned on #246

[REQ][python] get user's mypy check passing with warn_redundant_casts turned on #246

nponsard commented Sep 29, 2023

spacether commented Sep 29, 2023 •

edited

Loading

jans23 commented Sep 29, 2023

spacether commented Oct 5, 2023 •

edited

Loading

spacether commented Oct 5, 2023

nponsard commented Oct 5, 2023

spacether commented Oct 5, 2023 •

edited

Loading

nponsard commented Oct 5, 2023 •

edited

Loading

spacether commented Oct 6, 2023 •

edited

Loading

spacether commented Oct 9, 2023

robin-nitrokey commented Oct 12, 2023

spacether commented Oct 12, 2023 •

edited

Loading

spacether commented Oct 12, 2023 •

edited

Loading

robin-nitrokey commented Oct 12, 2023

spacether commented Oct 12, 2023

robin-nitrokey commented Oct 12, 2023

spacether commented Oct 12, 2023 •

edited

Loading

robin-nitrokey commented Oct 13, 2023

spacether commented Oct 13, 2023

robin-nitrokey commented Oct 13, 2023 •

edited

Loading

spacether commented Oct 13, 2023 •

edited

Loading

spacether commented Oct 16, 2023 •

edited

Loading

robin-nitrokey commented Oct 17, 2023

spacether commented Oct 17, 2023

robin-nitrokey commented Oct 17, 2023

spacether commented Oct 17, 2023 •

edited

Loading

[REQ][python] get user's mypy check passing with warn_redundant_casts turned on #246

[REQ][python] get user's mypy check passing with warn_redundant_casts turned on #246

Comments

nponsard commented Sep 29, 2023

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

spacether commented Sep 29, 2023 • edited Loading

jans23 commented Sep 29, 2023

spacether commented Oct 5, 2023 • edited Loading

spacether commented Oct 5, 2023

nponsard commented Oct 5, 2023

spacether commented Oct 5, 2023 • edited Loading

nponsard commented Oct 5, 2023 • edited Loading

spacether commented Oct 6, 2023 • edited Loading

spacether commented Oct 9, 2023

robin-nitrokey commented Oct 12, 2023

spacether commented Oct 12, 2023 • edited Loading

spacether commented Oct 12, 2023 • edited Loading

robin-nitrokey commented Oct 12, 2023

spacether commented Oct 12, 2023

robin-nitrokey commented Oct 12, 2023

spacether commented Oct 12, 2023 • edited Loading

robin-nitrokey commented Oct 13, 2023

spacether commented Oct 13, 2023

robin-nitrokey commented Oct 13, 2023 • edited Loading

spacether commented Oct 13, 2023 • edited Loading

spacether commented Oct 16, 2023 • edited Loading

robin-nitrokey commented Oct 17, 2023

spacether commented Oct 17, 2023

robin-nitrokey commented Oct 17, 2023

spacether commented Oct 17, 2023 • edited Loading

spacether commented Sep 29, 2023 •

edited

Loading

spacether commented Oct 5, 2023 •

edited

Loading

spacether commented Oct 5, 2023 •

edited

Loading

nponsard commented Oct 5, 2023 •

edited

Loading

spacether commented Oct 6, 2023 •

edited

Loading

spacether commented Oct 12, 2023 •

edited

Loading

spacether commented Oct 12, 2023 •

edited

Loading

spacether commented Oct 12, 2023 •

edited

Loading

robin-nitrokey commented Oct 13, 2023 •

edited

Loading

spacether commented Oct 13, 2023 •

edited

Loading

spacether commented Oct 16, 2023 •

edited

Loading

spacether commented Oct 17, 2023 •

edited

Loading