_get_kwargs does not produce correct payload when there are data and files present

[skuo1-ilmn]

Describe the bug
When an API endpoint accepts data and files as a part of a multipart form data, the generated code puts the data fields (expected by httpx) into files, which leads to a TypeError in httpx when trying to encode the multipart form data.

To Reproduce
Steps to reproduce the behavior:

1. Generate client
2. Try to use the endpoint

Expected behavior
Should not encounter TypeError in httpx. More specifically, the _get_kwargs() function of the module for the endpoint needs to return an additional field, data, in addition to files for the plain text fields. That probably means multipart_data.to_multipart() should return both data and files. Currently, it only returns files.

OpenAPI Spec File
The openapi.yaml is private, but I can do my best to produce a snippet that would reproduce the error. The actual endpoint shouldn't matter because the TypeError is encountered before sending any requests.

openapi: 3.0.1
info:
  title: Rest API
  version: "3"
servers:
- url: /rest
security:
- JwtAuth: []
- ApiKeyAuth: []
paths:
  /api/projects/{projectId}/depositFile:
    post:
      tags:
      - Deposit File
      summary: Deposit a file.
      operationId: depositFile
      parameters:
      - name: projectId
        in: path
        description: The ID of the project
        required: true
        schema:
          type: string
      requestBody:
        content:
          multipart/form-data:
            schema:
              $ref: '#/components/schemas/DepositFileDto'
      responses:
        "201":
          description: The file is deposited successfully.
          content:
            application/json: {}
components:
  schemas:
    DepositFileDto:
      required:
      - code
      - description
      - mainFile
      - parametersXmlFile
      type: object
      properties:
        code:
          maxLength: 255
          minLength: 1
          type: string
          description: The code of the file
        description:
          maxLength: 4000
          minLength: 1
          type: string
          description: The description of the file
        mainFile:
          type: string
          description: The main file.
          format: binary
        versionComment:
          type: string
          nullable: true

Desktop (please complete the following information):

• OS: Windows 10
• Python Version: 3.9.4
• openapi-python-client version: 0.10.4

[dbanty]

Thanks for reporting! I'm going to do a sweep of PRs and bugs this weekend so I'll try and get this one in too.

[PKizzle]

For me this issue is still present in version 0.10.5.

Instead of producing something like this:

files = {}
data = {}
for key, value in multipart_data.to_dict().items():
    if isinstance(value, File):
        files[key] = value
    else:
        data[key] = value

return {
    "url": url,
    "headers": headers,
    "cookies": cookies,
    "timeout": client.get_timeout(),
    "data": data,
    "files": files,
    "verify": client.verify_ssl,
}

I receive this output instead:

multipart_multipart_data = multipart_data.to_multipart()

return {
    "url": url,
    "headers": headers,
    "cookies": cookies,
    "timeout": client.get_timeout(),
    "files": multipart_multipart_data,
}

[dbanty]

Looks like this was broken in 0.10.0 with #372. That was a really big PR, I should have done more testing on it 😞. Of course I don't want to break the feature that was added either. Sounds like in order to fix this properly is going to require really overhauling how multipart/form works (which will require me spending a bunch of time learning about it).

If anyone else is really familiar with multipart/form and wants to take a crack at righting the various wrongs in this client, I'd be really happy to see another PR on this. Otherwise, it might be a bit before I can spend enough time to really get this right.

[kairntech]

I wonder if the explanation of the issue is linked to the fact that with httpx version 0.19.0 a breaking change was made that forces file upload content to be bytes type and won't accept str type anymore.

So my suggestion would be to replace everywhere in templates the tuples (None, string, 'text/plain') or (None, string, 'application/json') by their bytes counterpart, i.e.

(None, string.encode(), 'text/plain') or (None, string.encode(), 'application/json')

For example in enum_property.py.jinja

{% if stringify %}
    {% set transformed = "(None, str(" + transformed + ").encode(), 'text/plain')" %}
    {% set type_string = "Union[Unset, Tuple[None, str, str]]" %}
{% endif %}

In list_property.py.jinja

{% if stringify %}
{{ stringified_destination }} = (None, json.dumps({{ destination }}).encode(), 'application/json')
{% endif %}

And in
In model_property.py.jinja

{% if stringify %}
    {% set transformed = "(None, json.dumps(" + transformed + ").encode(), 'application/json')" %}
    {% set type_string = "Union[Unset, Tuple[None, str, str]]" %}
{% else %}
    {% set type_string = property.get_type_string(json=True) %}
{% endif %}

What do you think?

[kairntech]

FYI I did this modification in my custom templates directory and it works well with httpx > 0.20

[dbanty]

Great find @kairntech! I confirmed that using the non-bytes variant (so as-generated today) in http 0.18.* works just fine, but I get the type error in the latest 0.21.1. I then added that little .encode() on the properties and confirmed it works for both 0.18.* and 0.21.1, so that seems like the fix (and a darn good one)!

I'll whip up a patch with those changes and get them in, but for all interested before the new release gets out, you can work around this by using httpx < 0.19.0

[dbanty]

Unfortunately I think the fix is going to be a breaking change, because I believe that the payload of a file can now only be BinaryIO and not TextIO.

[dbanty]

I've got a fix over in #548 which I've tested locally with some simple cases (though CI is annoyingly failing for everything right now). I'd really appreciate it if some of you could take a look, test it out with your own clients, and give it a review. It is a breaking change, so it'll wait until I'm ready to ship 0.11.0. Again, folks can work around this issue for now by using an older version of httpx (before 0.19.0).