Support responses without Content-Length

[zvxr]

Unfortunately HUE light hubs do not return HTTP responses with the Content-Length header (!!)

Response._remaining is based on Content-Length header. This inits with Nonetype value which causes comparison issues:

>>> resp = requests.get("http://1.2.3.4/api/username/lights/1")
>>> resp.content
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "adafruit_requests.py", line 311, in content
  File "adafruit_requests.py", line 365, in iter_content
  File "adafruit_requests.py", line 252, in close
TypeError: unsupported types for __gt__: 'NoneType', 'int'

Even when coercing or defaulting Response._remaining to zero instead, I think all methods now use Response._remaining so the actual content is never readable or drained properly.

>>> resp = requests.get("http://1.2.3.4/api/username/lights/1")
>>> resp.content
b''
>>> resp = requests.get("http://1.2.3.4/api/username/lights/2")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "adafruit_requests.py", line 632, in get
  File "adafruit_requests.py", line 540, in request
  File "adafruit_requests.py", line 537, in request
  File "adafruit_requests.py", line 459, in _send_request
  File "adafruit_requests.py", line 449, in _send
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 82, in send
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 710, in socket_write
RuntimeError: Failed to send 2 bytes (sent 0)

In version 1.6 it looks like the Content-Length is used for the content property, but Response.iter_content does not (It simply streams data). I believe that this is at least part of why it works when interacting with HUE.

[tannewt]

@zvxr What are the headers returned by HUE? Is it a chunked response?

[zvxr]

@tannewt raw response:

HTTP/1.1 200 OK
Server: nginx
Date: Mon, 05 Oct 2020 20:52:18 GMT
Content-Type: application/json
Connection: close
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Expires: Mon, 1 Aug 2011 09:00:00 GMT
Access-Control-Max-Age: 3600
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE, HEAD
Access-Control-Allow-Headers: Content-Type
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'
Cache-Control: no-store
Pragma: no-cache
Referrer-Policy: no-referrer

{"state" ...}   // data

[tannewt]

https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 Ok, looks like a bug on my part with HTTP1.1.

[ladyada]

@zvxr hi can you try the latest version we have released?

[zvxr]

@ladyada unfortunately I still receive an error using requests 1.7.4.

For a PUT request it successfully executes but has issues reading response:

  File "/lib/adafruit_hue.py", line 235, in set_scene
  File "/lib/adafruit_hue.py", line 217, in set_group
  File "/lib/adafruit_hue.py", line 260, in _put
  File "/lib/adafruit_requests.py", line 343, in json
ValueError: syntax error in JSON

For a GET request:

Socket missing recv_into. Using more memory to be compatible
Traceback (most recent call last):
  File "/lib/adafruit_hue.py", line 185, in get_light
  File "/lib/adafruit_hue.py", line 270, in _get
  File "/lib/adafruit_requests.py", line 343, in json
ValueError: syntax error in JSON

ln 343 self._raw.read() returns an empty string for both requests.

I tried also upgrading to latest adafruit_esp32spi (1.3.6).

[tannewt]

@zvxr Thanks for the second test! I suspected that it wouldn't work. Any idea how we should know when the response is complete? We could read to \r\n but that will break if the contents include it.

[zvxr]

@tannewt makes sense-- I can't think of another way besides reading \r\n, but can see that being brittle. Maybe continue reading if it is not valid JSON? (but I could see that getting messy)

For HUE it does looks like all API responses include no whitespace in response.

[anecdata]

Since there's no Content-Length header, and no Transfer-Encoding': 'chunked' header, but there is a Connection: close header, this is probably the case where the server has closed the socket and the client is expected to detect that and stop reading (scenario 5 in section 4.4 at the link above).

@zvxr What board is this? ESP32SPI has access to socket state and it could be checked so that Requests and other higher-level protocols could handle this scenario. Something similar would need to be exposed for other socket sources.

For CPython compatibility, it looks like socket.fileno() might be the API (-1 on failure).

[zvxr]

@anecdata I have a couple projects using both Adafruit AirLift Shield and Adafruit Metro M4 Express AirLift -- both ESP32.

[syndat]

Hi, i tried the Hue-Example with a PyPortal!
Circuitpython 7.2.5 ESP Firmware 1.7.4 and lib from 20220426

i get this error before i can press the button on my hub:

Finding bridge address...
Attempting to register username, press the link button on your Hue Bridge now!
Traceback (most recent call last):
File "", line 51, in
File "adafruit_hue.py", line 123, in register_username
File "adafruit_requests.py", line 471, in json
ValueError: syntax error in JSON

is ther a solution to overcome this? Many Thanks for Help!

[anecdata]

Closing as fixed by #123