wget Does not check for validity of headers prior to use

[samuelcomeau6]

In the event of an HTTP error, such as polling Adafruit IO api too often, the statement content_length = int(r.headers['content-length']) at line 638 is not valid which causes an unhandled exception.

Traceback (most recent call last):
File "code.py", line 33, in
File "code.py", line 29, in
File "adafruit_pyportal.py", line 845, in fetch
File "adafruit_pyportal.py", line 842, in fetch
File "adafruit_pyportal.py", line 838, in fetch
File "adafruit_pyportal.py", line 832, in fetch
File "adafruit_pyportal.py", line 638, in wget
KeyError: content-length
Code done running. Waiting for reload.
Auto-reload is on. Simply save files over USB to run them or enter REPL to disab
le.
Press any key to enter the REPL. Use CTRL-D to reload.

[samuelcomeau6]

Proposed solution:

        if 'content-length' in r.headers:
            content_length = int(r.headers['content-length'])
            remaining = content_length
        else:
            raise RuntimeError("content-length invalid")

[siddacious]

That looks like a good solution; please submit a PR

[samuelcomeau6]

#52

[kevinjwalters]

What's the HTTP status response code when the service is polled too frequently? I noted lack of checking there in #37

[kevinjwalters]

Strictly speaking the exception text should be "Content-Length missing"?

An absent Content-Length in an HTTP/1.1 response is not an error for chunked encoding. The adafruit_request library does not currently support that (https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/master/adafruit_requests.py#L228-L230) but if that was enhanced then the proposed change would stop it being used. Having said that the code probably needs adjusting anyway because it has its own "chunked" transfer mechanism. Apologies if that's all obvious and already been taken into consideration.

[samuelcomeau6]

So yes, it turns out the issue was a little bit more complicated than first look. I made a hack, not a fix.
The real issue is that the code raises a KeyError, a condition that should be recoverable, and crashes. This is one of the issues that makes running simple example code very frustrating and needs to be addressed in some way.
(I didn't think that #37 applied since it was in a different routine, but they appear connected)

[HerbertUnterberger]

will you go on with wget or should I finde an other way to download a file to my PyPortal ?
sorry for my simple question, but I not so experienced in http

[anecdata]

Also reported on Discord. It appears that this may be due to server responses that do not contain a content-length HTTP header. In that case, the client should just read until there are no more bytes (the server has closed the connection). The lower-level Requests library handles this case by setting content-length to 0, which triggers socket.recv to read until it can't: https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/43017e30a1e772b67ac68293a944e863c031e389/adafruit_requests.py#L107

[tannewt]

I've got chunked support in my pending Adafruit_Requests library as well. https://github.com/tannewt/Adafruit_CircuitPython_Requests/tree/http11

[kevinjwalters]

@anecdata The connection close is an interesting rather old skool case, they are rare nowadays because making a new connection is expensive, particularly with TLS. Perhaps simple http servers are generating them, do you have a full dump of request and response headers? It's down as the fifth and final method in preference order in the 1.1 spec: https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4

[anecdata]

I'm mainly going by the KeyError: content-length, triggered in line 638 in the top-most comment above

Adafruit_CircuitPython_PyPortal/adafruit_pyportal.py

Line 638 in 1416bc0

content_length = int(r.headers['content-length'])

It's possible some other bug causes the headers to not come in intact. But this client is (currently) HTTP/1.0, maybe some servers downgrade since they can't count on transfer-encoding (or are simply HTTP/1.0 servers). Contrast with handling of Key error in Requests library (link above). So this one is probably recoverable with a change to the library. I'll ask in Discord to see if we can get server headers. Client headers should just be the default CircuitPython GET HTTP/1.0 headers:

http-host: example.com
http-user-agent: Adafruit CircuitPython