Getting a request.py error in the code for the "MagTag Lists From Google Spreadsheets" tutorial

[wavesailor]

I'm getting an error running the following code: https://github.com/adafruit/Adafruit_Learning_System_Guides/blob/main/MagTag_Google_Sheets/weekly_planner/code.py

Connecting to AP MyWiFI
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Traceback (most recent call last):
  File "code.py", line 101, in <module>
  File "adafruit_portalbase/network.py", line 478, in fetch
  File "adafruit_requests.py", line 769, in get
  File "adafruit_requests.py", line 758, in request
  File "adafruit_requests.py", line 702, in request
  File "adafruit_requests.py", line 378, in close
ValueError: invalid syntax for integer with base 16

Code done running.

I use wget on my Pi to retrieve my URL just fine, but it does not work on the MagTag

I'm using the latest libraries and Circuit Python for MagTag

adafruit-circuitpython-adafruit_magtag_2.9_grayscale-en_US-7.2.3.uf2
adafruit-circuitpython-bundle-7.x-mpy-20220321.zip

[wavesailor]

I tested the code with a dummy URL: https://api.publicapis.org/entries
and I don't get an error.

but using the google spreadsheets URL (https://docs.google.com/spreadsheets/d/e/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/pub?output=tsv)
I get the error.

Could it be how the data is being returned?

[anecdata]

Google may have changed up the URLs and something may be getting lost in a redirect. Can you test with curl, something like:

curl -iL -A "Adafruit CircuitPython" --http1.1 "https://docs.google.com/spreadsheets/d/e/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/pub?output=tsv"

and see what the sequence of URLs is.

edit: found the uRL, and it is getting a 307 Temporary Redirect. Requests should handle routine redirects, so I'm not sure if or why this one would be an issue.

[anecdata]

Looks to me like Requests will do redirects within the same host:

Adafruit_CircuitPython_Requests/adafruit_requests.py

Line 737 in 2705656

if "location" in resp.headers and 300 <= resp.status_code <= 399:

but Google does the 307 to a new subdomain & domain.

[wavesailor]

I kinda follow what you saying .... my initial request is redirected (Temporary redirect 307) to another host but still Google.

But does this mean requests.py needs to be corrected/fixed?

[anecdata]

I believe adafruit_requests would need a change, since the host changes on redirect. You might be able to put the final https://doc-14-2g-sheets.googleusercontent.com/... URL into the CircuitPython code for it to work. It's a temporary redirect, so it could change again in the future.

[wavesailor]

Okay thanks. Do I have to log a request somewhere to get it fixed?

PS. I did try use the redirected URL in but it expires are a certain period :-(

[anecdata]

Yeah, I couldn't get it to work in curl either. A bit unfriendly to require a heavy redirect to access an asset.

This issue serves as the request to get the issue addressed, but it is dependent on resources and priorities, or someone motivated enough to fix it sooner ;-). I'm going to also file a new issue in the Guide to keep track there... maybe there's an alternate way to serve up the spreadsheet.

[anecdata]

@wavesailor It might work to get the response headers and parse out the new temporary Location URL and then turn around and fetch that.

[wavesailor]

The results I get are definitely a bit strange. I changed the code to retry every minute. This happens if it is successful or if there is a runtime error.

It would not work for a few tries, then it would miraculously work, then it would fail again and finally it would crash circuit python.

This is the output I got:

Connecting to AP MyWifi
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...OK
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...OK
Updating time
Getting time for timezone America/New_York
Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Traceback (most recent call last):
  File "code.py", line 105, in <module>
  File "adafruit_portalbase/network.py", line 478, in fetch
  File "adafruit_requests.py", line 769, in get
  File "adafruit_requests.py", line 758, in request
  File "adafruit_requests.py", line 702, in request
  File "adafruit_requests.py", line 378, in close
ValueError: invalid syntax for integer with base 16

Code done running.

[wavesailor]

Now I cannot even get it to run at all :-(

I only get this:

Retrieving data...Traceback (most recent call last):
  File "code.py", line 105, in <module>
  File "adafruit_portalbase/network.py", line 478, in fetch
  File "adafruit_requests.py", line 769, in get
  File "adafruit_requests.py", line 758, in request
  File "adafruit_requests.py", line 702, in request
  File "adafruit_requests.py", line 378, in close
ValueError: invalid syntax for integer with base 16

Code done running.

So I cannot even try get the redirected URL in because it is failing in the close function of the adafruit_requests.py

[wavesailor]

Besides the Temporary redirect 307 of the URL, I feel some error or value is not being checked in the close function of the response class in adafruit_requests.py

[dhalbert]

The invalid syntax for integer with base 16 is because it thinks it's reading a chunk header with a hex size and trying to parse it as a hex number. So probably it's reading something else that does not look like a number. You could print chunk_header in close() and see what it's trying to parse.

[wavesailor]

I tried to debug this further but have not had success.
I copied adafruit_requests.py to the lib folder on my MagTag but it does not seem to use the file as I put a slew of print statements in the code but none appear .... am I doing something wrong???

[anecdata]

make sure there is no adafruit_requests at the root level, /

[FoamyGuy]

adafruit_requests library is frozen into the build for MagTag: https://github.com/adafruit/circuitpython/blob/main/ports/espressif/boards/adafruit_magtag_2.9_grayscale/mpconfigboard.mk#L24

In order to test changes you'll need either A) a build without requests frozen in so that it will use the modified copy in lib or B) a build with your modified requests as the one that is frozen in.

Both options require making a custom build though as far as I know. Option B would then also require making a build for each new change you make to requests which can get tedious.

If you're interested in trying it out but haven't already, or don't want to go through the building process perhaps one of us can create a custom build that you can use temporarily for testing this.

[anecdata]

I think putting the library in / should override the frozen library.

[wavesailor]

@FoamyGuy Ahhh .... I was starting to pull my hair out
Yeah a custom build could would help - Option A would be preferable so I could edit adafruit_requests myself.

or if you have one already then you could just test with this URL and see what is causing it to crash
https://docs.google.com/spreadsheets/d/e/2PACX1vTA8pXQodbEiz5idGT21YkL1Vy8waW0aAHM1uX7D4TqBq6DrUU8qXVlON1QVaWSlmoC3OBL4Iokyiyy/pub?output=tsv

[FoamyGuy]

Turns out I was missing another (much nicer) option.

If you put your modified adafruit_requests in the root of the drive right next to code.py it should use that copy instead of the frozen one. So that would allow you to test modifications without needing the custom build.

I probably won't have time to test the URL on a reasonable timeline. Try with the library in the root of the drive. If it still doesn't seem to be using your modified one let me know and I can make you a build without requests frozen in and share that.

[anecdata]

You can also run many different ESP32-S2 UF2s on a MagTag, just find one with the same flash and RAM, maybe the Saola Wrover. The pins won't have nice names, but it will function.

[wavesailor]

Okay I got it to read adafruit_requests.py but it does not work for network.py - any idea how to get that to work?

[FoamyGuy]

You mean network.py from the adafruit_magtag library? I think you should be able to modify that and include your modified version in the root as well. you'd have these files in the root of your drive:

code.py
adafruit_requests.py
adafruit_magtag/

MagTag library is a folder instead of single file, but should work similarly.

[wavesailor]

@FoamyGuy Thanks - I'll give that a try.

Trying to debug this issue, I find myself a little out of my depth. I can't understand why you would convert a string into bytes and then try convert it into an integer? To me it seems obvious that it will fail

                while True:
                    chunk_header = _buffer_split0(self._readto(b"\r\n"), b";")
                    chunk_size = int(bytes(chunk_header), 16)
                    if chunk_size == 0:
                        break

[anecdata]

The chunk header is a length specified as "hexadecimal number in ASCII":
https://en.wikipedia.org/wiki/Chunked_transfer_encoding#Format

[wavesailor]

@anecdata Thanks for that info.

Well I think this would then point to a problem elsewhere - because this is what is in the bytearray:
bytearray(b'<HTML>\n<HEAD>\n<TITLE>Temporary Redirect</TITLE>\n</HEAD>\n<BODY BGCOLOR="#FFFFFF" TEXT="#000000">\n<H1>Temporary Redirect</H1>\nThe document has moved <A HREF="https://doc-0s-8o-sheets.googleusercontent.com/pub/70cmver1f290kjsnpar5ku2h9g/aq8d5fkt07hteeh15p45kbnj04/1647975070000/114735129096105625183/*/e@2PACX-1vRtM4cXAyv0R1vv1t4oH5m9rNP0WbZIwDWHjQ2gD7nn55KdQeo78jI0wRaQrMYD2nu6Vq264_4IFUzY?output=tsv">here</A>.\n</BODY>\n</HTML>\n')

[anecdata]

It's possible that the chunked 307 response is malformed, but curl seems to skip the body:

Transfer-Encoding: chunked

< 
* Ignoring the response-body

(curl -iLv --raw --http1.1 "https://docs.google.com/spreadsheets/d/e/2PACX-1vR1WjUKz35-ek6SiR5droDfvPp51MTds4wUs57vEZNh2uDfihSTPhTaiiRovLbNe1mkeRgurppRJ_Zy/pub?output=tsv")

so I'm not sure how to test that. But assuming that the 307 response is good, the issue seems to arise as close tries to drain the socket and throw away the content. I wonder if there's a way we can just skip that and start a new request and response.

Just putting in some debug messages into Requests, it appears that the chunk header is broken:

code.py output:
DEBUG location https://doc-14-2g-sheets.googleusercontent.com/pub/70cmver1f290kjsnpar5ku2h9g/np3od705vsf2319pfepg6q795s/1647995020000/109226138307867586192/*/e@2PACX-1vR1WjUKz35-ek6SiR5droDfvPp51MTds4wUs57vEZNh2uDfihSTPhTaiiRovLbNe1mkeRgurppRJ_Zy?output=tsv
DEBUG close chunk_header b'1aa'
DEBUG close chunk_size 426
# self._throw_away(chunk_size + 2) gets called next...
# then code loops back to process the next chunk header, which looks more like chunk than chunk header...
DEBUG close chunk_header b'k6SiR5droDfvPp51MTds4wUs57vEZNh2uDfihSTPhTaiiRovLbNe1mkeRgurppRJ_Zy?output=tsv">here</A>.\n</BODY>\n</HTML>\n'
Traceback (most recent call last):
  File "code.py", line 15, in <module>
  File "adafruit_requests.py", line 724, in get
  File "adafruit_requests.py", line 713, in request
  File "adafruit_requests.py", line 655, in request
  File "adafruit_requests.py", line 332, in close
ValueError: invalid syntax for integer with base 16

But again, that's more likely a library issue than a Google issue.

BTW, Requests does handle the redirect to the alternate host successfully.

[wavesailor]

looking at micropython code here https://github.com/micropython/micropython-lib/blob/master/micropython/urllib.urequest/urllib/urequest.py I see the following that redirects not yet supported:

    s = usocket.socket(ai[0], ai[1], ai[2])
    try:
        s.connect(ai[-1])
        if proto == "https:":
            s = ussl.wrap_socket(s, server_hostname=host)

        s.write(method)
        s.write(b" /")
        s.write(path)
        s.write(b" HTTP/1.0\r\nHost: ")
        s.write(host)
        s.write(b"\r\n")

        if data:
            s.write(b"Content-Length: ")
            s.write(str(len(data)))
            s.write(b"\r\n")
        s.write(b"\r\n")
        if data:
            s.write(data)

        l = s.readline()
        l = l.split(None, 2)
        # print(l)
        status = int(l[1])
        while True:
            l = s.readline()
            if not l or l == b"\r\n":
                break
            # print(l)
            if l.startswith(b"Transfer-Encoding:"):
                if b"chunked" in l:
                    raise ValueError("Unsupported " + l)
            elif l.startswith(b"Location:"):
                raise NotImplementedError("Redirects not yet supported")
    except OSError:
        s.close()
        raise

I don't know enough to understand everything but am trying to find the problem.
Does CircuitPython use any of the MicroPython libraries??

[anecdata]

Generally not, as-is. The core CircuitPython C code is forked from MicroPython but has some key changes in architecture in parts of the system. Adafruit CircuitPython libraries are often written from scratch, but also modified from MicroPython or other open source libraries. Comments in adafruit_requests.py indicate its origins were from MicroPython, but by now it has been modified extensively.

[wavesailor]

I feel this adafruit_requests.py is really in need of some TLC. There are other strange issues that I've come across with it - an others (#62

I have the code running on MU - @anecdata are you using MU to debug? I'm keen to try and debug it some more.

[anecdata]

@wavesailor I use different apps for editing and serial console, but I don't think the actual choice of apps matters much, whatever works best for you.

[wavesailor]

I think this issues is somehow related: #104

I use his code and it half works now - I don't understand it all

[MarkTsengTW]

@wavesailor Glad you're making some progress. I'm working on a pull request for this but have already noticed a mistake in that code - the else block is indented one tab too far. That could be mangling your headers.

[MarkTsengTW]

@wavesailor Just finished my pull request. If you're using my code I suggest changing it to:

                if title == "set-cookie" and title in self._headers:
                    self._headers[title] = self._headers[title] + ", " + content
                else:
                    self._headers[title] = content

[wavesailor]

Thanks @MarkTsengTW ... the funny thing is the code original code you put forward works first time around for me but the second time it causes an error.

You new updated code causes the same error the stock-standard adafruit_requests.py gives me

I hoping that this information may help one of the really clever circuit python guys figure out the issue in adafruit_requests.py

[MarkTsengTW]

@wavesailor I'm in stitches! That's hilarious. If it helps, I'm pretty sure the control flow in my original code was handling cookies correctly but deleting all other headers. Perhaps that's a clue to your problem.

[dhalbert]

I recently fixed some issues with socket.recv_into() so make you all are you using the latest version of the library. If you have a potential fix a PR would be welcome!

EDIT: sorry, this fix was for ESP32SPI, not for native ESP32-S2 wifif.

[wavesailor]

Hi @dhalbert ,

I've been trying the latest versions but I'm still running into the same issue.

Fetching and updating tasks
Retrieving data...Unexpected err=invalid syntax for integer with base 16, type(err)=<class 'ValueError'>
Traceback (most recent call last):
  File "code.py", line 108, in <module>
  File "adafruit_portalbase/network.py", line 505, in fetch
  File "adafruit_requests.py", line 723, in get
  File "adafruit_requests.py", line 712, in request
  File "adafruit_requests.py", line 656, in request
  File "adafruit_requests.py", line 337, in close
ValueError: invalid syntax for integer with base 16

Code done running.

I downloaded and used these versions - Adafruit CircuitPython 7.3.0 on 2022-05-23; Adafruit MagTag with ESP32S2:

adafruit-circuitpython-bundle-7.x-mpy-20220621.zip
adafruit-circuitpython-adafruit_magtag_2.9_grayscale-en_US-7.3.0.uf2

[dhalbert]

Just to be clear, you replaced all copies of adafruit_requests.mpy or .py with the .mpy from the latest bundle? Make sure there is no .py version, and that there is no version in the top-level directory of CIRCUITPY.

[klocs]

I've been looking into this issue too and I've been able to reproduce it on MagTag with CP 7.3.0 and on Blinka 8.0.2 both with adafruit_requests.py 1.12.0. I've found a code change that appears to make it work (at least it doesn't error out) but I wasn't sure I found the real root cause.

So I set up a unit test in Adafruit_CircuitPython_Requests that mocks the call to Google servers and step through the code. But it doesn't seem to reproduce in a unit test (yet.)

Dan, do you have a commit id for the change to socket.recv_into() you mentioned? I want to make sure I am working with the latest code.

[dhalbert]

Dan, do you have a commit id for the change to socket.recv_into() you mentioned? I want to make sure I am working with the latest code.

Sorry, the fix I mentioned above was for ESP32SPI (AirLift), not for native ESP32-S2 wifi. I have been looking at several wifi issues and got them confused. The ESP32SPI error caused a chunking error in adafruit_requests, but that was higher up, and did not reflect an error I found in adafruit_requests specifically.

What is the code change you made to make it work?

[klocs]

This change seems to work OK but I haven't done much testing and I'm not sure why self._throw_away() isn't working when the redirect includes a (big) chunked response body.

Note this is in Response.close().

diff --git a/adafruit_requests.py b/adafruit_requests.py
index 2225ca9..213be28 100644
--- a/adafruit_requests.py
+++ b/adafruit_requests.py
@@ -332,13 +332,9 @@ class Response:
             if self._remaining and self._remaining > 0:
                 self._throw_away(self._remaining)
             elif self._chunked:
-                while True:
-                    chunk_header = bytes(self._readto(b"\r\n")).split(b";", 1)[0]
-                    chunk_size = int(bytes(chunk_header), 16)
-                    if chunk_size == 0:
-                        break
-                    self._throw_away(chunk_size + 2)
-                self._parse_headers()
+                # read the remaining response chunks into a small temporary buffer and discard
+                buf = bytearray(32)
+                while self._readinto(buf) != 0: pass
         if self._session:
             self._session._free_socket(self.socket)  # pylint: disable=protected-access

[wavesailor]

I'm still getting an error with regards to this issue

code.py output:
Connecting to AP MyWifi
Updating time
Getting time for timezone America/New_York
Fetching and updating tasks
Retrieving data...Unexpected err=invalid syntax for integer with base 16, type(err)=<class 'ValueError'>
Traceback (most recent call last):
  File "code.py", line 108, in <module>
  File "adafruit_portalbase/network.py", line 505, in fetch
  File "adafruit_requests.py", line 723, in get
  File "adafruit_requests.py", line 712, in request
  File "adafruit_requests.py", line 656, in request
  File "adafruit_requests.py", line 337, in close
ValueError: invalid syntax for integer with base 16

Code done running.

I'm using these versions of circuit python and libraries:

adafruit-circuitpython-adafruit_magtag_2.9_grayscale-en_US-7.3.1.uf2
adafruit-circuitpython-bundle-7.x-mpy-20220704.zip

This is what the MagTag files look like

 Volume in drive D is CIRCUITPY
 Volume Serial Number is 4984-024B
 Directory of D:\
07/05/2022  03:10 PM             6,399 code.py
12/04/2016  12:18 AM    <DIR>          lib
12/04/2016  12:18 AM               114 boot_out.txt
12/02/2021  04:46 PM               390 secrets.py
03/21/2022  01:50 PM    <DIR>          bitmaps
03/21/2022  01:50 PM    <DIR>          fonts

 Directory of D:\lib

12/04/2016  12:18 AM    <DIR>          .
12/04/2016  12:18 AM    <DIR>          ..
07/05/2022  02:58 PM    <DIR>          adafruit_bitmap_font
07/05/2022  02:58 PM    <DIR>          adafruit_display_shapes
07/05/2022  02:58 PM    <DIR>          adafruit_display_text
07/05/2022  02:58 PM    <DIR>          adafruit_io
07/05/2022  02:58 PM    <DIR>          adafruit_magtag
07/04/2022  05:19 AM             1,761 simpleio.mpy
07/04/2022  05:19 AM             6,597 adafruit_miniqr.mpy
07/04/2022  05:19 AM             1,313 neopixel.mpy
07/04/2022  05:19 AM               363 adafruit_fakerequests.mpy
07/05/2022  02:58 PM    <DIR>          adafruit_minimqtt
07/05/2022  02:58 PM    <DIR>          adafruit_portalbase
07/04/2022  05:19 AM             8,796 adafruit_requests.mpy

If it helps, this is the URL I'm having issues with:
https://docs.google.com/spreadsheets/d/e/2PACX-1vTA8pXQodbEiz5idGT21YkL1Vy8waW0aAHM1uX7D4TqBq6DrUU8qXVlON1QVaWSlmoC3OBL4Iokyiyy/pub?output=tsv

[dhalbert]

@klocs Do you have any idea why your fix might not be working for @wavesailor?

[klocs]

@klocs Do you have any idea why your fix might not be working for @wavesailor?

Let me take a look.

[klocs]

@wavesailor @dhalbert I'm pretty sure the problem is that 7.3.1 has the older version of adafruit_requests frozen in and it was released 13 days ago. I confirmed this on my magtag with CircuitPython 7.3.1 and library bundle 20220704.

Until there is a new release of CP for MagTag, you can copy the adafruit_requests.mpy from lib/ to the root directory (same directory as code.py). Make sure to do a hard reboot after copying the file. Soft reboot and code reload didn't seem to pick up the file change for me.

I tested it and it works for me™. 😉

[wavesailor]

@klocs Thank you!!!!
@dhalbert It finally works.

I also didn't know that adafruit_requests is baked into CircuitPython. Do you have to even include it in the lib/ folder then? What else is baked in?

[klocs]

@wavesailor You can see what modules are included by typing help("modules") at the REPL prompt >>>.

[dhalbert]

Ahh, thanks! I'd forgotten that adafruit_requests is frozen. @wavesailor just to be clear, the search order is:

>>> import sys
>>> sys.path
['', '/', '.frozen', '/lib']

So a frozen module overrides what is in lib/, but not what is in the top-level directory.