Add Adafruit_CircuitPython_Requests Compatibility #7

brentru · 2020-03-04T15:58:25Z

Updated _socket for compatibility with Adafruit_CircuitPython_Requests library
Added: NEW EXAMPLE to post to Adafruit IO using Requests
Fixed: _socket readline not parsing multiple lines correctly
Fixed: _socket's recv for UDP packets
Fixed: adafruit_wiznet5k_dhcp w.r.t incorrect address parsing
Updated wiznet5k_simpletest to mirror https://github.com/adafruit/Adafruit_CircuitPython_ESP32SPI/blob/master/examples/esp32spi_simpletest.py

sprinkled some gc.collect()'s where i could in adafruit_wiznet5k_dhcp

  master.:
  GC Allocated:  34528
  GC Freed:  125088
  P/R
  GC Allocated:  33168
  GC Freed:  126448

Tested on: Adafruit CircuitPython 5.0.0-beta.5 on 2020-02-05; Adafruit Feather M4 Express with samd51j19

Test output: wiznet5k_simpletest

code.py output:
Wiznet5k WebClient Test
Chip Version: w5500
MAC Address: ['0xde', '0xad', '0xbe', '0xef', '0xfe', '0xed']
My IP address is: SNIP!
IP lookup adafruit.com: 104.20.39.240
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)

----------------------------------------

Fetching json from http://api.coindesk.com/v1/bpi/currentprice/USD.json
----------------------------------------
{'time': {'updated': 'Mar 4, 2020 15:46:00 UTC', 'updatedISO': '2020-03-04T15:46:00+00:00', 'updateduk': 'Mar 4, 2020 at 15:46 GMT'}, 'disclaimer': 'This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org', 'bpi': {'USD': {'code': 'USD', 'description': 'United States Dollar', 'rate_float': 8732.83, 'rate': '8,732.8350'}}}
----------------------------------------
Done!

Test: requests_simpletest.py (https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/master/examples/requests_simpletest.py), modified for eth. object instead of esp32spi object.

Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
Text Response:  This is a test of Adafruit WiFi!
If you can read this, its working :)

----------------------------------------
Fetching JSON data from http://httpbin.org/get
----------------------------------------
JSON Response:  {'url': 'http://httpbin.org/get', 'headers': {'User-Agent': 'Adafruit CircuitPython', 'Host': 'httpbin.org', 'X-Amzn-Trac
e-Id': 'Root=1-5e5fce66-ca1da55eca173bfe7905aabe'}, 'args': {}, 'origin': '73.38.253.75'}
----------------------------------------
POSTing data to http://httpbin.org/post: 31F
----------------------------------------
Data received from server: 31F
----------------------------------------
POSTing data to http://httpbin.org/post: {'Date': 'July 25, 2019'}
----------------------------------------
JSON Data received from server: {'Date': 'July 25, 2019'}
----------------------------------------

…ting query types, TODO: same approach for ans. response

anecdata · 2020-03-05T00:30:05Z

Not sure if I've pieced together all the right things, but...

My boot_out says Adafruit CircuitPython 5.0.0-beta.5 on 2020-02-05; Adafruit Feather M4 Express with samd51j19 but I've updated several times. os.uname says: (sysname='samd51', nodename='samd51', release='5.0.0', version='5.0.0 on 2020-03-02', machine='Adafruit Feather M4 Express with samd51j19')
+
CP Library bundle: adafruit-circuitpython-bundle-5.x-mpy-20200304
+
brentru / Adafruit_CircuitPython_Wiznet5k library: https://github.com/brentru/Adafruit_CircuitPython_Wiznet5k/tree/update-for-requests

Running example code from
https://github.com/brentru/Adafruit_CircuitPython_Wiznet5k/blob/update-for-requests/examples/wiznet5k_simpletest.py

with debug=True ...

Wiznet5k WebClient Test
* Initializing DHCP
*** Get socket
Allocated socket #0:68
*** Connecting: Socket# 0, conn_mode: 2
*** Opening socket 0
w5k socket begin, protocol=2, port=67, ip=255.255.255.255
* DHCP: Discover
* DHCP: Parsing OFFER
* socket_available called with protocol 2
* socket_available called with protocol 2
* socket_available called with protocol 2
Bytes avail. on sock:  556
         * Processing 8 bytes of data
* UDP Bytes Remaining:  548
Bytes avail. on sock:  548
         * Processing 548 bytes of data
* UDP Bytes Remaining:  0
DHCP Response:  b'\x02\x01\x06\x00\xca\x02\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\xc0\xa8\x03\xdb\x00\x00\x00\x00\x00\x00\x00\x00\xde\xad\xbe\xef\xfe\xed\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00c\x82Sc5\x01\x026\x04\xc0\xa8\x04\x013\x04\x00\x01Q\x80\x01\x04\xff\xff\xf8\x00\x03\x04\xc0\xa8\x04\x01\x06\x08\x01\x01\x01\x01\x01\x00\x00\x01\x8a\x04\xc0\xa8\x04\x01\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
* DHCP: Request
Traceback (most recent call last):
  File "code.py", line 17, in <module>
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k.py", line 186, in __init__
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k.py", line 200, in set_dhcp
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k_dhcp.py", line 361, in request_dhcp_lease
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k_dhcp.py", line 210, in send_dhcp_message
NotImplementedError: array/bytes required on right side```

anecdata · 2020-03-05T01:39:53Z

With commit 7db5a2c trace is now:

Traceback (most recent call last):
  File "code.py", line 17, in <module>
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k.py", line 186, in __init__
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k.py", line 200, in set_dhcp
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k_dhcp.py", line 359, in request_dhcp_lease
  File "/lib/adafruit_wiznet5k/adafruit_wiznet5k_dhcp.py", line 322, in parse_dhcp_response
IndexError: bytes index out of range

(all preceding is same)

There's an end mark on the discovery, but not on offer (at least on wikipedia). It's a reputable router, whatever it's sending I would expect to be legit.

brentru · 2020-03-05T14:11:18Z

@anecdata CircuitPython doesn't have a native SSL client, nina-fw uses a mbedTLS implementation. I'll add a commit to make that error more readable.

EDIT: fixed in 088421e,

AssertionError: Error: SSL/TLS is not currently supported by CircuitPython.

I haven't been able to replicate the ValueError you're getting, though.

anecdata · 2020-03-05T15:04:50Z

Looks good to me. Got the friendlier AssertionError for TLS. Other errors are rare, seem similar to communication glitches on esp. I'd expect to catch exceptions and handle in user code. If any patterns emerge, I'll report back.

CircuitPython takes flight on Cat5e.

jerryneedell

tried on feather_nrf52840 with eathernet featherwing

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.0.0-15-ge98991fd7 on 2020-03-03; Adafruit Feather nRF52840 Express with nRF52840
> import wiznet5k_simpletest
> Wiznet5k WebClient Test
> Chip Version: w5500
> MAC Address: ['0xde', '0xad', '0xbe', '0xef', '0xfe', '0xed']
> My IP address is: 192.168.2.32
> IP lookup adafruit.com: 104.20.39.240
> Fetching text from http://wifitest.adafruit.com/testwifi/index.html
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "wiznet5k_simpletest.py", line 30, in <module>
>   File "adafruit_requests.py", line 280, in get
>   File "adafruit_requests.py", line 242, in request
>   File "adafruit_requests.py", line 194, in request
>   File "adafruit_wiznet5k/adafruit_wiznet5k_socket.py", line 149, in connect
>   File "adafruit_wiznet5k/adafruit_wiznet5k.py", line 520, in socket_connect
> RuntimeError: Failed to establish connection.
> >>>

enabled debug at @brentru request
https://gist.github.com/jerryneedell/48367e4c0cc05fe11d4ce2be4d5ac104

brentru · 2020-03-05T15:50:37Z

@jerryneedell seems it's closing the socket immediately after opening it, which causes the failure.

Your trace:

*** Opening socket 0
w5k socket begin, protocol=33, port=80, ip=104.236.193.178
*** Closing socket #0
Traceback (most recent call last):

My trace:

*** Get socket
Allocated socket #0:1024
*** Connecting: Socket# 0, conn_mode: 33
*** Opening socket 0
w5k socket begin, protocol=33, port=80, ip=104.236.193.178
* socket_available called with protocol 33
* socket_available called with protocol 33
* socket_available called with protocol 33
* socket_available called with protocol 33
* socket_available called with protocol 33
Bytes avail. on sock:  310
         * Processing 310 bytes of data
----------------------------------------
*** Closing socket #0
This is a test of Adafruit WiFi!

brentru · 2020-03-05T16:30:06Z

@jerryneedell Figured it out - code was sometimes re-using a registered port number for TCP sockets, which was causing the server to send an ACK-RST back to the device disconnecting it. 406c7fb switches to randomly assigned dynamic port numbers.

jerryneedell · 2020-03-05T16:40:09Z

tried with same result -- updated gist https://gist.github.com/jerryneedell/48367e4c0cc05fe11d4ce2be4d5ac104

…, not set anything

siddacious

Works for me!

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Wiznet5k WebClient Test
Chip Version: w5500
MAC Address: ['0xde', '0xad', '0xbe', '0xef', '0xfe', '0xed']
My IP address is: 192.168.11.145
IP lookup adafruit.com: 104.20.39.240
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)

----------------------------------------

Fetching json from http://api.coindesk.com/v1/bpi/currentprice/USD.json
----------------------------------------
{'time': {'updated': 'Mar 5, 2020 21:22:00 UTC', 'updatedISO': '2020-03-05T21:22:00+00:00', 'updateduk': 'Mar 5, 2020 at 21:22 GMT'}, 'disclaimer': 'This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org', 'bpi': {'USD': {'code': 'USD', 'description': 'United States Dollar', 'rate_float': 9147.05, 'rate': '9,147.0517'}}}
----------------------------------------
Done!



Press any key to enter the REPL. Use CTRL-D to reload.

brentru · 2020-03-05T21:24:49Z

@jerryneedell Could you test again if you have time?

jerryneedell · 2020-03-05T21:28:30Z

still not working -


Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Wiznet5k WebClient Test
Chip Version: w5500
MAC Address: ['0xde', '0xad', '0xbe', '0xef', '0xfe', '0xed']
My IP address is: 192.168.2.32
IP lookup adafruit.com: 104.20.39.240
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
Traceback (most recent call last):
  File "code.py", line 30, in <module>
  File "adafruit_requests.py", line 283, in get
  File "adafruit_requests.py", line 245, in request
  File "adafruit_requests.py", line 194, in request
  File "adafruit_wiznet5k/adafruit_wiznet5k_socket.py", line 149, in connect
  File "adafruit_wiznet5k/adafruit_wiznet5k.py", line 515, in socket_connect
RuntimeError: Failed to establish connection.



Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.0.0-alpha.4-127-g22b7050c4 on 2019-10-03; Adafruit Feather M4 Express with samd51j19

jerryneedell · 2020-03-05T21:39:32Z

hmmm --- FYI -- the example/library from the master branch is also not workig for me -- it just hangs after displaying the IP address

jerryneedell · 2020-03-05T21:50:27Z

it is working for me now

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.0.0-alpha.4-127-g22b7050c4 on 2019-10-03; Adafruit Feather M4 Express with samd51j19
>>> 
>>> 
>>> import nodebug
Wiznet5k WebClient Test
Chip Version: w5500
MAC Address: ['0xde', '0xad', '0xbe', '0xef', '0xfe', '0xed']
My IP address is: 192.168.2.32
IP lookup adafruit.com: 104.20.39.240
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)

----------------------------------------

Fetching json from http://api.coindesk.com/v1/bpi/currentprice/USD.json
----------------------------------------
{'time': {'updated': 'Mar 5, 2020 21:48:00 UTC', 'updatedISO': '2020-03-05T21:48:00+00:00', 'updateduk': 'Mar 5, 2020 at 21:48 GMT'}, 'disclaimer': 'This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org', 'bpi': {'USD': {'code': 'USD', 'description': 'United States Dollar', 'rate_float': 9125.52, 'rate': '9,125.5233'}}}
----------------------------------------
Done!

jerryneedell · 2020-03-05T21:52:59Z

also test on feather_nrf52840 -- works

jerryneedell

tested on feather_m4_express and feather_nrf52840_express with ethernet_featherwing -- both work now.

brentru · 2020-03-06T14:34:47Z

Tested working on Metro M4 Express w/Ethernet Shield (https://www.adafruit.com/product/2971)
Adafruit CircuitPython 5.0.0 on 2020-03-02; Adafruit Metro M4 Express with samd51j19

Wiznet5k WebClient Test
Chip Version: w5500
MAC Address: ['0xde', '0xad', '0xbe', '0xef', '0xfe', '0xed']
My IP address is:
IP lookup adafruit.com: 104.20.39.240
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)

----------------------------------------

Fetching json from http://api.coindesk.com/v1/bpi/currentprice/USD.json
----------------------------------------
{'time': {'updated': 'Mar 6, 2020 14:33:00 UTC', 'updatedISO': '2020-03-06T14:33:00+00:00', 'updateduk': 'Mar 6, 2020 at 14:33 GMT'}, 'disclaimer': 'This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org', 'bpi': {'USD': {'code': 'USD', 'description': 'United States Dollar', 'rate_float': 9115.01, 'rate': '9,115.0117'}}}
----------------------------------------
Done!

brentru added 12 commits March 3, 2020 11:46

add some updates for Adafruit_Requests

eae6025

return buffer and firstline properly

af4a485

only add to buffer if bytes are avail on the socket

3b0a686

fix incorrect read within dns, add correct join in socket

2da4616

add query count

b0a1cdc

parse based on answer instead of fixed buff size lazy parsing

28d1c7b

lintin

d17a9ff

rebuild docx!

be5eac4

ESP32SPI_Simpletest-style-test

29eb7b5

correct parse, TODO: guards

bc82d38

lintin

32f18e0

add checks for types

4ca248d

brentru requested a review from a team March 4, 2020 15:58

brentru added 13 commits March 4, 2020 11:08

modify readme for new example, spnx test buildin docs

ea7f98d

throw assertionerror if dns was failed to resolve

39385ad

add resolved ip for debugging

5d96a2d

debugging for DHCP server

42a2faf

add debug iface to DNS

6dc07ae

add way more verbose debugging within dns

4580e6d

trying a different approach for parsing out the query name and valida…

d65fc58

…ting query types, TODO: same approach for ans. response

Better parsing for answer

2043d7d

remove gc

08bb7c9

add ip to socket begin

cb352dd

add debug to dhcp

9ec9ed0

add old params

ec49329

remove cruft, new impl

e0b3d23

brentru added 2 commits March 4, 2020 19:56

run buf til pointer is at end

7d7ef0e

add more verbose output for parsing DHCP requests if debug

7db5a2c

add TLS_mode conntype, assertion for connecting to HTTPS

088421e

jerryneedell reviewed Mar 5, 2020

View reviewed changes

use dynamic port numbers 41952-65535, randomize to prevent reuse!

406c7fb

switch to wait for sncr to clear?

4467ffe

brentru added 6 commits March 5, 2020 11:46

wait less time between checking sn_sr

1eb1a51

read and write to/from subnet registers better in ifconfig setter/getter

ebf2d0e

testing port configuration tweaks

7df14f1

prune extra from get_socket, just allocate a socket from the chipset

74e6d7b

continue separating functionality: socket_open should only issue OPEN…

4030144

…, not set anything

fix indentation for lintin

dab3551

siddacious approved these changes Mar 5, 2020

View reviewed changes

jerryneedell approved these changes Mar 6, 2020

View reviewed changes

brentru mentioned this pull request Mar 6, 2020

Add Simpletest for Ethernet Library adafruit/Adafruit_CircuitPython_Requests#22

Merged

brentru added 3 commits March 6, 2020 12:11

add Adafruit IO example!

55be280

add example for manually setting the network configuration

30569bd

fix newline

f510eb2

ladyada approved these changes Mar 6, 2020

View reviewed changes

brentru merged commit 5cd1e5b into adafruit:master Mar 6, 2020

brentru deleted the update-for-requests branch March 6, 2020 19:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Adafruit_CircuitPython_Requests Compatibility #7

Add Adafruit_CircuitPython_Requests Compatibility #7

brentru commented Mar 4, 2020 •

edited

Loading

anecdata commented Mar 5, 2020 •

edited

Loading

anecdata commented Mar 5, 2020 •

edited

Loading

brentru commented Mar 5, 2020 •

edited

Loading

anecdata commented Mar 5, 2020

jerryneedell left a comment

brentru commented Mar 5, 2020

brentru commented Mar 5, 2020

jerryneedell commented Mar 5, 2020

siddacious left a comment

brentru commented Mar 5, 2020 •

edited

Loading

jerryneedell commented Mar 5, 2020

jerryneedell commented Mar 5, 2020

jerryneedell commented Mar 5, 2020 •

edited

Loading

jerryneedell commented Mar 5, 2020

jerryneedell left a comment

brentru commented Mar 6, 2020

Add Adafruit_CircuitPython_Requests Compatibility #7

Add Adafruit_CircuitPython_Requests Compatibility #7

Conversation

brentru commented Mar 4, 2020 • edited Loading

anecdata commented Mar 5, 2020 • edited Loading

anecdata commented Mar 5, 2020 • edited Loading

brentru commented Mar 5, 2020 • edited Loading

anecdata commented Mar 5, 2020

jerryneedell left a comment

Choose a reason for hiding this comment

brentru commented Mar 5, 2020

brentru commented Mar 5, 2020

jerryneedell commented Mar 5, 2020

siddacious left a comment

Choose a reason for hiding this comment

brentru commented Mar 5, 2020 • edited Loading

jerryneedell commented Mar 5, 2020

jerryneedell commented Mar 5, 2020

jerryneedell commented Mar 5, 2020 • edited Loading

jerryneedell commented Mar 5, 2020

jerryneedell left a comment

Choose a reason for hiding this comment

brentru commented Mar 6, 2020

brentru commented Mar 4, 2020 •

edited

Loading

anecdata commented Mar 5, 2020 •

edited

Loading

anecdata commented Mar 5, 2020 •

edited

Loading

brentru commented Mar 5, 2020 •

edited

Loading

brentru commented Mar 5, 2020 •

edited

Loading

jerryneedell commented Mar 5, 2020 •

edited

Loading