Adding MQTT support for Azure IoT Central and IoT Hub #11

jimbobbennett · 2020-04-12T21:01:34Z

An implementation of #7.

Azure has 2 IoT Services:

Azure IoT Hub - this provides connectivity between devices and Azure
Azure IoT Central - this is an IoT software-as-a-service platform that sits on top of Azure IoT Hub

Currently the code is working based off copying it to a PyBadge with Airlift Featherwing and supports all the device capabilities of IoT Hub and Central:

IoT Hub

Device to cloud messages
Cloud to device messages
Direct Methods
Updating twin reported properties
Receiving twin desired properties

IoT Central

Telemetry
Receiving and setting properties
Commands

The code structure is:

device_registration.py - contains code to register a device with IoT Central and get the underlying IoT Hub connection details
iot_mqtt.py - the underlying MQTT communications layer. This talks to IoT Hub and can send device to cloud, receive direct methods, receive cloud to device messages, update the device twin and receive twin updates
iothub_device.py - the client for IoT Hub connections. This will connect to a hub via a device connection string, send device to cloud messages, update the twin and has callbacks you can set up for direct methods, cloud to device messages and twin updates
iotcentral_device.py - the client for IoT Central connections. This will connect to IoT Central via the id scope, device id and key using the device registration service. It can send telemetry, update properties and has callbacks for commands and property updates

I've removed the HTTP SDK as MQTT provides better capabilities. The HTTP SDK also had both device and client methods on it, the MQTT implementation is device only. If we want to add a client SDK with features such as querying for devices, then that should be a different library, to be in keeping with the official Azure IoT SDKs.

…Python_AzureIoT

Also adding support for B tier hubs by lazy subscribing to twins

brentru · 2020-04-16T21:14:58Z

@jimbobbennett When you're ready for a review, please request me to review it or tag me. Thanks for continuing working on this.

jimbobbennett · 2020-04-16T21:16:47Z

@jimbobbennett When you're ready for a review, please request me to review it or tag me. Thanks for continuing working on this.

Will do. I just wanted to open it early to get the code reviewed by others, including some kind folks from the Azure IoT SDK team who have given me some pointers.

Appreciate all your help and support with this.

jimbobbennett · 2020-04-17T02:47:56Z

@brentru - this should be ready for review now. All the code appears to work, and I've tested all the examples with no issues. I don't seem to be able to set a reviewer, hence tagging you in.

brentru

@jimbobbennett This is awesome!!

Requesting some changes. I'll test on a PyPortal Titano after.

adafruit_azureiot/constants.py

brentru · 2020-04-17T14:16:09Z

adafruit_azureiot/device_registration.py

@@ -0,0 +1,235 @@
+"""


Could you include the same header used in __init__.py here? (reference: https://github.com/adafruit/Adafruit_CircuitPython_Wiznet5k/tree/master/adafruit_wiznet5k)

File headers added - can you verify they are ok please.

Look good, may want to update the copyright to be current:
# Copyright (c) 2019 Jim Bennett
to
# Copyright (c) 2020 Jim Bennett

D'oh! Will fix that....

brentru · 2020-04-17T14:29:00Z

adafruit_azureiot/iot_mqtt.py

+import json
+import time
+import adafruit_esp32spi.adafruit_esp32spi_socket as socket
+from adafruit_esp32spi.adafruit_esp32spi_wifimanager import ESPSPI_WiFiManager


Instead of importing esp32spi and the esp32spi socket, could we the socket object in instead?

This will allow us to use multiple types of socket objects with this library like wifi, ethernet, and cellular.

We do this here in the latest MiniMQTT: https://github.com/adafruit/Adafruit_CircuitPython_MiniMQTT/blob/master/examples/minimqtt_simpletest_eth.py#L64

Yup - I was following the pattern of the existing code, relying on the WiFi manager. Happy to change it to a socket instead!

That'd be great. Let me know if you have any Q's. I'll test with ESP32 and Ethernet sockets.

Updated to use the socket and interface

adafruit_azureiot/iot_mqtt.py

examples/iotcentral_commands.py

brentru · 2020-04-17T14:36:23Z

examples/iotcentral_commands.py

+
+while True:
+    # Poll every second for messages from the cloud
+    device.loop()


You may want to handle network (physical, not mqtt) disconnection from the esp here. ESP32 can be flaky at times.

https://github.com/adafruit/Adafruit_CircuitPython_ESP32SPI/blob/master/examples/esp32spi_aio_post.py#L66

Added exception catching and reconnection

brentru · 2020-04-17T14:37:30Z

examples/iotcentral_simpletest.py

+
+message_counter = 60
+
+while True:


may want to handle esp32 network and status issues in here.

Would that be too much for an example? The example is meant to show the simple case of sending telemetry.

It would increase the length of the example, however, adding network handling in the user-code is important to make sure new users have a good "first five minutes" and don't hit errors they can't resolve (the ESP32 can be flaky).

In adding this, I'm also going to add reconnect logic. In doing this I hit another bug in MiniMQTT - the reconnect method gets stuck in an infinite loop. Bug here

Added reconnection on errors

brentru · 2020-04-17T14:38:50Z

requirements.txt

@@ -1,2 +1,6 @@
 Adafruit-Blinka
-Adafruit_CircuitPython_ESP32SPI


Same as before - I'd like to not add a requirement for esp32spi with this library to allow for new transport sockets in the future.

jimbobbennett · 2020-04-17T21:46:16Z

@jimbobbennett This is awesome!!

Requesting some changes. I'll test on a PyPortal Titano after.

All changes should be in, let me know what you think.

ladyada · 2020-04-19T01:32:45Z

@jimbobbennett this is so awesome, thank you for the PR - we're really psyched to make Azure easier for folks to use. would you be interested in writing a guide/tutorial or publishing a project to show this off?

jimbobbennett · 2020-04-19T20:55:39Z

@jimbobbennett this is so awesome, thank you for the PR - we're really psyched to make Azure easier for folks to use. would you be interested in writing a guide/tutorial or publishing a project to show this off?

Yes, definitely! I'll be building a few hands on workshops in my own GitHub repo to show this in action, and we are hoping to run some kids events in Microsoft spaces using Adafruit boards later on in the year when things open up and we can get some hardware.

Would it be possible for me to create a guide for the Adafruit Learning System? I'm also building a CircuitPython library for one of our AI services which I'd love to build a guide for once it's complete.

ladyada · 2020-04-19T21:35:30Z

oh this is so cool - yes Anne B (@TheKitty) will be happy to help you get set up on Learn to author a guide. You can model the guide off of @brentru 's Azure starter guide (you can also use it as a prereq so there's less duplication)

brentru · 2020-04-22T14:19:00Z

@jimbobbennett What does your secrets.py file look like for the simpletest? (Please strip out all tokens/private information before posting)

jimbobbennett · 2020-04-22T17:04:07Z

Here you go:

Wifi details are self explanatory.

For the IoT Central connection details, create a device in IoT Central and grab the details from the Connect dialog.

For the IoT Hub device connection string, create an IoT Hub, create a device in that hub and grab one of the connection strings for the device.

# This file is where you keep secret settings, passwords, and tokens!
# If you put them in the code you risk committing that info or sharing it
# which would be not great. So, instead, keep it all in this one file and
# keep it a secret.

"""
Contains the secrets for your app including WiFi connection details.
DO NOT CHECK THIS INTO SOURCE CODE CONTROL!!!!11!!!
"""

secrets = {
    # WiFi connection details
    "ssid": "",
    "password": "",

    # IoT Central device connection details
    "id_scope": "",
    "device_id": "",
    "key": "",

    # IoT Hub device connection string
    "device_connection_string": "",
}

jimbobbennett · 2020-04-22T23:02:46Z

The last commit added one small update to the reconnect logic to replace the workarounds for two MiniMQTT bugs.

jpecor · 2020-04-23T20:48:22Z

@jimbobbennett - thanks for all the work on this! @brentru pointed me to it earlier today in response to some questions I had about the AzureIoT library.

I was able to get the iothub_messages.py example to work, and I'm seeing data in the Azure cloud shell.

I did have to install several additional libraries from your GitHub repos for it to work:

circuitpython_base64
circuitpython_hmac
circuitpython_parse

Will these eventually get folded into the final library in some way?

jimbobbennett · 2020-04-23T20:51:15Z

@jimbobbennett - thanks for all the work on this! @brentru pointed me to it earlier today in response to some questions I had about the AzureIoT library.

I was able to get the iothub_messages.py example to work, and I'm seeing data in the Azure cloud shell.

I did have to install several additional libraries from your GitHub repos for it to work:

circuitpython_base64

circuitpython_hmac

circuitpython_parse

Will these eventually get folded into the final library in some way?

These are available in the CircuitPython community bundle. This is in the Readme, but I probably should add a note to all the examples as well.

jpecor · 2020-04-23T20:53:30Z

Ah! Sure enough. Ok, thanks!

brentru · 2020-04-29T21:42:52Z

@jimbobbennett Ok, I looked through the code, didnt see the connection string started with HostName=. Looks great!

mu

azure

Ok - one final ask before I pull this in and release: Could you please add the secrets file to examples as azureiot_secrets_example.py. Please include example values for what the id_scope, device_id, key and device_connection_string values should look like so users don't get confused like I did.

jimbobbennett · 2020-04-29T23:48:44Z

@brentru - added more info, a secrets file and some images in the readme to show where to get the values from. Took a few commits because I couldn't get the rest syntax right, then PyLint became case sensitive for ignores where it wasn't before, then couldn't get sphinx working! all good fun.

brentru · 2020-04-30T13:06:17Z

@jimbobbennett The library check process is a bit tricky. But it looks perfect now.

Updating https://github.com/adafruit/Adafruit_CircuitPython_NeoPixel to 6.0.0 from 5.1.1: > Merge pull request adafruit/Adafruit_CircuitPython_NeoPixel#83 from adafruit/tannewt-patch-1 > Merge pull request adafruit/Adafruit_CircuitPython_NeoPixel#82 from dunkmann00/pixelbuf Updating https://github.com/adafruit/Adafruit_CircuitPython_NeoPixel_SPI to 0.8.0 from 0.4.1: > Merge pull request adafruit/Adafruit_CircuitPython_NeoPixel_SPI#15 from dunkmann00/pypixelbuf-2.0-compat Updating https://github.com/adafruit/Adafruit_CircuitPython_AzureIoT to 2.0.0 from 1.1.0: > Merge pull request adafruit/Adafruit_CircuitPython_AzureIoT#11 from jimbobbennett/master > build.yml: add black formatting check Updating https://github.com/adafruit/Adafruit_CircuitPython_Pypixelbuf to 2.0.0 from 1.0.3: > Merge pull request adafruit/Adafruit_CircuitPython_Pypixelbuf#19 from dunkmann00/dotstar_brightness > Merge pull request adafruit/Adafruit_CircuitPython_Pypixelbuf#18 from dunkmann00/_pixelbuf_cp > Merge pull request adafruit/Adafruit_CircuitPython_Pypixelbuf#17 from adafruit/black-update > build.yml: add black formatting check > Merge pull request adafruit/Adafruit_CircuitPython_Pypixelbuf#16 from kattni/update-example-name > Merge pull request adafruit/Adafruit_CircuitPython_Pypixelbuf#12 from adafruit/pylint-update > update code of coduct: discord moderation contact section > Merge pull request adafruit/Adafruit_CircuitPython_Pypixelbuf#11 from sommersoft/patch_coc > update pylintrc for black

Jim Bennett added 14 commits April 11, 2020 14:27

Copying in code from the external repo

c384d42

Merge branch 'master' of https://github.com/adafruit/Adafruit_Circuit…

1fb4de0

…Python_AzureIoT

Upping line length

0a01591

Black formatting

4f6b3fe

Update iotcentral_simpletest.py

89db973

Addin exports to __init__

707a7ed

Passing wifi manager around to set the socket

5bd2f83

Added IoT Hub examples

359adcf

Added iot central examples

0a7d52c

Trying to get sphinx to not error

5cbd59e

Update conf.py

f683715

Makin examples work

8b3baba

Fixing based of code reivew comments

0d85cd2

Also adding support for B tier hubs by lazy subscribing to twins

Starting to add readme documentation

bdece0b

jimbobbennett marked this pull request as draft April 16, 2020 21:17

Jim Bennett added 6 commits April 16, 2020 14:34

More readme updates

79aab5a

Update README.rst

e86e9b3

Update README.rst

6c04a3e

Updating API docs

0a043d4

Setting the logging level for the MQTT client from the top level logger

45071bb

Update iot_mqtt.py

0aa0e9c

jimbobbennett changed the title ~~WIP - Adding MQTT support for Azure IoT Central and IoT Hub~~ Adding MQTT support for Azure IoT Central and IoT Hub Apr 17, 2020

jimbobbennett marked this pull request as ready for review April 17, 2020 02:47

brentru self-requested a review April 17, 2020 14:10

brentru requested changes Apr 17, 2020

View reviewed changes

Jim Bennett added 2 commits April 17, 2020 09:24

Added file headers

e235d39

Removing some rogue print statements

0c034f4

Jim Bennett added 4 commits April 17, 2020 10:33

PR review changes

e8caff8

More updates from PR feedback

5251721

Update requirements.txt

e54804e

Fixing the build script

dbc03d6

Fixing reconnect logic after bug fixes in mini MQTT

1f3d37b

Tweaked the samples to have more details on library dependencies

e1e13c0

Jim Bennett added 6 commits April 29, 2020 16:13

Adding more documentation around the keys

a9833dd

Update README.rst

b49197e

Update README.rst

9195773

Update azureiot_secrets_example.py

90c53ff

Update device_registration.py

4ef5e13

Trying to get images in both the readme and docs

571f496

brentru approved these changes Apr 30, 2020

View reviewed changes

brentru merged commit f3ff110 into adafruit:master Apr 30, 2020

		@@ -1,2 +1,6 @@
		Adafruit-Blinka
		Adafruit_CircuitPython_ESP32SPI


		message_counter = 60

		while True:

Adding MQTT support for Azure IoT Central and IoT Hub #11

Adding MQTT support for Azure IoT Central and IoT Hub #11

Conversation

jimbobbennett commented Apr 12, 2020 • edited Loading

brentru commented Apr 16, 2020

jimbobbennett commented Apr 16, 2020

jimbobbennett commented Apr 17, 2020 • edited Loading

brentru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimbobbennett commented Apr 17, 2020

ladyada commented Apr 19, 2020

jimbobbennett commented Apr 19, 2020

ladyada commented Apr 19, 2020

brentru commented Apr 22, 2020

jimbobbennett commented Apr 22, 2020

jimbobbennett commented Apr 22, 2020

jpecor commented Apr 23, 2020

jimbobbennett commented Apr 23, 2020

jpecor commented Apr 23, 2020

brentru commented Apr 29, 2020 • edited Loading

jimbobbennett commented Apr 29, 2020

brentru commented Apr 30, 2020

jimbobbennett commented Apr 12, 2020 •

edited

Loading

jimbobbennett commented Apr 17, 2020 •

edited

Loading

brentru commented Apr 29, 2020 •

edited

Loading