Skip to content

Fetch all messages in loop, non-blocking mode #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Oct 10, 2022

Conversation

calcut
Copy link
Contributor

@calcut calcut commented Sep 5, 2022

First PR, so bear with me!

Basically, I've found issues with callling loop() too frequently or too infrequently, that I'm trying to address here.
Calling too frequently (with timeout >0) can lead to a nasty MemoryError. #101
Calling too infrequently can cause a build up of messages on the broker. #108

I've changed the default timeout to 0 (and added error handling of EAGAIN)
This might be a controversial change - but it means the function is really non-blocking as the comments say it should be. (Perhaps there is a history of why it was set to 1 by default?, I'd be happy keeping it at 1 if it prevents issues for others)

I've also made it call _wait_for_msg() in a while loop, similar to how the ping() function works.
I've used the 10s _recv_timeout to make sure it can't get stuck in the loop.
This means that all waiting messages are received, not just the next in the queue.

@tekktrik tekktrik requested a review from a team September 6, 2022 15:07
@tekktrik
Copy link
Member

Do you have any code that can reproduce the issues you listed? I'm going to try to review this and didn't know if you had something like that already.

@calcut
Copy link
Contributor Author

calcut commented Sep 24, 2022

There is code in the 2 issues I linked. The MemoryError may be harder to reproduce (e.g. if it depends on connection latency etc).

I can try to help further if it still isn't clear

@FoamyGuy
Copy link
Contributor

FoamyGuy commented Oct 3, 2022

I'm not super familiar with the specifics of MQTT so I can't comment much on the details of the change. I did try it out on a device though.

Using code posted in the issue on a Feather ESP32-S2 8.0.0beta.1.

Can confirm I observed the same issue with currently released libraries, mqtt time "falling behind" due to only getting one callback per loop call and request getting out of retries problem. Eventually it "catches up" with a bunch of callbacks at once (in my case it was dozens, I didn't count exactly)

The new version of this library from the PR does seem to resolve it. With this version I am getting fairly consistently 5 mqtt callbacks per loop with the final one matching the time given by the http request.

I inclined to approve, but we should probably try out a few of the learn guide projects that use minimqtt to ensure the differing functionality with callbacks does not have adverse effects in them. Assuming no issues with them, I think this is good.

@calcut
Copy link
Contributor Author

calcut commented Oct 4, 2022

Regarding the "catching up" I figured out that happens because of the keep_alive feature. where it runs the ping() function periodically. The ping function ends up fetching all messages in a loop. So my change is to make the loop() function behave the same way

Copy link
Contributor

@FoamyGuy FoamyGuy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me.

I tested successfully with PyPortal using this learn guide: https://learn.adafruit.com/adafruit-io-a-c-power-relay/circuitpython-setup

And Feather ESP32-S2 TFT with the time subscription reproducer script from the issue. And Fun House using the native networking example inside the examples folder of this repo.

Everything appears to function as intended to me even with the new behavior that gets all messages in the queue every loop instead of only 1 most loops.

Thanks for working on this fix @calcut

@FoamyGuy FoamyGuy merged commit aec7777 into adafruit:main Oct 10, 2022
adafruit-adabot added a commit to adafruit/Adafruit_CircuitPython_Bundle that referenced this pull request Oct 11, 2022
Updating https://github.com/adafruit/Adafruit_CircuitPython_BNO055 to 5.4.7 from 5.4.6:
  > Merge pull request adafruit/Adafruit_CircuitPython_BNO055#108 from tcfranks/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_RFM69 to 2.1.13 from 2.1.12:
  > Merge pull request adafruit/Adafruit_CircuitPython_RFM69#42 from tcfranks/main

Updating https://github.com/adafruit/Adafruit_CircuitPython_DisplayIO_Layout to 1.19.9 from 1.19.8:
  > Merge pull request adafruit/Adafruit_CircuitPython_DisplayIO_Layout#76 from The-Debarghya/bug-75-zerodivisionerror

Updating https://github.com/adafruit/Adafruit_CircuitPython_Logging to 4.2.0 from 4.1.6:
  > Merge pull request adafruit/Adafruit_CircuitPython_Logging#40 from tekktrik/dev/allow-root-logger

Updating https://github.com/adafruit/Adafruit_CircuitPython_MiniMQTT to 5.5.2 from 5.5.1:
  > Merge pull request adafruit/Adafruit_CircuitPython_MiniMQTT#122 from calcut/fetch_all_messages_in_loop

Updating https://github.com/adafruit/Adafruit_CircuitPython_PYOA to 2.5.12 from 2.5.11:
  > Merge pull request adafruit/Adafruit_CircuitPython_PYOA#37 from tekktrik/dev/fix-install
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants