Receiver Disabled Every Other Save (toggling state)

[DJDevon3]

Been working with rfm95 featherwing for hours & hours over the past week and have finally narrowed down the erratic behavior to the receiver disabling itself every other save. It toggles between working and not receiving anything, on save, every time. Almost as if a rfm9x.receiver() mode state is being toggled. This might also explain some other bug reports that could account for receiver being disabled.

During issue all other code works including sending a packet. Only rfm9x.receive() gets broken.

In my example I have both transmitting and receiving on both boards. Using transmitter.py and receiver.py just helps me to differentiate them when opening USB instead of both having identical files however both are slightly different because I'm still working on the project but they do work perfectly as long as you ctrl+s at just the right time on both boards.

Project is transmitting a fake chat stream over and over by design for testing purposes. The time.monotonic timestamps change each message so I can track each original message.

Issue happens on both 7.3.3 and 8.0.beta, Currently running mixed CP versions but again issue happens regardless of CP version and I've flashed both to 7.33 and 8.0 trying to track down this bug. Also happens on an RP2040 feather. The lora boards or library is the common denominator.

Code.py

# SPDX-FileCopyrightText: 2022 DJDevon3 for Adafruit Industries
# SPDX-License-Identifier: MIT
# Code.py (1 file per board)
# RFM95 LoRa Featherwing Example
import gc
import time
import board
import busio
import digitalio
import adafruit_rfm9x

# Choose the Mode (receive or transmit)

# RECEIVER MODE (HOUSE)
# import receiver

# TRANSMITTER MODE (MAILBOX)
import transmitter

Receiver.py

# SPDX-FileCopyrightText: 2022 DJDevon3 for Adafruit Industries
# SPDX-License-Identifier: MIT
# Receiver.py
# Adafruit CircuitPython 7.3.3 on 2022-08-29; Adafruit Feather ESP32S2 with ESP32S2
import gc
import time
import board
import busio
import digitalio
import adafruit_rfm9x

CS = digitalio.DigitalInOut(board.D10)
RESET = digitalio.DigitalInOut(board.D6)
# CircuitPython build:
# CS = digitalio.DigitalInOut(board.RFM9X_CS)
# RESET = digitalio.DigitalInOut(board.RFM9X_RST)

# Define the onboard LED
LED = digitalio.DigitalInOut(board.D13)
LED.direction = digitalio.Direction.OUTPUT

# Initialize SPI bus.
spi = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)

# Initialze RFM radio
RADIO_FREQ_MHZ = 915.0  # Frequency of the radio in Mhz
rfm9x = adafruit_rfm9x.RFM9x(spi, CS, RESET, RADIO_FREQ_MHZ)

# Radio configured in LoRa mode. You can't control sync
# encryption, frequency deviation, or other settings!
rfm9x.tx_power = 13  # Transmit Power. Default 13dB. RFM95 (0-23)dB:
rfm9x.enable_crc = False
rfm9x.xmit_timeout = 2.0 # Recommended at least 2 seconds for HW reliability
with_header = True  # Set if you want header bytes printed for debug

# Node (0-255) Only packets addressed to this node will be accepted.
# Comment out to disable and receive all packets from anywhere
# rfm9x.node = 13

# Wait to receive packets.  Note that this library can't receive data at a fast
# rate, in fact it can only receive and process one 252 byte packet at a time.
# This means you should only use this for low bandwidth scenarios, like sending
# and receiving a single message at a time.
print("Waiting for packets...")

while True:
    # Time in seconds from when board was powered on
    now = time.monotonic()
    # Changing timeout and sleep duration definitely has an effect on dropped packets.
    # Timeout 5 and sleep 0.5 works great with no dropped packets in my experience.
    packet = rfm9x.receive(keep_listening=True, timeout=rfm9x.xmit_timeout, with_header=False)
    # If no packet was received during timeout then packet=None is returned.
    if packet is None:
        # Packet has not been received
        LED.value = False
        print("Packet: None - Sleep for " + str(rfm9x.xmit_timeout) + " seconds...")
    else:
        # Now we're inside a received packet!
        LED.value = True
        try:
            # Header bytes are raw bytes. Good for debugging.
            if with_header:
                packet_text = str(packet)
            # No header strips header bytes and decodes to ascii.
            if not with_header:
                packet_text = str(packet, "ascii")
        except (UnicodeError) as e:
            print("Unicode Error", e)
            continue
        
        rssi = rfm9x.last_rssi
        last_snr = rfm9x.last_snr
        # Print out the raw bytes of the packet:
        print("Received (raw bytes): {0}".format(packet))
        # And decode to ASCII text and print it too.  Note that you always
        # receive raw bytes and need to convert to a text format like ASCII
        # if you intend to do string processing on your data.  Make sure the
        # sending side is sending ASCII data before you try to decode!
        print("Timestamp:", now)
        print("Signal Noise: {0} dB".format(last_snr))
        print("Signal Strength: {0} dB".format(rssi))
        print("Received (ASCII): {0}".format(packet_text))
        gc.collect
        
    message = str("M4 Express Msg Test 😇 ")
    rfm9x.send(bytes(str(now) + " " + message + "\r\n", "utf-8"), keep_listening=True)
    print("(Sent)" + " " + message + "\n")
    time.sleep(2)

Transmitter.py

# SPDX-FileCopyrightText: 2022 DJDevon3 for Adafruit Industries
# SPDX-License-Identifier: MIT
# Transmitter.py
# Adafruit CircuitPython 8.0.0-alpha.1 on 2022-06-09; Adafruit Feather M4 Express with samd51j19
import gc
import time
import board
import busio
import digitalio
import adafruit_rfm9x

# Define pins connected to the chip, use these if wiring up the breakout according to the guide:
CS = digitalio.DigitalInOut(board.D10)
RESET = digitalio.DigitalInOut(board.D6)
# Or uncomment and instead use these if using a Feather M0 RFM9x board and the appropriate
# CircuitPython build:
# CS = digitalio.DigitalInOut(board.RFM9X_CS)
# RESET = digitalio.DigitalInOut(board.RFM9X_RST)

# Define the onboard LED
LED = digitalio.DigitalInOut(board.D13)
LED.direction = digitalio.Direction.OUTPUT

# Initialize SPI bus.
spi = busio.SPI(board.SCK, MOSI=board.MOSI, MISO=board.MISO)

# Initialze RFM radio
RADIO_FREQ_MHZ = 915.0  # Frequency of the radio in Mhz
rfm9x = adafruit_rfm9x.RFM9x(spi, CS, RESET, RADIO_FREQ_MHZ)

# Radio configured in LoRa mode. You can't control sync
# encryption, frequency deviation, or other settings!
rfm9x.tx_power = 13  # Transmit Power. Default 13dB. RFM95 (0-23)dB:
rfm9x.enable_crc = False  # Ensures packets are correctly formatted
rfm9x.ack_retries = 5  # Number of retries to send an ack packet
rfm9x.ack_delay = .1  # If ACK's are being missed try .1 or .2
rfm9x.xmit_timeout = 2.0 # Recommended at least 2 seconds for HW reliability
with_header = True  # Set if you want header bytes printed for debug

def _format_datetime(datetime):
    return "{:02}/{:02}/{} {:02}:{:02}:{:02}".format(
        datetime.tm_mon,
        datetime.tm_mday,
        datetime.tm_year,
        datetime.tm_hour,
        datetime.tm_min,
        datetime.tm_sec,
    )
    
print("Waiting for packets... \n")

while True:
    # Time in seconds from when board was powered on
    now = time.monotonic()
    # Show fictional timestamp starting in year 2000.
    # Fake date are great for syn/ack timestamps on boards with no RTC or Wifi
    # Just don't print timestamps to end user in these scenarios.
    # It would obviously confuse them if they think the timestamps are wrong, they're not.
    
    # If your board DOES have wifi time server or RTC (not included) 
    # plug your fetched unix time into unix_time and uncomment below
    # unix_time = 1660764970 # Wed Aug 17 2022 19:36:10 GMT+0000
    # tz_offset_seconds = -14400  # NY Timezone
    # get_timestamp = int(unix_time + tz_offset_seconds)
    # and swap out CURRENT_UNIX_TIME with this one
    # current_unix_time = time.localtime(get_timestamp)
    
    current_unix_time = time.localtime()
    current_struct_time = time.struct_time(current_unix_time)
    current_date = "{}".format(_format_datetime(current_struct_time))
    
    # Changing timeout and sleep duration definitely has an effect on dropped packets.
    # Timeout 5 and sleep 0.5 works great with no dropped packets in my experience.
    packet = rfm9x.receive(keep_listening=False, timeout=rfm9x.xmit_timeout, with_header=with_header)
    
    # If no packet was received during timeout then packet=None is returned.
    if packet is None:
        # Packet has not been received
        LED.value = False
        print("Packet: None - Sleep for " + str(rfm9x.xmit_timeout) + " seconds...")
    else:
        # Now we're inside a received packet!
        LED.value = True
        try:
            # Header bytes are raw bytes. Good for debugging.
            if with_header:
                packet_text = str(packet)
            # No header strips header bytes and decodes to ascii.
            if not with_header:
                packet_text = str(packet, "ascii")
        except (UnicodeError) as e:
            print("Unicode Error", e)
            continue
        
        # received sig strenth and sig/noise
        rssi = rfm9x.last_rssi
        last_snr = rfm9x.last_snr
        # Debug printing of packet data:
        print("Received (raw bytes): {0}".format(packet))
        print("Timestamp:", now)
        print("Localtime: ", current_date)
        print("Signal Noise: {0} dB".format(last_snr))
        print("Signal Strength: {0} dB".format(rssi))
        print("(Received): {0}".format(packet_text))

    # Original New Packet Transmit (252 byte maximum)
    # Each send waits for previous send to finish
    message = str("Orignating Transmit Test 🙂")
    rfm9x.send(bytes(str(now) + " " +  message + "\r\n", "utf-8"))
    print("(Sent)" + " " + message + "\n")
    time.sleep(0.5)

[jerryneedell]

I'm not sure what you mean by "save" "...disabling itself every other save. "
I'll see if I can reproduce this, but it is not completely clear to me what I am looking for.

[jerryneedell]

Have you tried removing the time.sleep(2) at the end of your Receiver.py ?
Since you have the timeout set to rf9x_xmit_timeout (2.0) you will wait for up to 2 seconds for a packet when you call rem9x.receive()

[jerryneedell]

Also. If I am reading this correctly, your Transmitter will look for a received packet for up to 2 seconds, then send a packet then resume listening for 2 seconds after a .5 second sleep. If the Receiver receives the packets and sends its message, it will likely arrive while the Transmitter in in the time.sleep(.5) and will be missed.

Perhaps I am not understating your intended flow.

[DJDevon3]

@jerryneedell save, as in save the code which uploads it to the board.
Hit save in Mu and only the receive() function stops working.
Hit save again and it works fine.
Hit save in Mu and only the receive() function stops working.
Hit save again and it works fine.
Hit save in Mu and only the receive() function stops working.
Hit save again and it works fine.
etc…. It repeats this behavior infinitely. No errors just refuses to ever receive.

This happens regardless of what I set the receiver timeout or sleep values to. It’s like the receive() function itself is toggling between an on/off state.

By having different sleep time it should eventually roll over to a time when there is an incoming message to process but that never happens. My assumption is the receiver is actually being disabled somehow every other reload.

[jerryneedell]

hmmm --- Thanks -- I'll see if I can reproduce and then try to understand what is going on.

[jerryneedell]

I don't use MU for my development, but I think I have set things up in a similar way and so far, I cannot reproduce this issue.
I have an esp32s2 with an rfm9x featherwing running your Receiver program (using code.py to start it)

I have the Transmitter.py running on a Raspberry Pi with rfm9x bonnet, but that should not matter.

I start the Transmitter on the Pi then start up the Receiver on the esp32s2
I am monitoring its output via a "screen" session on my Linux computer
With the Receiver.py running, I repeatedly copied the Receiver.py code from my Linux computer to the esp32s2 forcing an auto-load. It works every time for me.

I also tried repeatedly copying code.py to the board and it also works ok for me.

Does this sound like a fair test of the issue you are seeing?

FYI - I am using the latest 8.0 beta build

Adafruit CircuitPython 8.0.0-beta.0-64-ge045415f5 on 2022-09-22; Adafruit Feather ESP32S2 with ESP32S2

example output showing am auto-reload

Received (raw bytes): bytearray(b'941.09349472 Orignating Transmit Test \xf0\x9f\x99\x82\r\n')
Timestamp: 1201.17
Signal Noise: 7.0 dB
Signal Strength: -67 dB
Received (ASCII): bytearray(b'941.09349472 Orignating Transmit Test \xf0\x9f\x99\x82\r\n')

Code stopped by auto-reload. Reloading soon.
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Waiting for packets...
Received (raw bytes): bytearray(b'948.911757885 Orignating Transmit Test \xf0\x9f\x99\x82\r\n')
Timestamp: 1207.9
Signal Noise: 6.5 dB
Signal Strength: -67 dB
Received (ASCII): bytearray(b'948.911757885 Orignating Transmit Test \xf0\x9f\x99\x82\r\n')
(Sent) M4 Express Msg Test � 

[jerryneedell]

I also tried using Mu -- I have it running Receiver.py and I kept making small changes to the file and "saving" it with Mu.
It roboots as expected.

[jerryneedell]

FYI -- I have tried this again - using Mu -- I have tried it with both Receiver.py as the open file and code.py as th open file. I have the Transmitter running on another system and on I am able to repeatedly hit "Save" in Mu and the esp32s2 does an auto-reload and restarts every time.
I am not sure what else I can do to try to help.

[DJDevon3]

I"m starting to think there's something wrong with my computer. I continually exhibit issues that others do not. Had an issue with the rp2040 reloading everything twice which jepler and neradoc didn't have. Maybe it's some kind of usb duplicator, keylogger, etc... ahhh i did install one of those PC DIN mounted USB hubs with like 10 ports. The more issues I report the more it's looking like a me problem.

[jerryneedell]

Are you OK with closing this issue for now. You can always reopen it if you think it is a library bug.

[DJDevon3]

Think I fixed my issue (USB enumeration controller corruption) but I still need to go through the paces of re-testing to see if the issue is gone. will be working on that this weekend.

[DJDevon3]

Nope issue persists and have found some evidence it's likely WiFi and multiple esp32's in proximity issue. Interference! espressif/arduino-esp32#2501 There might be a work around by specifically disabling and reinitializing the WiFi module. By allowing it to soft reset if there are duplicate boards around the router won't handshake until it's been cleared. Could be a router cache issue refusing to death when it should or the esp32 failing to send deauth.

In any event I believe this issue extends beyond circuit Python. This is an esp32 module & router communication issue. Not even rfm related except via rabbit hole set of circumstances.

[jerryneedell]

Ah - I have not tried it with multiple esp32s. I'll see if I can set that up sometime soon.

[jerryneedell]

I am now running your code on 2 esp32s2s with rfm9x feathering's and I still cannot reproduce the issue.
I did find that the Transmitter was sometimes missing the packet from the Receiver but that was due to the time.sleep(0.5) at the end of the while loop. I removed that and the problem was resolved.
I have never seen the Receiver stop receiving packets from the Transmitter.

also -- both esp32s2s are set up with .env files to connect to my WiFi.

[DJDevon3]

It's got to be my PC then. Every issue I'm running into lately no one else can replicate, on multiple different boards and it's always some kind of reload/reset/usb issue. Almost like serial itself is buggy on my pc. Is that even a thing?

[DJDevon3]

@jerryneedell Can you please try it with web workflow off? There's some new evidence to suggest this bug might only exist for users who are not using web workflow. I do not use web workflow.

[dhalbert]

On Espressif chips, most pins are reset to high with a weak pullup when CircuitPython stops (or soft-restarts). I have not read the whole thread carefully, but could this cause your problem?

[DJDevon3]

Sat down and pulled up both the S2 and LORA PCB design files. I'm using identical wiring as shown in the guide https://learn.adafruit.com/radio-featherwing/wiring IRQ goes to Pin B, Pin B goes to the 6th GPIO from the top right on the featherwing. On my ESP32-S2 that equates to GPIO 10. Guess which pin I'm using for CS and shouldn't be sharing a pin with IRQ.

CS = digitalio.DigitalInOut(board.D10)
RESET = digitalio.DigitalInOut(board.D6)

CS is causing the LORA IRQ to toggle every save. Honestly I'm surprised it even worked at all in that configuration.

Not a bug. User Error

[jerryneedell]

Glad you found it. I use Pin D for IRQ -- following the feather M0 guidelines from long ago...
A = RESET
B = CS
D = IRQ

Nice to have the world in sync again ;-)