Nano 33 BLE: Threads not working on nano 33 ble sense v2

[mjs513]

Describe the bug
Testing thread example on nano 33 ble sense v2 fails. Running in debug mode shows no output on serial or serial1. then to reload sketch have to double click reset to reload sketch.

Target board + cli verbose compilation output
Full verbose compilation output, ideally with arduino-cli invocation or from IDE 2.3.3+
Issues without the full verbose output will be discarded as invalid.

loading library from d:\Users\Merli\Documents\Arduino\libraries\Arduino-EasyTransfer-master: invalid library: no header files foundloading library from d:\Users\Merli\Documents\Arduino\libraries\ArduinoCore-API-master: invalid library: no header files foundloading library from d:\Users\Merli\Documents\Arduino\libraries\FastAhrs: invalid library: no header files foundloading library from d:\Users\Merli\Documents\Arduino\libraries\Lidar-Lite-v3-3d-Scanner: invalid library: no header files foundloading library from d:\Users\Merli\Documents\Arduino\libraries\M5Stack: invalid library: no header files foundloading library from d:\Users\Merli\Documents\Arduino\libraries\esp32-camera-master: invalid library: no header files found
FQBN: arduino:zephyr:nano33ble:debug=true
Using board 'nano33ble' from platform in folder: C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0
Using core 'arduino' from platform in folder: C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0

Detecting libraries used...
C:\Users\Merli\AppData\Local\Arduino15\packages\zephyr\tools\arm-zephyr-eabi\0.16.8/bin/arm-zephyr-eabi-g++ -g -Os -std=c++17 -c -w -DLL_EXTENSION_BUILD -DCONFIG_ARDUINO_API_SERIAL_BUFFER_SIZE=256 -DNRF52840_XXAA -DKERNEL -DK_HEAP_MEM_POOL_SIZE=32768 -DPICOLIBC_LONG_LONG_PRINTF_SCANF -D__LINUX_ERRNO_EXTENSIONS__ -D__PROGRAM_START -D__ZEPHYR__=1 -imacrosC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense/llext-edk/include/zephyr/include/generated/zephyr/autoconf.h -imacrosC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense/llext-edk/include/zephyr/include/zephyr/toolchain/zephyr_stdint.h -fno-exceptions -fno-rtti -fno-threadsafe-statics -fno-unwind-tables -fno-use-cxa-atexit -fdata-sections -ffunction-sections -fno-unwind-tables -fno-strict-aliasing -fno-printf-return-value -fno-common -mcpu=cortex-m4 -mthumb -mabi=aapcs -mfp16-format=ieee -mtp=soft -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wdouble-promotion -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -fno-asynchronous-unwind-tables -ftls-model=local-exec -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop --specs=picolibc.specs -D_POSIX_THREADS -mlong-calls -nodefaultlibs -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -w -x c++ -E -CC -DARDUINO=10607 -DARDUINO_ARDUINO_NANO33BLE -DARDUINO_ARCH_ZEPHYR -DARDUINO_ARCH_ZEPHYR -DARDUINO_LIBRARY_DISCOVERY_PHASE=1 -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\cores\arduino -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\cores\arduino/api/deprecated -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\cores\arduino/api/deprecated-avr-comp -iprefixC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense @C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense/includes.txt C:\Users\Merli\AppData\Local\arduino\sketches\C14FD5273126A35C1639FF955FA3C7A5\sketch\k_thread_create.ino.cpp -o nul
Generating function prototypes...
C:\Users\Merli\AppData\Local\Arduino15\packages\zephyr\tools\arm-zephyr-eabi\0.16.8/bin/arm-zephyr-eabi-g++ -g -Os -std=c++17 -c -w -DLL_EXTENSION_BUILD -DCONFIG_ARDUINO_API_SERIAL_BUFFER_SIZE=256 -DNRF52840_XXAA -DKERNEL -DK_HEAP_MEM_POOL_SIZE=32768 -DPICOLIBC_LONG_LONG_PRINTF_SCANF -D__LINUX_ERRNO_EXTENSIONS__ -D__PROGRAM_START -D__ZEPHYR__=1 -imacrosC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense/llext-edk/include/zephyr/include/generated/zephyr/autoconf.h -imacrosC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense/llext-edk/include/zephyr/include/zephyr/toolchain/zephyr_stdint.h -fno-exceptions -fno-rtti -fno-threadsafe-statics -fno-unwind-tables -fno-use-cxa-atexit -fdata-sections -ffunction-sections -fno-unwind-tables -fno-strict-aliasing -fno-printf-return-value -fno-common -mcpu=cortex-m4 -mthumb -mabi=aapcs -mfp16-format=ieee -mtp=soft -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wdouble-promotion -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -fno-asynchronous-unwind-tables -ftls-model=local-exec -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop --specs=picolibc.specs -D_POSIX_THREADS -mlong-calls -nodefaultlibs -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -w -x c++ -E -CC -DARDUINO=10607 -DARDUINO_ARDUINO_NANO33BLE -DARDUINO_ARCH_ZEPHYR -DARDUINO_ARCH_ZEPHYR -DARDUINO_LIBRARY_DISCOVERY_PHASE=1 -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\cores\arduino -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\cores\arduino/api/deprecated -IC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\cores\arduino/api/deprecated-avr-comp -iprefixC:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense @C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\variants\arduino_nano_33_ble_sense/includes.txt C:\Users\Merli\AppData\Local\arduino\sketches\C14FD5273126A35C1639FF955FA3C7A5\sketch\k_thread_create.ino.cpp -o C:\Users\Merli\AppData\Local\Temp\966410908\sketch_merged.cpp
C:\Users\Merli\AppData\Local\Arduino15\packages\builtin\tools\ctags\5.8-arduino11/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives C:\Users\Merli\AppData\Local\Temp\966410908\sketch_merged.cpp
Compiling sketch...
"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\zephyr\\tools\\arm-zephyr-eabi\\0.16.8/bin/arm-zephyr-eabi-g++" -g -Os -std=c++17 -c -DLL_EXTENSION_BUILD -DCONFIG_ARDUINO_API_SERIAL_BUFFER_SIZE=256 -DNRF52840_XXAA -DKERNEL -DK_HEAP_MEM_POOL_SIZE=32768 -DPICOLIBC_LONG_LONG_PRINTF_SCANF -D__LINUX_ERRNO_EXTENSIONS__ -D__PROGRAM_START -D__ZEPHYR__=1 "-imacrosC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\variants\\arduino_nano_33_ble_sense/llext-edk/include/zephyr/include/generated/zephyr/autoconf.h" "-imacrosC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\variants\\arduino_nano_33_ble_sense/llext-edk/include/zephyr/include/zephyr/toolchain/zephyr_stdint.h" -fno-exceptions -fno-rtti -fno-threadsafe-statics -fno-unwind-tables -fno-use-cxa-atexit -fdata-sections -ffunction-sections -fno-unwind-tables -fno-strict-aliasing -fno-printf-return-value -fno-common -mcpu=cortex-m4 -mthumb -mabi=aapcs -mfp16-format=ieee -mtp=soft -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wdouble-promotion -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -fno-asynchronous-unwind-tables -ftls-model=local-exec -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop --specs=picolibc.specs -D_POSIX_THREADS -mlong-calls -nodefaultlibs -MMD -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DARDUINO=10607 -DARDUINO_ARDUINO_NANO33BLE -DARDUINO_ARCH_ZEPHYR -DARDUINO_ARCH_ZEPHYR -DARDUINO_LIBRARY_DISCOVERY_PHASE=0 "-IC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\cores\\arduino" "-IC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\variants\\arduino_nano_33_ble_sense" "-IC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\cores\\arduino/api/deprecated" "-IC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\cores\\arduino/api/deprecated-avr-comp" "-iprefixC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\variants\\arduino_nano_33_ble_sense" "@C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0\\variants\\arduino_nano_33_ble_sense/includes.txt" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5\\sketch\\k_thread_create.ino.cpp" -o "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5\\sketch\\k_thread_create.ino.cpp.o"
Compiling libraries...
Compiling core...
Using previously compiled file: C:\Users\Merli\AppData\Local\arduino\sketches\C14FD5273126A35C1639FF955FA3C7A5\core\variant.cpp.o
Using precompiled core: C:\Users\Merli\AppData\Local\arduino\cores\arduino_zephyr_nano33ble_debug_true_7f25305957607ee930a6b20eac5960bd\core.a
Linking everything together...
"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\zephyr\\tools\\arm-zephyr-eabi\\0.16.8/bin/arm-zephyr-eabi-g++" "-LC:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5" -r -Wl,--gc-sections -e main -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -std=c++17 -fno-exceptions -fno-rtti -fno-threadsafe-statics -fno-unwind-tables -fno-use-cxa-atexit "-TC:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\zephyr\\0.1.0/variants/llext/linker_script.ld" "-Wl,-Map,C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.map" --specs=nosys.specs -o "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.elf" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5\\sketch\\k_thread_create.ino.cpp.o" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5\\core\\variant.cpp.o" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/..\\..\\cores\\arduino_zephyr_nano33ble_debug_true_7f25305957607ee930a6b20eac5960bd\\core.a" -lstdc++ -lsupc++ -lnosys -nostdlib
"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\zephyr\\tools\\arm-zephyr-eabi\\0.16.8/bin/arm-zephyr-eabi-objcopy" -O binary "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.elf" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.bin"
"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\zephyr\\tools\\arm-zephyr-eabi\\0.16.8/bin/arm-zephyr-eabi-objcopy" -O ihex -R .eeprom "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.elf" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.hex"
"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\zephyr\\tools\\arm-zephyr-eabi\\0.16.8/bin/arm-zephyr-eabi-strip" --strip-debug "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.elf" "-oC:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.llext"
"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\zephyr-post-build-tool\\0.0.2/post_build" "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.llext" debug
File copied and saved as C:\Users\Merli\AppData\Local\arduino\sketches\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.llext.dfu
Bootloader file specified but missing: C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\zephyr\0.1.0\bootloaders\zephyr-arduino_nano_33_ble_sense.bin

"C:\\Users\\Merli\\AppData\\Local\\Arduino15\\packages\\zephyr\\tools\\arm-zephyr-eabi\\0.16.8/bin/arm-zephyr-eabi-size" -A "C:\\Users\\Merli\\AppData\\Local\\arduino\\sketches\\C14FD5273126A35C1639FF955FA3C7A5/k_thread_create.ino.elf"
Sketch uses 1792 bytes (0%) of program storage space. Maximum is 786432 bytes.
Global variables use 2576 bytes (0%) of dynamic memory, leaving 521048 bytes for local variables. Maximum is 523624 bytes.
Device       : nRF52840-QIAA
Version      : Arduino Bootloader (SAM-BA extended) 2.0 [Arduino:IKXYZ]
Address      : 0x0
Pages        : 256
Page Size    : 4096 bytes
Total Size   : 1024KB
Planes       : 1
Lock Regions : 0
Locked       : none
Security     : false
Erase flash

Done in 0.001 seconds
Write 7828 bytes to flash (2 pages)

[                              ] 0% (0/2 pages)
[===============               ] 50% (1/2 pages)
[==============================] 100% (2/2 pages)
Done in 0.324 seconds

Output of Serial Monitor
Serial OUTPUT:

�[m
�[1;32muart:~$ �[msketch

Serial1 Output:

same as  serial

Output of readelf
You can find the loaction of the elf file by compiling in Verbose mode and looking near the end of the compilation output (after Linking everything together..)
Paste (or attach) the output of arm-none-eabi-readelf -a $your_sketch_elf_file
elf_Output.txt

Optional: attach the elf file
k_thread_create.ino.zip

Optional: attach the sketch

/* main.c - Hello World demo */

/*
 * Copyright (c) 2012-2014 Wind River Systems, Inc.
 *
 * SPDX-License-Identifier: Apache-2.0
 */


/* size of stack area used by each thread */
#define STACKSIZE 1024

/* scheduling priority used by each thread */
#define PRIORITY 7

/* delay between greetings (in ms) */
#define SLEEPTIME 500


K_THREAD_STACK_DEFINE(threadA_stack_area, STACKSIZE);
static struct k_thread threadA_data;

/* threadA is a static thread that is spawned automatically */

void threadA(void *dummy1, void *dummy2, void *dummy3)
{
	ARG_UNUSED(dummy1);
	ARG_UNUSED(dummy2);
	ARG_UNUSED(dummy3);

	Serial.print("thread_a: thread started \n");

	while (1)
	{
		Serial.print("thread_a: thread loop \n");
		k_msleep(SLEEPTIME);
	}

}


void setup() {
	k_thread_create(&threadA_data, threadA_stack_area,
			K_THREAD_STACK_SIZEOF(threadA_stack_area),
			threadA, NULL, NULL, NULL,
			PRIORITY, 0, K_FOREVER);
	k_thread_name_set(&threadA_data, "thread_a");

	k_thread_start(&threadA_data);
}

void loop() {
  // put your main code here, to run repeatedly:

}

Additional context
Add any other context about the problem here.

[mjs513]

Playing more with threads on the nano found that the k_create sketch had to add in k_msleep(SLEEPTIME); it will run in "Standard" mode with the default config file. However if I try to run in "Debug" mode if fails to run after I type "sketch"???

I also tried to run the k_define example sketch:

/*
 * Copyright (c) 2017 Linaro Limited
 *
 * SPDX-License-Identifier: Apache-2.0
 */
#include <zephyr/kernel.h>

#include "elapsedMillis.h"
elapsedMillis timer;

/* STEP 2 - Define stack size and scheduling priority used by each thread */
#define STACKSIZE 1024

#define THREAD0_PRIORITY 7
#define THREAD1_PRIORITY 7

void thread0(void)
{
  Serial.print("Hello Thread 0 Started\n");
  while (1) {
    /* STEP 3 - Call printk() to display a simple string "Hello, I am thread0" */
    Serial.print("Hello, I am thread0\n");
    /* STEP 6 - Make the thread yield */
    // k_yield();
    /* STEP 10 - Put the thread to sleep */
    k_msleep(5);
    /* Remember to comment out the line from STEP 6 */
  }
}

void thread1(void)
{
  Serial.println("Thread 1 Started...");
  while (1) {
    /* STEP 3 - Call printk() to display a simple string "Hello, I am thread1" */
    Serial.print("Hello, I am thread1\n");
    /* STEP 8 - Make the thread yield */
    // k_yield();
    /* STEP 10 - Put the thread to sleep */
    k_msleep(10);
    /* Remember to comment out the line from STEP 8 */
  }
}

/* STEP 4 - Define and initialize the two threads */
K_THREAD_DEFINE(thread0_id, STACKSIZE, thread0, NULL, NULL, NULL, THREAD0_PRIORITY, 0, 0);
K_THREAD_DEFINE(thread1_id, STACKSIZE, thread1, NULL, NULL, NULL, THREAD1_PRIORITY, 0, 0);

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  while (!Serial && millis() < 5000) {};
  Serial.println("Threading sketch #2 started");

  delay(100);

k_tid_t our_tid = k_sched_current_thread_query();
  int main_pri = k_thread_priority_get(our_tid);
  Serial.print("main TID: ");
  Serial.print((uint32_t)our_tid, HEX);
  Serial.print(" pri: ");
  Serial.println(main_pri);
  printk("main TID:%x pri:%d\n", (uint32_t)our_tid, main_pri);
  k_thread_priority_set(our_tid, THREAD0_PRIORITY+1);
  main_pri = k_thread_priority_get(our_tid);
  Serial.print("\tupdated pri: ");
  Serial.println(main_pri);
  printk("main TID:%x pri:%d\n", (uint32_t)our_tid, main_pri);

  Serial.println("End Setup");
}

void loop() {
  // put your main code here, to run repeatedly:
  if(timer > 5000){
    Serial.println("called from loop");
    timer = 0;
  }
  k_msleep(100);
}

in standard mode but after it loads it looses serial connection (ie. looses comport in the ide). In debug mode after sketch if fails to run and show anything in the serial or serial1 window.

This happens even if we increase the stack size:

##CONFIG_SHELL_STACK_SIZE=2048
##CONFIG_HEAP_MEM_POOL_SIZE=16384
##CONFIG_LLEXT_HEAP_SIZE=128
##CONFIG_MAIN_STACK_SIZE=32768

CONFIG_HEAP_MEM_POOL_SIZE=2048
CONFIG_SHELL_STACK_SIZE=32768
CONFIG_MAIN_STACK_SIZE=32768
CONFIG_LLEXT_HEAP_SIZE=128

##CONFIG_STACK_USAGE=y
##CONFIG_STACK_SENTINEL=n
##CONFIG_STACK_CANARIES=y

So question is will threads be supported on the Nano?

[mjs513]

A little more progress. By adding in config_fpu = y the k_define test sketch posted now runs. However, the mutex and semaphore test sketches still fail to run - loose com port. More investigation.

[mjs513]

Merged changes as of this morning
thread_create: works
thread_define: looses com port
thread_mutest: seems to start and then stops:

Thread 1 is accessing the shared resource
Thread 2 is accessing the shared resource
"THEN STOPS"

threads_semsphore: looses com port

@facchinm - is there a way to have printk send its output to Serial instead of Serial1?

[facchinm]

It used to work that way when compiling with "Debug" menu selected but stopped for some reason, me and @pillo79 are investigating

[pillo79]

K_THREAD_STACK_DEFINE(threadA_stack_area, STACKSIZE);

Note that this will not work - the macros that define Zephyr kernel objects are useless in a sketch/extension and result in undefined behavior. I will take care of posting a patch to error in these situations, as it is a major issue. The workaround is to use dynamic objects but it is obviously a limited approach.

[mjs513]

Note that this will not work - the macros that define Zephyr kernel objects are useless in a sketch/extension and result in undefined behavior. I will take care of posting a patch to error in these situations, as it is a major issue. The workaround is to use dynamic objects but it is obviously a limited approach.

Does this also apply to the giga since the sketch I have tested with seem to be working on the giga with no issue.

[pillo79]

Does this also apply to the Giga since the sketch I have tested with seem to be working on the Giga with no issue.

Yes, it is an LLEXT limitation - most kernel macros rely on special sections in the ELF file, and those are IIRC silently ignored at extension load time. Those variables end up "anywhere in memory", so while it is possible you are not corrupting anything (yet... 🙂), it's unfortunately out of pure luck.

EDIT - TBH the THREAD_STACK is kind of different among kernel objects. It may not require special sections but absolutely requires alignment, and that too is unsupported/ignored by LLEXT at the moment 🙄

[mjs513]

@pillo79
Learning quit a bit from this discussion. Not trying to be argue but to learn.

Understand what you are saying when you say:

Yes, it is an LLEXT limitation - most kernel macros rely on special sections in the ELF file, and those are IIRC silently ignored at extension load time. Those variables end up "anywhere in memory", so while it is possible you are not corrupting anything (yet... 🙂), it's unfortunately out of pure luck.

To get threads working we did modify the linker script:

    .text 0x00000000 : ALIGN(4) {
        *(.text)
        *(.stub)
        *(.text*)
        *(.text.*)
        *(.text._*)

        KEEP (*(.init))
        KEEP (*(.ctors))
        KEEP (*(.dtors))
        KEEP (*(.fini))

        __static_thread_data_list_start = .;
        KEEP(*(SORT_BY_NAME(.__static_thread_data.static.*)));
        __static_thread_data_list_end = .;
    }

Are you saying this is not sufficient? What else are we missing?

[pillo79]

It appears LLEXT improved recently, as in a quick test I was able to define some kernel stuff (e.g. a semaphore) in an extension without issues 🙂 Have not looked into these details in some time, so I need to dig deeper.

Zephyr maintains lists of kernel objects for its own reasons, and those are managed globally when linking the main Zephyr binary by merging together those weird per-file *.static.* iterable sections. There's lookup logic that expects to find all objects of the same kind there, and this definitely won't work if a sketch defines some of them in an extension. I guess it depends on the actual object type and the application; you might not experience any issue.

One surely problematic area is with CONFIG_USERSPACE enabled (e.g. with MMU/MPU enforcing memory access rules), because that requires precise alignment of structures in memory (say, a thread stack must start at an address that is multiple of 4k). This kind of constraint is stored in the ELF but (currently) totally ignored by LLEXT at load time.

[mjs513]

Yeah we ran into iterable_sections issue when we first started down this path. That's how we ended up modifing the linker_script - I say we but @KurtE came up with that solution. As for CONFIG_USERSPACE we didnt define that in the conf file. Found we didnt need to for what we were testing.

Out of curiousity I decided to modify one of my mutex sketches to access sdram:

K_MUTEX_DEFINE(my_mutex);

#include "SDRAM.h"
uint8_t *mySmallArray = nullptr;
uint8_t multiplier = 1;

void thread1(void)
{
    while (1) {
        k_mutex_lock(&my_mutex, K_FOREVER);
        Serial.print("Thread 1 is accessing the shared resource\n");
        mySmallArray = (uint8_t*)SDRAM.malloc(128);
        for (int i = 0; i<128; i++) {
            mySmallArray[i] =  i * multiplier;
        }
        k_sleep(K_MSEC(100));
        k_mutex_unlock(&my_mutex);
        k_sleep(K_MSEC(1000));
    }
}

void thread2(void)
{
    while (1) {
        k_mutex_lock(&my_mutex, K_FOREVER);
        Serial.print("Thread 2 is accessing the shared resource\n");
        for(uint8_t i = 0; i < 128; i++) {
          Serial.print(mySmallArray[i]); Serial.print(", ");
        }
        Serial.println();
        // free the memory when you don't need them anymore
        SDRAM.free(mySmallArray);
        if(multiplier == 1) {
          multiplier = 2;
        } else {
          multiplier = 1;
        }
        k_sleep(K_MSEC(100));
        k_mutex_unlock(&my_mutex);
        k_sleep(K_MSEC(1000));
    }
}

K_THREAD_DEFINE(thread1_id, 1024, thread1, NULL, NULL, NULL, 4, 0, 0);
K_THREAD_DEFINE(thread2_id, 1024, thread2, NULL, NULL, NULL, 4, 0, 0);


void setup() {
  Serial.begin(115200);
  while(!Serial && millis() < 5000);

  SDRAM.begin();
}

void loop() {
  // put your main code here, to run repeatedly:
  k_msleep(100);
}

Seems to be working on the giga - guess getting lucky:

Thread 1 is accessing the shared resource
Thread 2 is accessing the shared resource
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 
Thread 1 is accessing the shared resource
Thread 2 is accessing the shared resource
0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254, 
Thread 1 is accessing the shared resource
Thread 2 is accessing the shared resource

thread1 writes and thread2 reads. Any suggestions on seeing if I can break it.

[mjs513]

Seems this issue is still present on the Nano however now seeing strange compile errors when trying to test thread_define - note thread_create works.

c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(USB.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(USB.cpp.o)
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(Print.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(Print.cpp.o)
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(main.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(main.cpp.o)
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(threads.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(threads.cpp.o)
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(zephyrCommon.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(zephyrCommon.cpp.o)
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(zephyrSerial.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(zephyrSerial.cpp.o)
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: error: C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/k_thread_define.ino_debug.elf uses VFP register arguments, C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(Common.cpp.o) does not
c:/users/merli/appdata/local/arduino15/packages/zephyr/tools/arm-zephyr-eabi/0.16.8/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.exe: failed to merge target specific data of file C:\Users\Merli\AppData\Local\arduino\sketches\EC68BAFA0BFC371C84BB1C90A7C244D1/core\core.a(Common.cpp.o)
collect2.exe: error: ld returned 1 exit status

exit status 1

Compilation error: exit status 1

[facchinm]

@mjs513 uses VFP register arguments error usually means that the floating point (FPU) flags are unsynced (different between loader and sketch worlds).
@pillo79 can you take a look and try to reproduce?

[mjs513]

@facchinm - @pillo79

Just retested this with my thread test sketches using the latest changes:
thread_define
thread_nested_mutexes
thread_semaphore.

Threads define and mutexes are working. However Semaphore hangs the Nano Sense. Going into debug mode, no errors or prints appear on the Serial Monitor or on Serial1. In serial monitor I see

�[1;32muart:~$ �[msketch
[00:00:07.369,201] �[1;33m<wrn> usb_device: USB device support already enabled�[0m
A0300

[mjs513]

@facchinm - @pillo79 - @KurtE

Looks like with the last set of changes I made to heap and exports.c semaphore is now working. so going to close this for now assuming Llext heap fix for nano33 and Fixes for LLEXT_EXPORTS.C #109
will be incorportated