esp32c3 GDB can't connect openocd successful with USB build-in jtag if do hard reset after flash (OCD-402)

[chuanjinpang]

JTAG 适配器类型。

esp32c3 build-in USB JTAG.
用于编译和加载正在调试的应用程序的 ESP-IDF 版本号。
IDF 4.3
用于调试的操作系统的详细信息。
windows8.1 msys2 mingw64 & mingw32 或者 ubuntu
操作系统是在本地计算机运行还是在虚拟机上运行？
本地计算机

创建一个能够演示问题的简单示例工程，描述复现该问题的步骤。且这个调试示例不能受到 Wi-Fi 协议栈引入的非确定性行为的影响，因而再次遇到同样问题时，更容易复现。
就是blink示例工程。
初步现象总结：
1.只要esp32c3刚烧写程序后处于Staying in flasher stub,GDB就能成功连接上，并能可以bt和dump 内存成功，si单步也能成功。可是这个时候调试不了应用程序，因为在flasher stub状态，没有跑应用。
2。在1的情况下，esp32复位后，应用程序跑起来了，但是GDB就连不上openocd，报 Failed to probe flash, size 0 KB。auto_probe failed。猜测是复位造成某个地方状态不对，造成openocd调用 esp_flash_get_size（）执行失败。

复制步骤与相关日志：
esp32复位后
然后gdb连接不成功，输出以下错误：
$ ./riscv32-esp-elf-gdb.exe /e/blink.elf
GNU gdb (crosstool-NG 1.24.0.123_64eb9ff) 8.3.1.20191211-git
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-host_w64-mingw32 --target=riscv32-esp-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from E:/blink.elf...
(gdb) target remote :3333
Remote debugging using :3333
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Remote replied unexpectedly to 'vMustReplyEmpty': PacketSize=4000;qXfer:memory-map:read+;qXfer:features:read+;qXfer:threads:read+;QStartNoAckMode+;vContSupported+
openocd输出
Info : accepting 'gdb' connection on tcp/3333
Warn : No symbols for FreeRTOS!
Error: Failed to write stub section!
Error: Failed to run flasher stub (-4)!
Warn : Failed to get flash mappings (-4)!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Failed to wait algorithm (-4)!
Error: Algorithm run failed (-4)!
Error: Failed to run flasher stub (-4)!
Warn : not enough working area available(requested 7640)
Error: no working area available, can't alloc space for stub code!
Error: Failed to run flasher stub (-308)!
Error: Failed to probe flash, size 0 KB
Error: auto_probe failed
Error: Connect failed. Consider setting up a gdb-attach event for the target to prepare target for GDB connect, or use 'gdb_memory_map disable'.
Error: attempted 'gdb' connection rejected
然后猜测是Failed to run flasher stub 造成的问题。
然后用telnet localhost 4444发现其实openocd可以工作，可以reg pc 成功dump pc的值，mdb能正常dump 内存内容。
当然bp断点工作有一些异常。所以USB通路上应该没有什么问题。

然后偶然发现，刚烧写程序后，GDB是能连上openocd. 也能基本的工作一下。可以bt和dump 内存成功
$ ./riscv32-esp-elf-gdb.exe /e/blink.elf
GNU gdb (crosstool-NG 1.24.0.123_64eb9ff) 8.3.1.20191211-git
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-host_w64-mingw32 --target=riscv32-esp-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from E:/blink.elf...
(gdb) target remote :3333
Remote debugging using :3333
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Remote replied unexpectedly to 'vMustReplyEmpty': PacketSize=4000;qXfer:memory-map:read+;qXfer:features:read+;qXfer:threads:read+;QStartNoAckMode+;vContSupported+
Undefined command: "". Try "help".
(gdb) target remote :3333

Remote debugging using :3333
0x40380786 in get_all_caps (heap=0x0)
at /root/ws/esp8266/esp-idf/components/heap/heap_private.h:56
56 /root/ws/esp8266/esp-idf/components/heap/heap_private.h: No such file or directory.
(gdb) (gdb) bt
#0 0x40380786 in get_all_caps (heap=0x0)
at /root/ws/esp8266/esp-idf/components/heap/heap_private.h:56
#1 heap_caps_realloc (ptr=0x0, size=1070137420, caps=1070153884)
at /root/ws/esp8266/esp-idf/components/heap/heap_caps.c:350
#2 0x4941484f in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
x /32x 0x40380786 可以bt和dump 内存成功
0x40380786 <heap_caps_realloc+72>: 0xa483dff5 0x478508ca 0x080aa623 0x02f10423
0x40380796 <heap_caps_realloc+88>: 0x0014c783 0x15234709 0x04a302e1 0xd60202f1
0x403807a6 <heap_caps_realloc+104>: 0x86634729 0x474d06e7 0x08e78063 0x45a13e2d
0x403807b6 <heap_caps_realloc+120>: 0x36391028 0x0034c703 0x0024c783 0x07226691
0x403807c6 <heap_caps_realloc+136>: 0x87938f5d 0xf5630106 0x051306e7 0x346d0db0
0x403807d6 <heap_caps_realloc+152>: 0x0dc00513 0x05133455 0x3c790ee0 0xbf793629
0x40380786 <heap_caps_realloc+72>: 0xa483dff5 0x478508ca 0x080aa623 0x02f10423
0x403807e6 <heap_caps_realloc+168>: 0x0007a023 0xb5d10791 0x403805b7 0x85934601
0x403807f6 <heap_caps_realloc+184>: 0x45150365 0xffc80097 0xe0a080e7 0x60000737
0x40380796 <heap_caps_realloc+88>: 0x0014c783 0x15234709 0x04a302e1 0xd60202f1
0x403807a6 <heap_caps_realloc+104>: 0x86634729 0x474d06e7 0x08e78063 0x45a13e2d
0x403807b6 <heap_caps_realloc+120>: 0x36391028 0x0034c703 0x0024c783 0x07226691
0x403807c6 <heap_caps_realloc+136>: 0x87938f5d 0xf5630106 0x051306e7 0x346d0db0
0x403807d6 <heap_caps_realloc+152>: 0x0dc00513 0x05133455 0x3c790ee0 0xbf793629
0x403807e6 <heap_caps_realloc+168>: 0x0007a023 0xb5d10791 0x403805b7 0x85934601
0x403807f6 <heap_caps_realloc+184>: 0x45150365 0xffc80097 0xe0a080e7 0x60000737
但是一旦复位后，blink程序跑起来，灯闪。GDB就连不上openocd. 报之前的问题。
并且openocd输出
Info : accepting 'gdb' connection on tcp/3333
Warn : No symbols for FreeRTOS!
Error: Failed to write stub section!
Error: Failed to run flasher stub (-4)!
Warn : Failed to get flash mappings (-4)!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Hart is not halted!
Error: Failed to wait algorithm (-4)!
Error: Algorithm run failed (-4)!
Error: Failed to run flasher stub (-4)!
Warn : not enough working area available(requested 7640)
Error: no working area available, can't alloc space for stub code!
所以从现象上看应该是某个地方状态出了点小问题，进一步发现只要芯片Staying in flasher stub，GDB就能100%连上
Staying in flasher stub时
root@localhost:~/ws# esptool.py --chip esp32c3 -p /dev/ttyUSB0 -b 460800 --before=default_reset --after=no_reset_stub write_flash --flash_mode dio --flash_freq 40m --flash_size 4MB 0x8000 ./partition-table.bin
esptool.py v3.1
Serial port /dev/ttyUSB0
Connecting....
Chip is unknown ESP32-C3 (revision 3)
Features: Wi-Fi
Crystal is 40MHz
MAC: 7c:df:a1:c0:11:48
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00008000 to 0x00008fff...
Compressed 3072 bytes to 103...
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.1 seconds (effective 366.9 kbit/s)...
Hash of data verified.

Leaving...
Staying in flasher stub.

[gerekon]

@chuanjinpang Could you describe your problem in English and attach verbose OpenOCD log generated with -d -l oocd.log options?

[chuanjinpang]

sorry, thanks for your reply. the issue can close now, it seem flasher stub have some related issue.
I have solved the issue by workaround.
it seem the openocd in esp_flash_probe() have some issue when run flasher_stub code.
after programe esp32c3 without hard reset , the GDB can connect the openocd successful and get flash info ok. however if do a hard reset, GDB can't connect the openocd due to can't get flash info successful due to execute the flasher_stub code failure.

the NG log :

Error: Failed to write stub section!
Error: Failed to run flasher stub (-4)!
Warn : Failed to get flash mappings (-4)!

sorry, I don't know why the flasher stub run failure, the stub code it's same by dump good case & NG case .
the stack seem OVF if enable ALGO_STUB_DEBUG at NG case.
but it's not my problem. I don't case the flasher stub issue.
just disable the flaher stub related code and fill some data to cheat openocd. the GDB work well for debug.
it can set breakpoint, and halt at bp, also can examine var successful. and continue to next bp. it's enough for me to debug.

workaround code:

int esp_flash_probe(struct flash_bank *bank){
...
#if 0
	int ret = esp_flash_get_mappings(bank,
		esp_info,
		&flash_map,
		esp_info->appimage_flash_base);
#else
        flash_map.maps_num=2;
        flash_map.maps[0].phy_addr=0x10020;
        flash_map.maps[0].load_addr=0x3c020020;
        flash_map.maps[0].size=23*1024;
        flash_map.maps[1].phy_addr=0x20020;
        flash_map.maps[1].load_addr=0x42000020;
        flash_map.maps[1].size=86*1024;
		for (uint32_t i = 0; i < flash_map.maps_num; i++){
			LOG_INFO("--Flash mapping %d: 0x%x -> 0x%x, %d KB",
				i,
				flash_map.maps[i].phy_addr,
				flash_map.maps[i].load_addr,
				flash_map.maps[i].size/1024);
	}
    int ret=ERROR_OK;
#endif
 ...
}

[chuanjinpang]

@gerekon it's strange issue, it esay to duplicate this issue (100% duplicate), however the ESP R&Der doesn't find it.
maybe just only my problem.

[gerekon]

it seem the openocd in esp_flash_probe() have some issue when run flasher_stub code.
after programe esp32c3 without hard reset , the GDB can connect the openocd successful and get flash info ok. however if do a hard reset, GDB can't connect the openocd due to can't get flash info successful due to execute the flasher_stub code failure.

@chuanjinpang What do you mean by hard reset ? Resetting chip by pressing CTRL+T+R in IDF terminal?

it esay to duplicate this issue (100% duplicate),

It will be very helpful if you could give the exact steps to reproduce the problem. Smth like.

1. Run OpenOCD with command line blablabla
2. Do a hard reset for the chip via IDF Terminal (CTRL+T+R)
3. Start GDB with command line blablabla

[chuanjinpang]

@chuanjinpang What do you mean by hard reset ? Resetting chip by pressing CTRL+T+R in IDF terminal?

no, it's press the board CHIP_EN key actually for a hard reset, then the board run the user applicate program after donwload bin files to flash.
my esp32c3 board is name"esp-c3-13-kit" which very like NodeMCU(esp8266) board.

It will be very helpful if you could give the exact steps to reproduce the problem. Smth like.

1. use flash_download_tool_v3.8.8(on windows8.1) to download blink(sample project in esp-idf) code to esp32c3 board.
2. Do a hard reset for the chip via press CHIP_EN key. it will pull the esp32c3 CHIP_EN pin to low and to high if release key. then we can see the LED will blink. so the user program is running well.
3. Run OpenOCD with command line on MSYS2 mingw32 to connect chip(also on winodws8.1):
   ./openocd.exe -f board/esp32c3-builtin.cfg -d2
4. Start GDB with command line :
   ./riscv32-esp-elf-gdb.exe /e/blink.elf
5. then the openocd will report failure log like below:

Error: Failed to wait algorithm (-4)!
Error: Algorithm run failed (-4)!
Error: Failed to run flasher stub (-4)!
Info : stub: base 0x0, start 0x403816d6, 2 sections
Info : addr 0x00000000, sz 7640, flags 1
Info : addr 0x00000000, sz 637, flags 0
Error: Failed to write stub section!
Error: Failed to run flasher stub (-4)!
Error: Failed to probe flash, size 0 KB
Error: auto_probe failed
Error: Connect failed. Consider setting up a gdb-attach event for the target to prepare target for GDB connect, or use 'gdb_memory_map disable'.
Error: attempted 'gdb' connection rejected

6. and the GDB side also report error.

that's all steps. if you still can't duplicate this issue, please let me know freely.
BTW, the ubuntu18.04 also have same issue with openocd.
openocd-esp32 tag: v0.10.0-esp32-20210721

[chuanjinpang]

@gerekon any update?

[xBiohazardx]

I think I've got the same problem.

[gerekon]

@chuanjinpang

@gerekon any update?

Can you try with disabled memory protection? Component config → ESP System Settings → Memory protection in menuconfig? You need to disable it or reset board after OpenOCD is run to prevent bootloader enabling it.

[gerekon]

The best way to start debugging esp32c3 is to run GDB with the following startup commands riscv32-esp-elf-gdb.exe -x gdbinit /e/blink.elf

gdbinit:

target remote :3333
mon reset halt
flushregs
tb app_main
c

[DmytroKorniienko]

@gerekon

Can you try with disabled memory protection? Component config → ESP System Settings → Memory protection in menuconfig? You need to disable it or reset board after OpenOCD is run to prevent bootloader enabling it.

Thanks grateful! I spent a lot of hours with similar issue and finally have a success. Now debugging working for me:

JTAG tap: esp32c3.cpu tap/device found: 0x00005c25 (mfg: 0x612 (Espressif Systems), part: 0x0005, ver: 0x0)
Temporary breakpoint 1 at 0x42005102: file ../main/blink_example_main.c, line 74.
Note: automatically using hardware breakpoints for read-only addresses.
[New Thread 1070175384]
[New Thread 1342184384]
[Switching to Thread 1342184824]

Thread 1 hit Temporary breakpoint 1, app_main () at ../main/blink_example_main.c:74
74          configure_led();

Is this something HW related? In my case boards - NodeMCU-Series-C3-32S-Kit-3297 & NodeMCU-Series ESP32-C3F-Kit-3252 with AiThinker modules.

Reset by pressing button on board - hang down debugging and breakpoint never reached

[gerekon]

@DmytroKorniienko

Is this something HW related?

When memory protection is enabled (by default) it prevents flasher stub from working properly, because it is loaded into target's memory and works from it. OpenOCD currently does nothing with memory protection. IDF bootloader does not enable it when the debugger is connected, so that's why you need to reset board after OpenOCD is connected.
We keep this ticket till it is fixed in OpenOCD.

[DmytroKorniienko]

@gerekon
ok, thanks for explanation

In my case I can't debug PIO projects... I tryed to do this with some dirty trick like put C3 into butloader mode outside of PIO this helpfull for strarting flashing debug stab but I can't back to normal execution after flashing. CTRL+T, CRTL+R or pressing reset button have identical result - OpenOCD stop working, debug session hanging.

Is there any way to fix this?

After reset:

...
Error: dmi_scan failed jtag scan
Error: failed read at 0x11, status=2
Error: Failed to read DMSTATUS (-4)!
esp_usb_jtag: usb sent only 0 out of 2544 bytes.
2
Error: esp_usb_jtag: usb sent only 0 out of 2560 bytes.
Error: missing data from bitq interface
missing data from bitq interface
dmi_scan failed jtag scan
failed read at 0x11, status=2
Failed to read DMSTATUS (-4)!
Error: dmi_scan failed jtag scan
Error: failed read at 0x11, status=2
2
esp_usb_jtag: usb sent only 0 out of 2560 bytes.
missing data from bitq interface
dmi_scan failed jtag scan
failed read at 0x11, status=2
Error: esp_usb_jtag: us
b sent only 0 out of 2576 bytes.
Error: missing data from bitq inte
rface
Error: dmi_scan failed jtag scan
Error: failed read at 0x11, status=2
Error: Failed to read DMSTATUS (-4)!
Error: esp_usb_jtag: usb sent only 0 out of 2592 bytes.
esp_usb_jtag: usb sent only 0 out of 2576 bytes.
missing data from bitq interface
Error: extra data from bitq interface
Error: dmi_scan failed jtag scan
Error: failed read at 0x11, status=2
dmi_scan failed jtag scan
failed read at 0x11, status=2
Failed to read DMSTATUS (-4)!
esp_usb_jtag: usb sent only 0 out of 2592 bytes.
extra data from bitq interface
dmi_scan failed jtag scan
failed read at 0x11, status=2
Error: esp_usb_jtag: usb sent only 0 out of 2608 bytes.
esp_usb_jtag: usb sent only 0 out of 2608 bytes.
Exception condition detected on fd 9
Remote communication error.  Target disconnected.: Success.

[gerekon]

Is there any way to fix this?

Yes, there is a problem with esp_usb_jtag driver when chip is reset externally. Should be fixed in the next release.
For now you can use SW reset mon reset run from GDB console. Or reset run from OpenOCD telnet connection at port 4444

[DmytroKorniienko]

@gerekon
Seems SW reset just ignored... Please look at attached screenshot: