Performance drop on Wifi (30%) and BT (60%) with latest build?

[cyberman54]

I am using this Arduino core for wifi and bluetooth packet sniffing on ESP32 devies, using monitor mode of ESP32 wifi stack and using ESP32 BT/BLE stack. The project can be found here.

I assessed that with latest build of arduino-esp32 core the overall performance on both wifi and bt sniffing decreased dramatically.

Test setup was 2 identical hardware boards with identical application software, but one build using arduino-esp32 core version 2.10000.0 (version number in platformio) and other one build using latest code of this repo by 26th Nov. 18.

The board build with latest code sniffs ~60% less bluetooth and ~30% less wifi devices compared with that one build on older code (under same conditions in same time).

I have no idea how to further analyze this. Since both BT and Wifi are affected this could mean a general performance drop on core 0 (running BT and Wifi stack), or a problem with the 2,4 GHz radio driver?

My code for bt and wifi sniffing can be found here:

Wifi sniffing
BT sniffing

[kscheff]

Have you tried to swap the boards in order to verify if it is not an issue with the hardware?

[cyberman54]

I tried with several boards before, because i expected a hardware problem. It's clearly reproducable on different kind of hardware, the drop comes in when comparing code running on newest core.

[chegewara]

Im not saying it is, but it may be because change in SW coex settings as per my suggestion was changed to wifi performance. In other hand you report that both, bt and wifi performance drop.
I tested yesterday ble library performance with new arduino-esp32 and i can send notifications in very tight loop, with only 3ms(maybe even 2ms) delay without packet drop or congest event.

[cyberman54]

@chegawara my code does not send any data neither on BT nor on Wifi, it uses only the monitor mode for sniffing. I think 2-3ms is for sniffing a very long time, i guess the monitor mode packet capture driver on esp32 will run much faster. So we may see here a performance problem on low layer. Any ideas how to further analyze this?

[chegewara]

I would rather try to open issue in esp-idf repo, because like you said it can be in low level driver.

[cyberman54]

I did intentionally open this issue here, because as far a i can see with platformio both my build version are based on same version of ESP-IDF core. So it looks like it must be something came in with latest changes in arduino-esp32 core. Maybe something which causes side effekt on esp32 drivers.

[cyberman54]

@chegewara : could you please point me to the code affected by "change in SW coex settings as per my suggestion"? I would like try to check if this change could have impact on my use case.

[chegewara]

This is change in arduino-esp32 v1.0.1-rc1 im talking about:
https://github.com/espressif/arduino-esp32/blob/master/tools/sdk/sdkconfig#L346-L350

It changes some bt and wifi library, but i dont know how.

[cyberman54]

@chegewara Thank you very much for this pointer. Since the change came in exactly at the same time when i discovered the issue in my software, i think it is the hot scent to the magic what happened.
I will try with different coexist settings.
Didn't find any descriptions on how that coexist settings work. It seems the coexist functions are only forward declared in the source code and point to a precompiled library of the esp32 wifi stack:

https://github.com/espressif/esp32-wifi-lib/blob/master/libcoexist.a

[cyberman54]

@chegewara i tested different coexist settings by switching it during runtime with esp_coex_preference_set(coexist_config). But does not show any effect on my sniffing results. Still far lower sniffing results than with previous build. This is annoying :-(

Maybe @nkolban can help here?

[chegewara]

Kolban is busy and is not working with esp32 or ble now.

[copercini]

empirically saying: the best results for BLE scan here is when SW coexistence is fully disabled, using only HW switching (ie. it disables BLE for a while and use the radio for wifi, then disables wifi and uses BLE...), this doens't work for classical BT, only BLE+Wifi

[cyberman54]

@copercini my sniffer is using Wifi+BT+BLE in parallel. How can disable SW coexist from within arduino-esp32 framework?

[copercini]

@cyberman54 no, AFIK you can't use Wifi+BT without SW coexistence =/

[chegewara]

Strange, because last version (1.0.0) was compiled with SW balanced.

[cyberman54]

When i disable BT in new version, Wifi sniffing values climb to the same level as i see with previous version when Wifi + BT enabled.

How can i set balanced coexist with new version, so that configuration of previous version is restored?

[chegewara]

@cyberman54 Maybe its not the SW settings problem, it can be change in esp-idf and SW setting enabled is affecting your app performance no matter what option is chosen.

[cyberman54]

@chegawara i guess this is exactly my problem. But how can i solve this? The settings are in the precompiled libs, i guess?

[kscheff]

Do you use active BLE scanning? BLE needs to send out a scan request via radio, so this will be affected by any coexistence of Wifi.

…

Am 01.12.2018 um 19:16 schrieb Verkehrsrot ***@***.***>: @chegawara i guess this is exactly my problem. But how can i solve this? The settings are in the precompiled libs, i guess? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2125 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIfahkJF-8SNTQIQ-3PY02K-Z8t4yLdnks5u0sd0gaJpZM4Y3n6J>.

[cyberman54]

@kscheff no, i do passive sniffing (not scanning!). My code does not transmit, so no active BLE scan.

[chegewara]

@cyberman54 you have 3 options:

• convince @me-no-dev to recompile arduino library with SW balanced option (if it really is the case),
• recompile esp-idf with options you want, Compiling core at 80Mhz #1142 (comment)
• try to use arduino as component in esp-idf, https://github.com/espressif/arduino-esp32/blob/master/docs/esp-idf_component.md

[cyberman54]

@chegewara that means, there is no way to set SW balanced option during runtime?
I tried in my code

esp_coex_prefer_t coexist_config = ESP_COEX_PREFER_BALANCE;
ESP_ERROR_CHECK(esp_coex_preference_set(coexist_config)); // configure Wifi/BT coexist lib
coex_resume();

but this does not seem to have any effect.

[cyberman54]

@me-no-dev may i ping you on this problem?

[cyberman54]

Additional finding: if i deactivate bluetooth in my app by NOT calling

 btStart();
 ESP_ERROR_CHECK(esp_bluedroid_init());
 ESP_ERROR_CHECK(esp_bluedroid_enable());

then the board compiled with current (02.12.18) arduino-esp32 core version shows exactly the same Wifi sniffing results as the board compiled with previos core (before 26.11.18).

If i enable bluetooth on the board with current core, using BT start code shown above, the Wifi sniffing results immediately lower by 30%, compared with board using old core running Wifi + BT in parallel.

I think this proves that it is some kind of coexist side effect, not a performance issue.

[cyberman54]

Looks like this was the change causing this problem:
9233db8

[chegewara]

Maybe or maybe not, you cant know that without testing with old settings. Maybe performance drop is caused by esp-idf changes itself and with SW balanced performance drop will be even worse.

[cyberman54]

@chegewara wara to test this i would need current build, but excluding commit #9233db8 .
How could i compile this?

[chegewara]

#1142 (comment)

Use this sdkconfig in project:
https://github.com/espressif/arduino-esp32/blob/master/tools/sdk/sdkconfig

only you have to do is make menuconfig and change SW coex option.

[cyberman54]

@chegewara so i am now set with my toolchain and can now compile libraries with tailored sdkconfig settings, thanks to your instruction. 👍

But which of the generated libraries shall i use for linking to my app? Only libesp32.a?

[chegewara]

Never been doing this, but i would start with all libraries from this:
9233db8

or at least those you think you need to change, like wifi, bt and coex.
There is few more people around here who has better knowledge in this topic.

@copercini @lbernstone

[kscheff]

I am using the ESP32 to receive data via the scan response paket, so this requires an active scan. One strange observation is that the ESP32 seems to have a very limited receiving distance compared to an iPhone. So customers are unhappy with the narrow receiving range. But when the iPhone is doing an active scan, suddenly the ESP32 receives data in a greater distance radius. First I thought it is related to any other field disturbance, but it is reproducible... so this might also be related to the BLE (coex?) configuration. Therefore I need to dig into the same direction in order to overcome the limited range problem.

…

Am 02.12.2018 um 23:45 schrieb chegewara ***@***.***>: Never been doing this, but i would start with all libraries from this: 9233db8 or at least those you think you need to change, like wif, bt and coex. There is few more people around here who has better knowledge in this topic. @copercini @lbernstone — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cyberman54]

I now have my toolchain ready to compile esp32 libs after customizing sdkconfig, so could start with testing szenarios. I made some slight progress in testing and want to share results.

Test setup is: two same esp32 hardware boards, same external antenna (U.FL connected), same RF conditions (nearly the same, both antennas close together in a medium busy environment of moving Wifi/BT clients, around 100 per hour).

Board A: fixed configuration; runs previous ardunio-esp32 core with coexist=no
Board B: varied configuration: runs current arduino-esp32 core with different libs/coexist settings

Test 1: B = coexist=yes (no libs modified), bluetooth enabled. Result: 30% loss on wifi, 60% loss on bt sniffing

Test 2: B = coexist=yes (no libs modified), bluetooth enabled in sdkconfig, but disabled during runtime by calling btStop(). Result: 5-10% loss in wifi sniffing

Test 3: B = coexist=no (libesp32.a changed), bluetooth enabled in sdkconfig, but disabled during runtime by calling btStop(). Result: same as A, no loss.

So we have the proof, that the coexist setting in sdkconfig triggers this issue.

But why? Could be
a) packet loss due to timing problems -> packets missed -> performance issue, perhaps not caused by coexist software but side effect by other changes in arduino-esp32 core
b) packet loss due to software stack problem -> packets not missed, but not delivered to user callback function
c) some side effect causing not a packet loss, but reducing the rssi reception level of esp32 RF radio -> missed packets

I will try to go deeper in analysis.

[cyberman54]

@kscheff iphone will probably have 2,4 GHz foil antenna inside. Those antennas usually have better dB level than the simple on chip pcb antenna of ESP32. So if you do active BT scan with iphone you will probably reach more bluetooth beacons in range than you would with ESP32 antenna. But ESP32 receives the answers of the BT clients, too. So far just theory, but would explain your finding and if valid this would not be caused by coexist software.

[kscheff]

On my side there are many ESP32 devices out in the field and I get reports that different firmware gets different results. But its far away from expected performance. Some devices do not even reach 5 meters. The issue here is that I have not good access and I have only access via the users (which are already a bit upset). The devices operate mainly in Bluetooth mode, but Firmware gets updated only via WiFi mode. So that’s why your findings might open up some tests on my side to play around with the same parameters. Thanks for sharing these.

…

Am 04.12.2018 um 23:11 schrieb Verkehrsrot ***@***.***>: @kscheff iphone will probably have 2,4 GHz foil antenna inside. Those antennas usually have better dB level than the simple on chip pcb antenna of ESP32. So if you so active BT scan with iphone you will probably reach more bluetooth beacons in range than you would with ESP32 antenna. But ESP32 receives the answers of the BT clients, too. So far just theory, but could explain you finding and if valid this would not be caused by coexist software. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cyberman54]

@kscheff 5 meters range is sure far too low.

[chegewara]

@cyberman54 earlier version of arduino was compiled WITH SW coex but set to balanced. Could you try this option either since you already have prepared environment, please?

[cyberman54]

@chegewara you're right, thanks for the hint. I will continue my test over weekend by reproducing the correct settings before and current.

Changes in CONFIG_SW_COEXIST were:
PREFERENCE: changed from "BALANCE" to "WIFI"
PREFERENCE_VALUE: changed from "2" to "0"

Does anyone here know the meaning of PREFERENCE_VALUE? I can't find anything on this, not in the ESP32 Docs, not with Google Search...

In earlier version settings in sdkconfig are:

CONFIG_SW_COEXIST_ENABLE=y
CONFIG_SW_COEXIST_PREFERENCE_WIFI=
CONFIG_SW_COEXIST_PREFERENCE_BT=
CONFIG_SW_COEXIST_PREFERENCE_BALANCE=y
CONFIG_SW_COEXIST_PREFERENCE_VALUE=2

In current version it's:

CONFIG_SW_COEXIST_ENABLE=y
CONFIG_SW_COEXIST_PREFERENCE_WIFI=y
CONFIG_SW_COEXIST_PREFERENCE_BT=
CONFIG_SW_COEXIST_PREFERENCE_BALANCE=
CONFIG_SW_COEXIST_PREFERENCE_VALUE=0

[chegewara]

CONFIG_SW_COEXIST_ENABLE=y
[0]CONFIG_SW_COEXIST_PREFERENCE_WIFI=y
[1]CONFIG_SW_COEXIST_PREFERENCE_BT=
[2]CONFIG_SW_COEXIST_PREFERENCE_BALANCE=
CONFIG_SW_COEXIST_PREFERENCE_VALUE=0

[cyberman54]

CONFIG_SW_COEXIST_ENABLE=y
[0]CONFIG_SW_COEXIST_PREFERENCE_WIFI=y
[1]CONFIG_SW_COEXIST_PREFERENCE_BT=
[2]CONFIG_SW_COEXIST_PREFERENCE_BALANCE=
CONFIG_SW_COEXIST_PREFERENCE_VALUE=0

okay, understood, VALUE is redundant NUM for preference setting.
So we have 6 options:

ENABLE {0, 1} * VALUE {0, 1, 2}

During runtime we have option to switch BT on/off by btStart() / btStop().

So alltogether 12 options to test. I will try to full explore this over weekend.

[chegewara]

Actually only 4 options.
When ENABLE=n no matter what VALUE is, does not count.

[cyberman54]

Actually only 4 options.
When ENABLE=n no matter what VALUE is, does not count.

We expect that it should be like that, but how do we know? coexist lib is not open sourced yet.

[cyberman54]

Did the test by re-compiling libesp32.a with different settings in make menuconfig. Shows no effect at all.
I assume that this issue is a side effect caused by changes in the BT and Wifi subsystem, since there were a lot of changes after my last stable version of Nov. 26th 2018:

https://github.com/espressif/esp32-bt-lib/commits/master/libbtdm_app.a

https://github.com/espressif/esp32-wifi-lib/commits/19162cc2a106fb1285c5ae7158894ed7939f4230/libcoexist.a

[chegewara]

@cyberman54 This is what i am trying to suggest from beginning.

[cyberman54]

@chegewara if i had listen to you, this would have saved me lot of time wasted on this :-(

[chegewara]

I would not say its waste of time. You learned how to use arduino as component, alos you can try to open issue in esp-idf repository.

[cyberman54]

Allright, we close this here. It's an ESP-IDF issue.

[cyberman54]

To complete this: I also tried build my code with all current (08th Dec 2018) libs of current esp-idf. Coexist then shows upgraded version 1.1.2, but the performance drop when wifi/bt sniffing in parallel persists, regardless of coexist settings in sdkconfig. If i disable bluetooth i get same wifi sniffing results i got before with previous version and wifi/bt sniffing operated simultaneously.

Very bad :-(