Not working with MPU-9250 IMU, reads fail #1761
Comments
|
@MiloMindbender there is a known problem with Release 1.0.0, a fix has been applied to the main branch. Until a new release has been compiled you can apply the fix by substituting the four i2c related files from the github source to your Arduino package location. The Four files are:
under Windows 7 pro the Arduino Board Manager installs the package into:
Chuck. |
|
I will give this a try. Last night I did regression testing and found the MPU-9250 worked with 1.0.0-rc3 but failed on anything newer than that. Also, I've noticed the i2c frequency on ESP32 is off-spec. it's running 98khz when set to 100 and 382khz when set to 400. |
|
@MiloMindbender The ESP32 hardware uses clock cycle counting to control bus frequency. When it is sourcing SCL a single SCL cycle is constructed of four parts:
So, the bus capacitance and the pullup values distort the bus frequency from the pre-programmed values. Chuck |
|
I just replaced the 4 files, it works better but is intermittant. Some MPU9250 samples will start up ok the first time and maybe a time or two after pressing reset, then they start to fail. A full power off is required to get it working again. Some of the samples such as MPU9250BasicAHRS_I2C still fail every time. These are trying to read the WHO_AM_I register and always fail at that point. I tried inserting a 2 second delay after Wire.begin to give things a chance to stabalize, but this did not help. |
|
@MiloMindbender are you seeing any debug output? |
|
The only output I get is the usual stuff when you press the reset button. ets Jun 8 2016 00:22:57 rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) Do I need to set anything to get more debugging output? |
|
@MiloMindbender set the Core Debug Level to at least DEBUG and run you code, if the i2c system has any problems it will vomit out what it detects. Also, inside \cores\esp32\esp32-hal-i2c.c there is a #define that will increase debug support and verbosity of messages; Chuck. |
|
With just the DEBUG set and the line 45 mod I am getting nothing when it works. After pressing reset a few times I got it to fail and got this: [W][esp32-hal-i2c.c:1363] i2cCheckLineState(): invalid state sda=0, scl=1 [E][esp32-hal-i2c.c:1379] i2cCheckLineState(): Bus Invalid State, TwoWire() Can't init Logic analyzer shows sda low, scl high also. Even when I press/hold the reset button they remain in this state. Note this is with 10k pullups on the mpu 9250 board plus an additional 3.3k added externally. |
|
More info. I removed the 3.3k pullups and now it takes 6-7 tries to get a freeze. I also tried clipping the logic analyzer to the MPU-9250 pins, getting the setup into a frozen state and then disconnecting SDA/SCL from the ESP, this showed SDA low and SCL high also. So it appears the 9250 has gotten into a state where it's pulling SDA low for some reason. Since the ESP can't communicate with it, a software reset of the 9250 isn't possible, so it remains stuck this way till you power off...not sure why this happens. |
|
@MiloMindbender you can use If you can detect the lockup, try: if(lockedUP){
uint32_t timeout =millis();
while((millis()-timeout<1000)&&(!Wire.begin())){ // twiddle thumb
delay(5); // maybe IMU is busy? give it a chance to finish
}
}
If(!Wire.begin()){
Serial.println(" I2C bus did not Init, Fatal")
} else {
// bus is good, back to regularly scheduled programming.
}Chuck. |
|
Not sure if this is it or not...In the 9250 datasheet they mention "Each byte transferred must be followed by an acknowledge bit. To acknowledge, the slave device pulls the SDA line LOW and keeps it LOW for the high period of the SCL line. " When it fails on a reset of the ESP32 it appears that SCL is high and SDA has just gone low and it stays this way. It is doing a "setup read". So maybe the 9250 was in the middle of an ACK when reset was pushed and is waiting for SCL to go low, but it never does? Across a reset, should the SCL/SDA lines go low and back to high or just stay high? Not an expert, just guessing. |
|
@MiloMindbender The SDA and SCL lines are only pulled high by the pullup resistors, The bus is a Wired 'AND' configuration. To get a HIGH, all devices on the bus must stop pulling the Signal Low before it can go High. The I2C protocol has a defined method to recover from a SDA low lockup. SDA is released by the Master and 9 SCL clock cycles are pulsed on the bus. All I2C slave devices must release SDA when they receive this sequence. I just had a thought; maybe the IMU is out of bit sequence. it thinks it is working on bit 4 and the ESP32 thinks it is the ACK bit. Try this change to`esp32-hal-i2c.c' in the i2cCheckLineState function if(!digitalRead(sda) || !digitalRead(scl)) { // bus in busy state
log_w("invalid state sda=%d, scl=%d\n", digitalRead(sda), digitalRead(scl));
digitalWrite(sda, HIGH);
digitalWrite(scl, HIGH);
delayMicroseconds(5);
digitalWrite(sda, LOW);
for(uint8_t a=0; a<9; a++) {
delayMicroseconds(5);
digitalWrite(scl, LOW);
delayMicroseconds(5);
digitalWrite(scl, HIGH);
}
delayMicroseconds(5);
digitalWrite(sda, HIGH);
}Changed code: if(!digitalRead(sda) || !digitalRead(scl)) { // bus in busy state
log_w("invalid state sda=%d, scl=%d\n", digitalRead(sda), digitalRead(scl));
digitalWrite(sda, HIGH);
digitalWrite(scl, HIGH);
delayMicroseconds(5);
digitalWrite(sda, LOW);
for(uint8_t a=0; a<9; a++) {
delayMicroseconds(5);
digitalWrite(scl, LOW);
delayMicroseconds(5);
digitalWrite(scl, HIGH);
delayMicroSeconds(2);
digitalWrite(sda,HIGH); // simulate STOP
if(digitalRead(sda)){ // bus recovered, all done
break;
}
}
delayMicroseconds(5);
digitalWrite(sda, HIGH);
}Chuck. |
|
Will try to get your test code in shortly. In the meantime maybe this logic analyzer trace is helpful? This shows the bus working ok, then reset is pressed and it in the low/high state forever. Hope I explained myself correctly earlier, what I meant was that maybe when the ESP32 was reset the 9250 was working on an ACK and had just pulled SDA low, but because of the reset SCL is now high. the 9250 is waiting on an SCL low pulse that never comes, so it just stays stuck in that state? Like I said, not an expert, does the logic analyzer trace help? |
|
Just to be totally clear, this lockup only happens when reset is pressed and you often have to reset 7 or more times before it ends up in this state. The I2C does not appear to lock up at any other time. I'm letting one of the test programs run for a while to be sure. Also to be totally clear, in the logic analyzer trace above, where the signal stops is where reset was pressed and there is no activity on I2C once the ESP32 finishes its reset cycle. Also, just checking, would you recommend I pull down a zip of the latest code here and install it before putting in and testing your patch above? |
|
Grab the latest github main branch. I'll look at your trace, brb |
|
This snip of your trace is questionable, the i2c decode is showing a STOP, but the raw voltage trace does not show SDA going high? So, no stop was generated? Can you add the I did a Google search on the IMU and saw other with i2c SDA issues. (AVR STM and even Intel Edison) So, the IMU is the locus, but, I haven't seen the triggering factors. Chuck.
|
|
One guy said he could recover the IMU by connecting Vdd to SDA directly! BAD! BAD! Worse than BAD! |
If the ESP32 resets, It should boot the Arduino code and You would see this reset pulse on the scope. Chuck. |
|
At least with the current code, I don't see anything appear on the scope when reset completes, it just stays stuck in the low-high state. The test program has been running for about 30 minutes with no problems, so I think it is safe to assume this hang is caused by resetting the ESP. In the process of installing latest code and your patch. |
|
If that So, keep me advised if you see and reaction to the patch. Chuck. |
|
Ok, getting really confusing, This is what I see now, I press reset and it hangs on restart. This is what the bus looks like. I think I need to try and tie a trace to the reset button so I can see exactly where on this trace the reset is occurring.
|
|
All I'm seeing is that the transaction is aborted. that last read operation is up to the 6th bit of the first byte. Then the reset of the ESP32 happens. the Hardware reset stops the SCL clocking, but the IMU is in the middle of a byte read. From it's point of view it is just waiting for the next SCL cycles. During a i2c read, the master(esp32) supplys the SCL and the slave(IMU) controls SDA except for the 9th bit ACK signal. Every 9th SCL cycle the IMU allow the master to either ACK(low) the byte, with causes the IMU to again exclusively control the SDA pin for the next 8 clock cycles. To Terminate a read cycle the last byte the master(esp32) wants to read needs to be NAK'd(HIGH). The current if(!digitalRead(sda) || !digitalRead(scl)) { // bus in busy state
log_w("invalid state sda=%d, scl=%d\n", digitalRead(sda), digitalRead(scl));
digitalWrite(sda, HIGH);
digitalWrite(scl, HIGH);
delayMicroseconds(5);
digitalWrite(sda, HIGH);
for(uint8_t a=0; a<9; a++) {
delayMicroseconds(5);
digitalWrite(scl, LOW);
delayMicroseconds(5);
digitalWrite(scl, HIGH);
if(digitalRead(sda)){ // bus recovered, all done. resync'd with slave
break;
}
}
}If we are out of sequence with the bit count of the IMU this code should resync. Chuck. |
|
Yes, very confused.... you can see channel 2 is the reset button. The hang up happens exactly when the reset button is pressed. However when the reset button is released (not shown here), I see nothing at all happening to sda/scl. I also tried adding back the 3.3k pullups which squared up the analog signals a bit, but no change in what it was doing |
|
Sorry, we wrote at the same time...the above trace is before the new code you just posted....I'll retest in a minute. |
|
lengthen your capture window, we need to see what happens after EN(reset) go high for at least 1 second. |
|
change your trigger to trigger when EN go high. That is where the interesting stuff is. What you are showing is the failure, I want to see the recovery attempts. Chuck. |
|
I am capturing out 4 seconds after the reset goes high, well past the time when your "i2cCheckLineState" message appears. I just didn't show you it because SDA and SCL remain perfectly flat, not so much as a bobble even in the analog. Here is another capture of the fail and when reset goes high SDA and SCL stay low/high for at least 4 seconds. |
|
I ran the capture out 10 seconds after the reset, still nothing at all after the freeze occurs. Gotta run for a bit, be back in about 2 hours if you have any more suggestions. |
|
is channel 2 EN? I want to see the LOW->HIGH transition of EN. That is were the ESP32 boots, and after the boot loader start Arduino. maybe 500ms after EN low->high is where the recover should show up. This capture is showing the reset, everything is correct. It is the recovery that is failing. If SCL is high and SDA is low and you are seeing:
Without any SCL activity something is not correct. I'll do some thinking and see if I can add some useful debugging statements to track down this problem Chuck. |
|
Channel 2 is connected to a pin labeled reset on my board, which I assume is same as EN. And yes, for 10 seconds after reset goes high there is absolutely no change in SDA (low) and SCL (high) I see the "invalid state" messages almost the instant reset goes high, so that code is being executed, but it doesn't seem to be doing anything to SDA and SDL, both the analog and digital versions are dead flat for at least 10 seconds. I am pretty sure your modified code is being executed because the first version you gave me had a typo I had to correct, so I'm it is being compiled in. I could put some code in i2cCheckLineState that would flip some unused GPIOs for my logic analizer so we can see the exact timing of when the code runs, if that would help. At the moment though, it looks like that code is running but there is no change in state of the SDA/SCL lines when it does. |
|
@MiloMindbender I have spent the last few hours setting up tests, and they fail. The code in It will probably take a couple of days to figure out what went sideways. The reproducible error sequence is to reset the ESP32 during a i2c read, at the exact moment when the next bit to be output by the slave is a zero. This causes the fatal cascade. Hopefully I can resurrect my stickbreaker fork. I have been working on slave mode and my local copy is broken. I'll have to redownload the last working copy from github. It will be at least tomorrow, maybe Saturday before I get a chance to do this, I'm done for the day. I'm working on 5hr sleep in the last 48hr. just ran out of steam. Chuck. |
|
Thanks for looking at this, I can continue to work with the code in it's current state. One question, do you think it is something in the ESC32 that actually causes the lock up? Or is the ESC32 issue just that it isn't able to reset the bus to a good state? |
|
@MiloMindbender Just posted a PR that fixes this problem, try it. #1767 Chuck. |
|
Replacing the Wire.cpp, Wire.h, esp32-hal-i2c.c and esp32-hal-i2c.h? Does Espressif wants to be trustable? They should be careful when releasing SDK updates :-/ |
|
@xVinny You get what you pay for, If it doesn't meet your specification, You can provide the fix. This is an open source project. Instead of complaining, use that effort to find and solve problems. Chuck. |
|
@stickbreaker you are right, I'm already doing this, but what I'm talking is about the releases and not the current branch. I believe that Espressif pays someone(s) to control the release approval. |
|
@xVinny Ok, so to meet your design goals of a perfect system, the only possible solution is to never release. That way there is no possibility of an error being released. Personally I would rather have a release with no known errors, when errors are found, work to solve them. Chuck. |
|
@stickbreaker My apologies for delaying testing, I had to convert my MPU9250 setup back to using a BNO055 (which isn't as good an IMU but doesn't have this problem) so I could make some progress on a project. Just realized that I had another ESP32 board though, so I can set that one up to test this issue, hopefully this week. I'm a little new to Github, if I do the "update" procedure here https://github.com/espressif/arduino-esp32/blob/master/docs/arduino-ide/windows.md will I get the latest code including your PR 1767? Sorry , still getting used to terminology and not clear if I have to do something to get a particular PR? |
|
@MiloMindbender yes that is correct procedure. After you follow the "update" procedure, I recommend you do one small modification: in C:\Users{your user name}\Documents\Arduino\hardware\espressif\esp32\platform.txt change:
to
When you want to select the github version go into Tools -> Board -> Chuck. |
|
@stickbreaker I got the test hardware remade and was able to reproduce the problem. Then I did the update and am having build problems. If I select my board under ESP32Arduino it builds fine but does not run because of the I2C bug in 1.0.0. When I select my board under the ESP32 Arduino GitHub Version I get a ton of errors related to pthreads, did something go wrong with my update? I haven't seen any issues posted related to this. I had a look through and the only thing I can figure is that the #include in gthr-default.h didn't work or grabbed the wrong file? I've attached a file with the full error output, if I use the old ESP Arduino version I get a successful build with only the "warning" at the top. I have to go now but let me know if you have a solution. If I don't hear from you I'll probably just erase all my Arduino ESP32 versions and reinstall them in the morning. |
|
That error.txt output looks strange. It has references to the run This is what my sketch directory looks like with github dev installed. Chuck. |
|
@stickbreaker Here's my directory listing, I'm thinking maybe I better clean out the ESP32 stuff just on principal and re-install it. I've been using it for a while and maybe something got messed up with all the installs, patches, reloads...etc. |
|
Hmmm....used the boards manager to uninstall the RC1.0.0, the "github version" is still showing up on the menu but not when I just choose "boards manager" The code compiles, but it is behaving like it's still running the RC version (ie: I2C isn't even starting) I think something is definately messed up, I'm going to try and clear out the remaining ESP32 files and reinstall just the github version. |
|
@stickbreaker looks like SUCCESS. Cleaned out everything and re-installed Github ESP32 from scratch. Everything builds ok now. The Recovery process appears to be working...the debug verbose output looks like this on a hang The logic analyzer sees this
|
|
The release version is in The github development version resides in chuck. |
|
The recovered cycle count is the number of SCL clocks required for the slave to release SDA. It has to synchronize with the slave. It should always be less than 8. Because, a slave can only output 8 bits before it waits for the ACK bit. When the recovery sequence see the ACK bit it can now issue a STOP to abort the in progress transaction. |
|
Sounds good, if this solves you problem, you can close this issue. Thanks. Chuck. |
|
@stickbreaker Caught one that went 8 cycles, it looked like this
Now that the reset issue is fixed I will be running some more tests with the MPU-9250 today, if I don't encounter any other problems communicating with it I'll close the issue. Thanks for fixing this! P.S. Does the ESP32 native dev environment us this same code for I2C? |
|
The release 1.0.0 version has a BUG with ReSTART operations ( You should use the GitHub dev version until me-no-dev creates a new release. Chuck. |
|
in your scope example, the Slave was emitting a '0', eight lows. And, the i2c bus cycle happened to be at bit zero when the interrupt or reset happened.
The github dev version is the most up to date. Release 1.0.0 does not have the correct reset code. |
|
I had a ton of issues today, but none of them were I2C, they were all bugs in people's MPU-9250 libraries! So I will close now. By the way when I asked about the native dev environment I was referring to the Native ESP-IDF environment that runs FreeRTOS and NOT arduino. I was just wondering if they use your code too or if they have something different. |
|
@MiloMindbender IDF uses totally different structure. They require the app programmer to interact with the hardware at a much lower level: Arduino init Wire#define CLOCKFREQ 400000
Wire.begin(SCL,SDA,CLOCKFREQ);IDF init in Master mode/**
* @brief i2c master initialization
*/
static void i2c_example_master_init()
{
int i2c_master_port = I2C_EXAMPLE_MASTER_NUM;
i2c_config_t conf;
conf.mode = I2C_MODE_MASTER;
conf.sda_io_num = I2C_EXAMPLE_MASTER_SDA_IO;
conf.sda_pullup_en = GPIO_PULLUP_ENABLE;
conf.scl_io_num = I2C_EXAMPLE_MASTER_SCL_IO;
conf.scl_pullup_en = GPIO_PULLUP_ENABLE;
conf.master.clk_speed = I2C_EXAMPLE_MASTER_FREQ_HZ;
i2c_param_config(i2c_master_port, &conf);
i2c_driver_install(i2c_master_port, conf.mode,
I2C_EXAMPLE_MASTER_RX_BUF_DISABLE,
I2C_EXAMPLE_MASTER_TX_BUF_DISABLE, 0);
}Arduino Read from slaveWire.requestFrom(SLAVEID,count);
while(Wire.available()){
Serial.print(Wire.read(),DEC);
}IDF Read from Slave/**
* @brief test code to read esp-i2c-slave
* We need to fill the buffer of esp slave device, then master can read them out.
*
* _______________________________________________________________________________________
* | start | slave_addr + rd_bit +ack | read n-1 bytes + ack | read 1 byte + nack | stop |
* --------|--------------------------|----------------------|--------------------|------|
*
*/
#define ESP_SLAVE_ADDR 0x50
static esp_err_t i2c_example_master_read_slave(i2c_port_t i2c_num, uint8_t* data_rd, size_t size)
{
if (size == 0) {
return ESP_OK;
}
i2c_cmd_handle_t cmd = i2c_cmd_link_create();
i2c_master_start(cmd);
i2c_master_write_byte(cmd, ( ESP_SLAVE_ADDR << 1 ) | READ_BIT, ACK_CHECK_EN);
if (size > 1) {
i2c_master_read(cmd, data_rd, size - 1, ACK_VAL);
}
i2c_master_read_byte(cmd, data_rd + size - 1, NACK_VAL);
i2c_master_stop(cmd);
esp_err_t ret = i2c_master_cmd_begin(i2c_num, cmd, 1000 / portTICK_RATE_MS);
i2c_cmd_link_delete(cmd);
return ret;
}
#define BUFFSIZE 25
uint8_t buff[BUFFSIZE];
if( i2c_example_master_read_slave(I2C_EXAMPLE_MASTER_NUM. &buff,BUFFSIZE)==ESP_OK){
// write out buff
}Chuck. |
Hardware:
Board: Adafruit ESP32 feather
Core Installation/update date: 1.0.0
IDE name: Arduino IDE
Flash Frequency: 80Mhz
Upload Speed: 921600
Description:
I noticed in some closed issues there were problems with the MPU-9250 IMU that were apparently resolved, but it appears these problems may have come back.
My ESP32 is unable to communicate with the https://www.sparkfun.com/products/13762 9250 i2c breakout board. When I got this it seemed to be running fine but with the latest release of ESP32 arduino support this doesn't seem to work any more.
Samples from this repository
https://github.com/sparkfun/SparkFun_MPU-9250-DMP_Arduino_Library
or this one
https://github.com/sparkfun/SparkFun_MPU-9250_Breakout_Arduino_Library
Both fail in similar ways, the writes appear to go through and be acked and all the reads fail with a nak according to my Saleae logic analyzer. It is not even possible to retrieve the "who am I" value.
I have tried adding 3.3k pullups to the setup, but this does not make any difference.
I have tried using 100,000 and 400,000 speeds on Wire, no difference.
I have also double-checked the wires, only 4 (2 power and SDA/SCL, and they are correct. Everything is powered from the 3.3v regulator on the Adafruit ESP32 Feather Board.
On one of my computers at home these samples DO seem to work, but I think that machine might have an older release of the ESP32 arduino libraries, I will check that and report back soon.
If it would be helpful, I can post a dump of the I2C traffic from my logic analyzer (software to view it is free). If anyone wants it, let me know, I need to re-capture the data to shrink it's size a bit.
The text was updated successfully, but these errors were encountered: