Checksum error in log file

[Eric-FR]

When trying to convert ubx files logged with Surveyor or Express to .pos file, I often get a checksum error (script : https://github.com/Stefal/ubx-to-gpx).

When logging with the adalogger, I never see such an error (logger : https://github.com/PaulZC/F9P_RAWX_Logger).

Error message:

Sample file displaying one checksum error:
SFE_Express_210704_224834.zip

Right now, it is not critical as I still get the expected .pos file. RTKLib is also not complaining.

[nseidle]

I believe this is related to previous issues. It will be very hard to eliminate all possible logging errors while supporting BT SPP in both directions at unknown throughput rates.

Just so I understand, you saw 1 error in 1MB log file? That's a good data point for me to know.

And you used Stefal's conversion script to convert UBX to POS. Ok, neat. That could be a much easier tool to test checksum errors.

What are the 'second must be in 0..59' errors?

[Eric-FR]

Concerning the previous issue with BT congestion, I don't know if this is related. I'm using 1 Hz for all frequencies/parameters so need for bandwidth should be limited. I will check circumstances on some samples.

About the '0..59' error, this seems to be related to underlying libraries used for reading and converting ubx files, something like time was 10 mn 59 sec 999 1/1000s. After conversion and rounding, it should end up like 10 mn 60 s (instead of 11 mm 00 s).

[Eric-FR]

I made a new try this night with static recording outside of the house and no BT/RTCM correction. I still get checksum errors (20 for 10 hours, acquisition rate at 1 Hz).

[pyrog]

During several test in v1.7RC I got corrupted data.

$GNGST,075122.25,63,5.5,2.4,74,1.1,2.2,2.9*46
”�{C �ÇW˚"��P†������ �Ä��Vé†àòåîì‰s.�*Ñ/Ó∂{Ëøú>ΩpfË�0‹fl‡�∞ƨ|ıÜÛy�¸áØÔ‰˛*}˜Ûh�\@‡-�(;�{H�Lj™™†™†�7œDq8∆vƒ–������������������"””�&C¿�…Qo‚���������� ���h(ÿfi±ƒ�%�ƒ�~Wî’�XÜÓ#”�mD`�ÇW˚"��(@�B���� ���v˙≤™zÚ™vfiâ±	ˇ1)ah¬M•W‘�ıQ›#e«>jç‘+·»«É–fi€z˛‘1ˇd;@`ÿˆÉÖ/Γü©mUUKAU@E4≤Îè,ı5�������������‰€q”��F@�ÇW `���������� ���Rd#B·¨˙R¿pÙÉ”��L‡�ÄÌÌ÷
$GNRMC,075123.00,A,4634.3597352,N,00544.7536378,E,0.030,,250921,,,A,V*1F
$GNGGA,075123.00,4634.3597352,N,00544.7536378,E,1,12,0.68,547.180,M,47.205,M,,*41

$GNGST,075123.00,63,5.5,2.4,74,1.1,2.1,2.9*43
™�é‡;`ÄlìÅÀí�ÒòIË’UMAU@E4≤Îè,ı5�������������‹i�”��F@�ÇW0����������� ���Rd�)¡†π“¿N¯n”��L‡�ÄÌÌ÷â∞ �1(¸-˜«∞Lï)x�fiÅ≥ÉŒ�≤$i¸É�Ó∫��Æ�fi™�é‡;`ÄlìÅÀí�ÒòIË’UMAU@E4≤Îè,ı5�������������‹i�”��F@�ÇW0����������� ���Rd�)¡†π“¿N¯n”��L‡�ÄÌÌ÷
$GNRMC,075124.00,A,4634.3597384,N,00544.7535659,E,0.018,,250921,,,A,V*1C
$GNGGA,075124.00,4634.3597384,N,00544.7535659,E,1,12,0.68,547.267,M,47.205,M,,*42

$GNGST,075124.00,61,5.4,2.4,74,1.1,2.1,2.9*47
”�äC �ÇX�b��P†��Ä��� �Ä��V£®"&#%$•¯ú£Ü†Î€¥´ûD:Ê%†Cê∆‚�}ƒYî˝Ò9“�Äx⁄·◊»ÇNR�ÚH5ñ¿øî}√��¿òRÈ�pú} ò��e’’–]–0�_=2D‚˘[�M�������������������������,¿@”�&C¿�…Qè"���������� ���h(ÿ÷©˛_≈G˛Nˇò�’éZflè�”�mD`�ÇX�b��(@�B���� ���v˙≤™zÚ™v‚â∞™�1(ñÈ-Aº7UÊ˙QflˇÏ¿]z’Ù¥�@òY∞ˇJ�zÍ�ÔºH;|fl˚���Ò¸�oÍl7UOAW@ET≤Îo¨ı5�������������˚¯–”�&F@�ÇW?†���������� ���iâ>‰Å£®ƒ_£û·î™•ÕX‚flè
$GNRMC,075125.00,A,4634.3597370,N,00544.7535167,E,0.032,,250921,,,A,V*14
$GNGGA,075125.00,4634.3597370,N,00544.7535167,E,1,12,0.68,547.320,M,47.205,M,,*40

…

In the car return trip, corrupted data appears when I tried to start WIFI AP (the car was parked at the red dot)

[pyrog]

35 errors during the first minute of the test.

SFE_Surveyor_210926_140214.ubx

[nseidle]

I have some decent progress to report. I believe I've reduced, and possibly eliminated, the number of write errors. I need to run some very long logs but it looks promising.

Changes I made:

• Increase size of RX buffer from 2k to 6k (with room to increase as needed)
• Move file synchronization and file date update out of the read task and into the main loop (file is updated every 5 seconds).
• Increase default SPI speed to 16MHz. This may cause issues with some SD cards so let me know if you see an issue. The speed can be lowered through the debug menu as needed.

[Eric-FR]

But, after upgrade, SPI frequency is still at 8 MHz in debug menu. I will make some testing at this frequency before increasing it to 16 MHz.

[nseidle]

The default was increased but your SD card settings file has it set to 8MHz. If you want, you can do a factory reset and it will make it 16MHz.

[pyrog]

Default settings (NMEA at 4 Hz)

Version	Distance	Duration	Errors	Comment
v1.7	48 km	45 mn	0
v1.7	38 km	3 days	12	2 records spaced 3 days apart sharing the same file
v1.8RC	38 km	35 mn	0	Without RTCM stream
v1.8RC	38 km	40 mn	0

The last test in v1.7 with errors was in fact a trip of 48 km and 48 mn, and a record of 87 seconds taken 3 days after (during WIFI AP test ??)

So it seem that since V1.8RC there is no corrupted data 😃

[pyrog]

Version	Distance	Duration	Errors	Comment
v1.8RC	34 km	31 mn	2	No binary data, only NMEA checksum errors

[pyrog]

Version	Distance	Duration	Errors	Comment
v1.8RC	7 km	1h47	0	Hiking
v1.8RC	41 km	1h14	0	Car trip

[nseidle]

Version	Distance	Duration	Errors	Comment
v1.8RC	0 km	~20 hours	0

@pyrog - How are you quantifying errors? I am running Stefal's ubx_convert and seeing no checksum errors.

900MB took more than 20 minutes 😲

[Eric-FR]

Last Monday, I made a car travel 2x300 km. I didn't found any checksum error in UBX files (and also no reboot).

v1.8RC-Oct1

[pyrog]

How are you quantifying errors?

I use JOSM (an OpenStreetMap editor).
I also used once gpsdecode that detected the same checksum errors.

$gpsdecode -v -D 1 < SFE_Surveyor_210730_08505.ubx >/dev/null

gpsdecode:WARN: NMEA0183: TXT: Warning: ESP_RST_POWERON
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: GPGSV field 3 value of 5 != actual count 13
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: NMEA0183: can't use GGA time until after ZDA or RMC has supplied a year.
gpsdecode:WARN: bad checksum in NMEA packet; expected 6B.
gpsdecode:WARN: bad checksum in NMEA packet; expected 41.

[nseidle]

Ah very good, thanks Pyrog!

I've just discovered @PaulZC integrity tool . It's pretty handy.

Processed 869965759 bytes
File size was 869965759
Longest UBX message was 2384 data bytes
Message types and totals were:
Message type: GNTXT   Total: 1
Message type: 0x02 0x15   Total: 286866
Message type: GNRMC   Total: 286866
Message type: GNGGA   Total: 286866
Message type: GNGSA   Total: 1147464
Message type: GNGST   Total: 286868
Message type: GPGSV   Total: 473039
Message type: GLGSV   Total: 364965
Message type: GAGSV   Total: 383338
Message type: GBGSV   Total: 215010
Message type: 0x02 0x13   Total: 1626321
Number of successful resyncs: 3

The above test was on RTK Express, 4Hz, 7 messages enabled, static antenna, no NTRIP or BT connection. On 5,357,603 messages, across 24 hours of logging, in a 870MB file, I had 3 corrupt messages or 0.00006%. So, while not flawless, it is 'good enough for government work' as we say in the USA.

Processed 759105239 bytes
File size was 759105239
Longest UBX message was 1744 data bytes
Message types and totals were:
Message type: GNTXT   Total: 1
Message type: GNRMC   Total: 345448
Message type: GNGGA   Total: 345448
Message type: GNGSA   Total: 1381792
Message type: GNGST   Total: 345448
Message type: GPGSV   Total: 552071
Message type: GLGSV   Total: 417581
Message type: GAGSV   Total: 411291
Message type: GBGSV   Total: 280427
Message type: 0x02 0x15   Total: 344481
Message type: 0x02 0x13   Total: 1184821

A 2nd test (above) was run on different hardware (RTK Express Plus), 4Hz, 7 messages enabled, static antenna, BT connected and NTRIP provided. 5.6M messages with no corruptions detected.

Thanks for your help guys! I'm closing this one as it seems to be fixed in v1.8 firmware. Please open a new issue if you start detecting errors again.

[Eric-FR]

I asked (here) @PaulZC for a repair tool. He added it to the integrity tool. It was able to repair a file with numerous errors (made with v1.7). So it should be OK for the few remaining errors with v1.8 (only one error for me until now).