"Packet received out-of-order. Expected 1; got 0." happening infrequently #612
Comments
|
@MosheL please provide information on your client-side enviroment (.NET Framework, version, OS, MySqlConnector version) and server-side environment (DB server brand, version, hosting provider). Are you able to provide a packet capture from when this happens? Are there any errors in your MySQL server logs? |
bgrainger
added
the
waiting for answer
|
.NET Framework 4.7.2, 4.6.1 I am reciving this error on a ~ 0.01% of the requests, while connecting. |
|
about packet capture - I will try to write a test code. it will take time. |
|
0.38.0 is fairly old; are you able to update to 0.49.2? I'm not aware of a specific bug fix that has been made that would solve this problem, but it's usually good to be running the latest version. As best I can tell, the call to A packet capture would likely show us exactly what's going on (if we could isolate it down to the failing thread easily). |
|
upgraded to the last version. the same error on production (MariaDB 10.3.11). I tried to repro the bug on my local machine (Windows, MariaDB 10.3.9) and I couldn't. for now, I am starting from a simple test (sorry for the VB, wrote years ago to explain a bug on Oracle connector. I am using this code under IIS, same IIS version like production. things I want to test: |
|
10.3.13: no problems |
|
update: tested 10.2.16. In the test no problem, in a production load - the same problem on the same rate. |
|
I'm not able to build a test harness that reproduces any connection problems with MariaDB 10.2.16; have you had any success? |
|
I didn't see any problem on any test, not with 10.2.16 and not with 10.3.11/12/13. |
|
Please test again with 0.50.0, which fixes #613. |
|
Tested on production for a month. |
|
Packet received out-of-order. Expected 1; got 84 MySQLConnector 0.56, 10.3.15-MariaDB |
|
I was sent by mail a PCAP with the problem. |
|
Here's an updated exception stack: The exception is coming from
Here's the response:
In yellow is the expected "OK" packet that is the response to the ping. In red is an unexpected sequence of bytes that's being interpreted as packet with length 14 (first three bytes) and sequence number 0 (4th byte). I believe this is causing the error. I'm not yet sure if Wireshark's warnings about "previous segment not captured" and "out of order" are relevant to this problem. Is that just an error with the packet capture that's leading me down a rabbit trail, or is that a symptom of whatever is causing the problem? |
|
|
|
On closer inspection, it looks like 11 bytes of garbage data, followed by one OK packet (sequence number 1, flags = AUTOCOMMIT | MORE_RESULTS_EXISTS) followed by another OK packet (sequence number 1 again, flags = AUTOCOMMIT).
If this is really what the client is receiving, it really looks like either a server-side bug or an unreliable network that's corrupting packets. One way to diagnose the latter might be to also perform a packet capture from the server-side, then compare the client-side and server-side captures. |
|
the packet you see was captured from server side. |
|
Does your comment at #651 (comment) apply to this case, too? I.e., removing the |
|
I am testing right now some ideas. I think the problem is not the var s2 = sql.Split(';');
for (int i = 0; i < 2200; i++)
{
foreach (var current in s2)
{
var Cmd = new MySqlCommand(current);
Cmd.Connection = Conn;
using (var oReader = Cmd.ExecuteReader())
{
oReader.Read();
}
}
}Is working OK. |
|
When I am stopping to pass two (or more) queries on the same job, the connection problem was disappeared also. for (int i = 0; i < 2200; i++)
{
using (var Conn = new MySqlConnection("server= pooling=true;Connection reset =false; Min Pool Size=5;Max Pool Size=20; Connect Timeout=10;default command timeout=30;UseAffectedRows=true;Allow User Variables=true; Allow User Variables=true; CharacterSet=utf8mb4"))
{
Conn.Open();
foreach (var current in s2)
{
var Cmd = new MySqlCommand(current);
Cmd.Connection = Conn;
using (var oReader = Cmd.ExecuteReader())
{
oReader.Read();
}
}
}
}
|
I haven't seen that to be a problem on any other MySQL version. (OTOH, I haven't seen user variables cause a problem either...) It's an interesting data point that it may require a combination of the two. |
|
Its the combination. string sql = @" SET @hereami = POINT(32.08373,34.8965);
select * from search_sites limit 1; ";
// failed
string sql = @" SET @hereami =1;
select * from search_sites limit 1; ";
// failed
string sql = @" /*SET @hereami = POINT(32.08373,34.8965);*/
select * from search_sites limit 1; SELECT types.id , concat( min(types.name) ,' (', count(st_site) ,')') as name, min(parent.name) parent, min(parent.id) parent_id , min(parent.display_order) display_order1, min(types.display_order) display_order FROM types ,types as parent , sites_types where st_type=types.id and parent.id=types.parent_id and types.id > 0 and types.parent_id=1686 ";
//working
Same code, working OK on a other 10.3.15, Installed on Windows 10 x64. |
bgrainger
added
invalid
and removed
waiting for answer
From https://jira.mariadb.org/browse/MDEV-16308:
The symptoms (two EOF packets, malformed OK packet) and causes (multi-statement batch where first statement is Closing this as an external bug, and duplicate of #651. |
|
Just to document it somewhere: on my dev machine (same version of 10.3.15 @ windows 10) - no problems, with stress test or not. |
|
This seems to be easily reproducible by shutting down the server while doing a query. Here's a gist describing how I reproduced it: https://gist.github.com/markus456/fee40f72d83b4ee47ba2625fb67e3cf3 Also there is a stacktrace and a network capture showing that the server does send non-sequential packets on shutdown. Edit: I'm no expect but a look at the code suggests that this could be avoided by adding a check for packets with sequence number 0 inside |
Agreed: #662. |
|
i got same error with "Packet received out-of-order. Expected 1; got 2.", thanks! |
I can see the same error message in a ~ 0.001% of my requests, while connecting to server.
I do not us a Galera cluster.
my stack:
Originally posted by @MosheL in #267 (comment)
The text was updated successfully, but these errors were encountered: