Expected to read 4 header bytes but only received 0

[yang-xiaodong]

Hello Bradley, I met a exception in my production environment, the exception message is the title.

Environment:

• MySqlConnector Version : 0.40.4
  

• .NET Core SDK Version : 2.1.300
  

• ProxySQL Version: 1.4.8
  

It looks similar to this one #495 issue comment

When I removed ProxySQL, the exception did not reappear. so I wrote a separate test program to connect to ProxySQL, but did not reproduce it.

Would you give me some guidance for this exception?

This is my stack:

MySql.Data.MySqlClient.MySqlException: Failed to read the result set.
 ---> System.IO.EndOfStreamException: Expected to read 4 header bytes but only received 0.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at MySqlConnector.Utilities.ValueTaskExtensions.<ContinueWith>d__0`2.MoveNext() at offset 235 in C:\projects\mysqlconnector\src\MySqlConnector\Utilities\ValueTaskExtensions.cs:line 8:col 141
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at MySqlConnector.Utilities.ValueTaskExtensions.<ContinueWith>d__0`2.MoveNext() at offset 123 in C:\projects\mysqlconnector\src\MySqlConnector\Utilities\ValueTaskExtensions.cs:line 8:col 141
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at MySqlConnector.Core.ServerSession.TryAsyncContinuation(System.Threading.Tasks.Task`1 task) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 1104:col 4
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at offset 36
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) at offset 82
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task& currentTaskSlot) at offset 158
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at MySqlConnector.Core.ResultSet.<ReadResultSetHeaderAsync>d__1.MoveNext() at offset 270 in C:\projects\mysqlconnector\src\MySqlConnector\Core\ResultSet.cs:line 43:col 6
--- End of inner exception stack trace ---
at MySql.Data.MySqlClient.MySqlDataReader.ActivateResultSet(MySqlConnector.Core.ResultSet resultSet) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 92:col 4
at MySql.Data.MySqlClient.MySqlDataReader.<ReadFirstResultSetAsync>d__88.MoveNext() at offset 146 in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 324:col 4
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at MySql.Data.MySqlClient.MySqlDataReader.<CreateAsync>d__87.MoveNext() at offset 160 in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 309:col 5
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at MySqlConnector.Core.TextCommandExecutor.<ExecuteReaderAsync>d__3.MoveNext() at offset 467 in C:\projects\mysqlconnector\src\MySqlConnector\Core\TextCommandExecutor.cs:line 73:col 6
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at Dapper.SqlMapper.<QueryAsync>d__33`1.MoveNext() at offset 458 in C:\projects\dapper\Dapper\SqlMapper.Async.cs:line 419:col 21
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at DotNetCore.CAP.MySql.MySqlStorageConnection.<GetPublishedMessagesOfNeedRetry>d__8.MoveNext() at offset 260
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at DotNetCore.CAP.Processor.NeedRetryMessageProcessor.<ProcessPublishedAsync>d__6.MoveNext() at offset 112
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) at offset 46
at DotNetCore.CAP.Processor.NeedRetryMessageProcessor.<ProcessAsync>d__5.MoveNext() at offset 191
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at offset 12
at DotNetCore.CAP.Processor.InfiniteRetryProcessor.<ProcessAsync>d__3.MoveNext() at offset 119

Reference:

Code of DotNetCore.CAP.Processor.InfiniteRetryProcessor

Code of DotNetCore.CAP.MySql.MySqlStorageConnection

[bgrainger]

A consistent reproducible test case would be great, but if that's not possible...

Does the exception occur frequently (or predictably) in production? Would it be feasible to get a packet capture (with Wireshark or tcpdump)? (This would require SslMode=None so we can read the traffic.) We would need some context from the request that throws the exception to be able to narrow down to the TCP stream that causes the problem.

[bgrainger]

(FWIW, we have used packet captures to identify problems in proxies before, e.g., #267, so it's not outside the realm of possibility that this is a new (or known) bug in ProxySQL.)

Does ProxySQL's error log contain any pertinent information?

[yang-xiaodong]

I found the following exception information in the ProxySQL log,

MySQL_Session.cpp:2816:handler(): [ERROR] Detected a broken connection during query on (2,10.xxx.xxx.49,3380) , FD (Conn:36 , MyDS:36) : 2013, Lost connection to MySQL server during query

Maybe this is a ProxySQL bug

https://github.com/sysown/proxysql/issues?utf8=%E2%9C%93&q=is%3Aissue+Detected+a+broken+connection+during+query

[yang-xiaodong]

We finally found the cause of this error.

This is ours mysql configuration :

show variables like '%timeout%'

interactive_timeout 30
wait_timeout 30

That the connection was dropped by MySQL because the connections got older then wait_timeout