Bug _mysql.c - binary flag?

[lonestarr86]

Hi,

Thanks for your effort providing us this driver.

However, I think there is a bug in _mysql.c. In the function "_mysql_field_to_python" there is a switch statement. In this statement the binary flag is always overwritten by the default block.
As a consequence I'm getting trouble processing binary data because the binary data is treated as text leading into conversion trouble.
I wanted to let you know about the problem, because you might want to check.

Thanks and regards!

[methane]

It is not a bug, as comment describes.
What field type do you using?

[lonestarr86]

Are you really sure about that?
I'm not an expert in C, but since there is no break after setting binary to 1, binary is always overwritten to 0 due to the default. The comment does not explain any implicit meaning behind that.

[methane]

You're right. But I don't have failing test case. What type do you use?

[lonestarr86]

I think it depends on the configuration. I use my own converter callbacks with the option "use_unicode=0"

conv = {
	252: ((128 ,_BINARYCONV  ),(None,_STRINGCONV)),
	250: ((128 ,_BINARYCONV  ),(None,_STRINGCONV)),
	251: ((128 ,_BINARYCONV  ),(None,_STRINGCONV)),
	253: ((128 ,_BINARYCONV  ),(None,_STRINGCONV)),
	254: ((128 ,_BINARYCONV  ),(None,_STRINGCONV)),
	....
}
MySQLdb.connect(
	...
	conv        = conv,
	use_unicode = 0,
	...
)

Hope this information is helpful...

[methane]

Uh, that's API I hate.... conv waste my hundreds hours. It's broken design.
Why you need it?

[lonestarr86]

Sorry to hear that. We tend to manage any encoding operations on our own. The conv seems to be a reasonable tool. Of course, we can workaround that with use_unicode, but that doesn't solve the bug.

[methane]

Of course, we can workaround that with use_unicode, but that doesn't solve the bug.

If you didn't override these types, this switch statement is not used.
That's why only you are affected by this bug.

https://github.com/PyMySQL/mysqlclient-python/blob/1a67ae2cc2da1a2bf511cf70b20dcd5fca5ba7fe/MySQLdb/_mysql.c#L1110-L1123

[lonestarr86]

I was already wondering why no others were affected. Thanks for your attention.

[fried]

I just hit this same bug today, and have pull request.

[fried]

So if this is broken it becomes really hard to deal with databases that have bad charsets or what have you. The example I have is we have some really old db's where the charset is utf8, but some bad C++ code was committing truncated data to the table, and thus was not possible to decode it, The team fixed the C++, but it might be still possible that some of this data still exists, so they have a converter that opens a task for a human to resolve, but returns the value by doing a decode(utf8, 'ignore')

[methane]

Even though this bug is fixed, I won't guarantee the callback is supported in such ways.
Use only use_unicode=False. Don't override callbacks for strs.