Getting unsigned values for INT column

[karmakaze]

Set up to use row-format replication from MySQL 5.6.5, I'm getting large positive values in place of small negative ones for a column defined as INT which should be 32-bit signed. Here's a part of the debug log:

2016/01/07 20:04:27 [DEBUG] === UpdateRowsEventV2 ===
Date: 2016-01-07 15:48:58
Log position: 1007
Event size: 72
TableID: 74
Flags: 1
Column count: 4
Values:
--
0:566
1:2379
2:4294967295
3:"2016-01-07 15:44:32"
--
0:566
1:2379
2:4294967294
3:"2016-01-07 15:44:32"

The column '2' value was changed from -1 (received as 4294967295) to -2 (received as 4294967294).
I looked at the TableMapEvent for this table and didn't see anywhere that an unsigned 'meta' bit was set.

I even changed the schema of column '1' to be unsigned and I still didn't see any unsigned 'meta' bit in the debug output. There's a lot of binary data and it's possible I missed it, but the first case of an INT field -1 => -2 being reported as 4294967295 => 4294967294 stands.

[karmakaze]

I think what's going on here is with binary replication the 'bits' are meant to be transferred and applied, as signed/unsigned have the same bit pattern, no problem. What the library is doing is being helpful returning int64 for all integral types to reduce the number of type cases to handle. Where this goes off is widening an 8, 16, 24, or 32 signed value.

e.g. return int64(binary.LittleEndian.Uint32(data)) will return values like 4294967295 for -1.

I tested some changes in a fork to always sign-extend while widening (even for the INT24 case to int32 then int64). This worked much better.

I continued on to then return the various int8 thru int64 interface{} types meaning the user will need to expect these. The advantage is that there's no possibility of data mangling (-1 -> 4294967295) and if you want the unsigned version a cast will convert.

I'll see about making a Pull Request to see this. My branch has import package renaming (not yet using Go 1.5 vendor'ing feature) that needs to be tidied before I do so.

Note: There might be a similar issue with date handling for (negative) dates before Jan. 1, 1970, but I didn't really look into that.

[siddontang]

Can we close it now?

[karmakaze]

Thanks for accepting PR#27, we can close the PR. Please also consider PR#28 as well before closing the issue.

The follow-up PR #28 returns smaller (and unsigned) integer or float32 types for interface{}. There's notes there for pros/cons. I think the upside is greater with a slight non-backward compatibility (users have to also handle int8, uint8, float32, etc in addition to int64 and float64).

[siddontang]

Thanks, I will close it and move another. :-)