Add more MySQL-8.0 meta data to GTIDEvent and TableMapEvent #468

huangjunwen · 2020-01-31T02:38:29Z

This pr adds more meta data added after MySQL-8.0:

GTIDEvent:

immediate/original commit timestamp for this trx
transaction length (in bytes) for all binlog events of this trx, including this GTIDEvent. This is useful to detect transaction boundry
immediate/original server version

TableMapEvent:

column names
primary key info
signedness info for numeric columns
collation info for character and enum/set columns
geometry type info

These can be hopefully to solve #427 if using MySQL-8.0

Example GTIDEvent dump (MySQL-8.0):

=== GTIDEvent === 
Date: 2020-02-01 19:15:26
Log position: 3440812
Event size: 88
Commit flag: 1                                                                                                                                           
GTID_NEXT: 5aa72a7f-44a8-11ea-947f-0242ac190002:55
LAST_COMMITTED: 54
SEQUENCE_NUMBER: 55
Immediate commmit timestamp: 1580555726309342 (2020-02-01T19:15:26.309342+08:00)
Orignal commmit timestamp: 0 (<n/a>)
Transaction length: 197
Immediate server version: 80019
Orignal server version: 0

Example GTIDEvent dump for the same event (MySQL-5.7):

=== GTIDEvent ===
Date: 2020-02-01 19:15:26
Log position: 15156
Event size: 65
Commit flag: 1
GTID_NEXT: 5aa72a7f-44a8-11ea-947f-0242ac190002:55
LAST_COMMITTED: 49                                                                                                                                       
SEQUENCE_NUMBER: 50
Immediate commmit timestamp: 0 (<n/a>)
Orignal commmit timestamp: 0 (<n/a>)
Transaction length: 0
Immediate server version: 0
Orignal server version: 0

Example TableMapEvent dump (MySQL-8.0):

=== TableMapEvent ===
Date: 2020-03-10 15:24:58
Log position: 78747
Event size: 580
TableID: 118
TableID size: 6
Flags: 1
Schema: test
Table: _types
Column count: 42
Column type: 
00000000  10 01 01 02 09 03 08 f6  04 05 01 02 09 03 08 f6  |................|
00000010  04 05 0d 0a 13 13 12 12  11 11 fe 0f fe 0f fc fc  |................|
00000020  fc fc fc fc fc fc fe fe  ff f5                    |..........|
NULL bitmap: 
00000000  00 00 fc c0 ff 03                                 |......|
Signedness bitmap: 
00000000  00 7f 80                                          |...|
Default charset: []
Column charset: [224 224 63 63 63 63 63 63 224 224 224 224]
Set str value: [[1 2]]
Enum str value: [[a b]]
Column name: [b_bit n_boolean n_tinyint n_smallint n_mediumint n_int n_bigint n_decimal n_float n_double nu_tinyint nu_smallint nu_mediumint nu_int nu_bigint nu_decimal nu_float nu_double t_year t_date t_time t_ftime t_datetime t_fdatetime t_timestamp t_ftimestamp c_char c_varchar c_binary c_varbinary c_tinyblob c_blob c_mediumblob c_longblob c_tinytext c_text c_mediumtext c_longtext e_enum s_set g_geometry j_json]
Geometry type: [0]
Primary key: []
Primary key prefix: []
Enum/set default charset: [224]
Enum/set column charset: []

Example TableMapEvent dump for the same event (MySQL-5.7):

=== TableMapEvent ===
Date: 2020-03-10 15:24:58
Log position: 31058
Event size: 133
TableID: 117
TableID size: 6
Flags: 1
Schema: test
Table: _types
Column count: 42
Column type: 
00000000  10 01 01 02 09 03 08 f6  04 05 01 02 09 03 08 f6  |................|
00000010  04 05 0d 0a 13 13 12 12  11 11 fe 0f fe 0f fc fc  |................|
00000020  fc fc fc fc fc fc fe fe  ff f5                    |..........|
NULL bitmap: 
00000000  00 00 fc c0 ff 03                                 |......|
Signedness bitmap: 
Default charset: []
Column charset: []
Set str value: []
Enum str value: []
Column name: []
Geometry type: []
Primary key: []
Primary key prefix: []
Enum/set default charset: []
Enum/set column charset: []

ref:

names/primary key

…ned integer

IANTHEREAL · 2020-02-22T08:05:41Z

Cool! We will take a look ASAP @huangjunwen

IANTHEREAL · 2020-02-22T08:08:15Z

@csuzhangxc @WangXiangUSTC This pr enhances the mysql 8.0 binlog replication protocol, PTAL

siddontang · 2020-02-23T03:43:30Z

can you help us test it for MariaDB 8.0? @huangjunwen

huangjunwen · 2020-02-23T04:28:58Z

ok, i will check the compatibility of MariaDB later

csuzhangxc

cool, rest LGTM.
BTW, can you add some unit tests?

csuzhangxc · 2020-02-24T09:39:39Z

mysql/util.go

+	case 1:
+		u |= uint64(b[0])
+	default:
+		panic(fmt.Errorf("BytesToUint64 byte slice length must be in range [1, 8]"))


How about returning an error instead of panic as other func did in this file?

csuzhangxc · 2020-02-24T12:15:56Z

replication/event.go

@@ -349,6 +350,18 @@ type GTIDEvent struct {
 	GNO            int64
 	LastCommitted  int64
 	SequenceNumber int64
+
+	// The followings are available only after MySQL-8.0


How about adding more details about these fields? (OriginalCommitTimestamp and OriginalCommitTimestamp were introduced in 8.0.1, TransactionLength was introduced in 8.0.2, ImmediateServerVersion and OriginalServerVersion were introduced in 8.0.14).

csuzhangxc · 2020-02-24T13:15:49Z

replication/row_event.go

+				continue
+			}
+
+			if collation, ok := collations[p]; ok {


Can we use collations[i] here?

hmm... i and p are two different counters in this loop: i is the index of all columns while p is the index of character columns, here is an example to illustrate:

column i p

a int 0 n/a

b char(8) 1 0

c bigint 2 n/a

d char(8) 3 1

👍 you're right, I found The column index is counted only in all character columns. now.

huangjunwen · 2020-02-25T00:44:23Z

cool, rest LGTM.
BTW, can you add some unit tests?

Sure, i'd like to add some unit tests for both mysql and mariadb once i have time.

siddontang · 2020-03-07T05:42:29Z

can we merge it now? @GregoryIan @csuzhangxc

csuzhangxc · 2020-03-07T06:18:40Z

I think we can merge it, and address comments later (in another PR) or let them go.

huangjunwen · 2020-03-07T07:09:30Z

Would you like to wait a while, i'm starting to write tests this morning ...

IANTHEREAL · 2020-03-07T09:19:45Z

👍 yep, we wait for your test case. This is very worthwhile, especially when encountering enum and set types, collation map will be wrong.

… on types to implement them correctly.

IANTHEREAL · 2020-03-10T06:14:03Z

@huangjunwen Cool, we will take a look ASAP

huangjunwen · 2020-03-10T07:40:25Z

I've removed types-related helper methods in TableMapEvent now. This is because these types-related optional meta data is stored in a compact format: for example, signedness bitmap only stores the signedness of numeric fields, default charset/column charset only stores collation id of character fields...

If we want to reconstruct the mapping of column index -> meta (signedness/collation id, ...), we need to know whether the field is considerred as a numeric field or a character field ... But there are many corner cases here, more investigation is needed to implement correctly. So i've decided not to implement them in this pr and open another issue to track this later.

csuzhangxc

LGTM

IANTHEREAL · 2020-03-11T00:20:51Z

LGTM Cool, let's make it perfect later

dhlin · 2020-03-19T20:43:23Z

Hi @GregoryIan @siddontang What do we expect on the commit standard in a PR to merge?
I saw there are some commits seem to be draft:

Move some code.
Remove unneccesary code.

and some merge commits

Merge branch 'master' of github.com:huangjunwen/go-mysql
Merge branch 'master' into master

These commits are essential but they can be organized and cleaned before the merge, do we bother to do so?

huangjunwen and others added 12 commits January 29, 2020 18:18

Add some optional meta fields for TABLE_MAP_EVENT: signedness/column

514d075

names/primary key

Add mysql 8.0 new fields to GTIDEvent

1e585a9

Add some timestamp helper methods for GTIDEvent

782e710

Move some code

3e9f1d7

Rename and move a function BytesToUint64 to mysql package

5dd4c32

Add some helper methods to TableMapEvent

c61b8a9

Make TableMapEvent's Dump prettier

c66d0a6

Remove unneccesary code

d3fef7b

Fix compile error before go 1.13: shift count type int, must be unsig…

a624aa6

…ned integer

Extract charset/collation info from TableMapEvent

27d6b7e

Minor fix: do not allocate collation map if not neccessary

fcf8f97

Merge branch 'master' into master

9ddfbf2

csuzhangxc reviewed Feb 24, 2020

View reviewed changes

huangjunwen added 8 commits March 8, 2020 15:40

Remove BytesToUint64 since FixedLengthInt already serve the same purpose

efddbd1

Add note for field type

52c09af

Fix CollationMap error when the table contains enum or set fields

41ecfa6

Refactor decodeOptionalMeta

3e4d0b3

Add support for other table map optional meta as well

6881af6

Remove helper functions in TableMapEvent now. Need more investigation…

2a96a8c

… on types to implement them correctly.

Add some test cases for TabeMapEvent optional meta decode

02e8369

Remove type.go

7626785

huangjunwen added 3 commits March 10, 2020 09:01

Merge branch 'master' of github.com:huangjunwen/go-mysql

569203c

Add some comments about new fields in GTIDEvent..

b758aa5

Add more test for GTIDEvent and TableMapEvent

d9d4b74

csuzhangxc reviewed Mar 10, 2020

View reviewed changes

IANTHEREAL merged commit 7a62847 into go-mysql-org:master Mar 11, 2020

IANTHEREAL mentioned this pull request Mar 11, 2020

Improve types-related helper methods of TableMapEvent #479

Open

shuhaowu mentioned this pull request May 19, 2020

Bump go-mysql dependency Shopify/ghostferry#187

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more MySQL-8.0 meta data to GTIDEvent and TableMapEvent #468

Add more MySQL-8.0 meta data to GTIDEvent and TableMapEvent #468

huangjunwen commented Jan 31, 2020 •

edited

Loading

IANTHEREAL commented Feb 22, 2020

IANTHEREAL commented Feb 22, 2020

siddontang commented Feb 23, 2020

huangjunwen commented Feb 23, 2020

csuzhangxc left a comment

csuzhangxc Feb 24, 2020

huangjunwen Feb 25, 2020

csuzhangxc Feb 24, 2020

huangjunwen Feb 25, 2020

csuzhangxc Feb 24, 2020

huangjunwen Feb 25, 2020

csuzhangxc Feb 25, 2020

huangjunwen commented Feb 25, 2020

siddontang commented Mar 7, 2020

csuzhangxc commented Mar 7, 2020

huangjunwen commented Mar 7, 2020 •

edited

Loading

IANTHEREAL commented Mar 7, 2020

IANTHEREAL commented Mar 10, 2020

huangjunwen commented Mar 10, 2020

csuzhangxc left a comment

IANTHEREAL commented Mar 11, 2020

dhlin commented Mar 19, 2020

Add more MySQL-8.0 meta data to GTIDEvent and TableMapEvent #468

Add more MySQL-8.0 meta data to GTIDEvent and TableMapEvent #468

Conversation

huangjunwen commented Jan 31, 2020 • edited Loading

IANTHEREAL commented Feb 22, 2020

IANTHEREAL commented Feb 22, 2020

siddontang commented Feb 23, 2020

huangjunwen commented Feb 23, 2020

csuzhangxc left a comment

Choose a reason for hiding this comment

csuzhangxc Feb 24, 2020

Choose a reason for hiding this comment

huangjunwen Feb 25, 2020

Choose a reason for hiding this comment

csuzhangxc Feb 24, 2020

Choose a reason for hiding this comment

huangjunwen Feb 25, 2020

Choose a reason for hiding this comment

csuzhangxc Feb 24, 2020

Choose a reason for hiding this comment

huangjunwen Feb 25, 2020

Choose a reason for hiding this comment

csuzhangxc Feb 25, 2020

Choose a reason for hiding this comment

huangjunwen commented Feb 25, 2020

siddontang commented Mar 7, 2020

csuzhangxc commented Mar 7, 2020

huangjunwen commented Mar 7, 2020 • edited Loading

IANTHEREAL commented Mar 7, 2020

IANTHEREAL commented Mar 10, 2020

huangjunwen commented Mar 10, 2020

csuzhangxc left a comment

Choose a reason for hiding this comment

IANTHEREAL commented Mar 11, 2020

dhlin commented Mar 19, 2020

huangjunwen commented Jan 31, 2020 •

edited

Loading

huangjunwen commented Mar 7, 2020 •

edited

Loading