Any ideas on changing API interface for a major semver update?

[lance6716]

This package is planned do a major semver update and we've got a chance to change public functions, interfaces, etc. To make full use of this update we developers can list our expected changes in this issue, maintainers will check all changes before finally do a update. Hope the package will be better after discussing and accepting contributions with our ideas.

I simply propose some rules on how to use this issue and welcome to comment on both rules and ideas:

• the deadline of posting expected changes is 3 weeks later, which is Dec-17-2022
• everyone can comment in this issue or open another issue and link this one, I will edit the description of this issue to make a summary
• no other one will make sure they would implement your idea! You should do it yourself if you really want it

---

• canal EventHandler expose more information on callback EventHandlerV2 support handle event with replication.EventHeader #740
• Add download function to support RDS archived binlog canal云平台RDS解决binlog被删除的问题 #745

[lance6716]

I randomly pick up some developers from recent PR, ptal @atercattus @BLAZZ @GMHDBJD @mcuelenaere @tech-sumit @skoef @shizhx @btoonk @nisainan @bingoohuang

You can turn off notification by click "Unsubscribe" button, sorry to bother you :P

[mcuelenaere]

One piece of feedback I have, is that go-mysql-org/go-mysql is allocating lots of memory (which gets freed very soon afterwards), when reading binlogs. In our production app, over 62% of all allocations (average of 1000 profiles taken in the last week) and 73% of all allocated memory are coming from BinlogParser.Parse.

Maybe having some kind of reusable pool of BinlogEvent/RowsEvent objects could help here, but that would require the interface to change as the consumer has to explicitly release these events after being done using them.

Also, more specifically for our usecase, it seems that quite a bit of those allocations are spent on JSON decoding, while actually we are not doing anything with those JSON objects. Maybe having an option to leave the raw JSON []byte as-is could help here.

[lance6716]

@mcuelenaere

we encountered a similar problem and came up with a solution, I guess we can also solve your unneeded JSON parsing issue in this way
Our problem is we are only interested in a part of tables in binlog, don't want to waste time parsing uninterested tables, so we add a function injection configuration to let user implement their customized parsing function, like skipping parsing tables based on table name.

#737

so maybe you can use a customized function to only skip JSON columns?