Skip to content

Feature/add update delete support #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jun 7, 2021

Conversation

ygnuss
Copy link

@ygnuss ygnuss commented Jun 7, 2021

We listen to the following binlog events:

  1. Inserts
  2. Updates
  3. Deletes

For updates and deletes we use ALTER TABLE command. We cannot use mempool to optimize.

The main issue seems to be CH is not able to handle all the update events. CH after applying mutations, before marking them as finished, checks if there are other pending mutations. In our case, mutations are constantly arriving so CH always find new pending mutations so never marks the applied mutations as finished. The number of mutations increases flooding of files the HDD and, if using replication, collapsing the zookeeper because it has too many entries on the mutations znode.

We use this approach for the BeBanjo PoC and it works, during a time, because didn't use replication. After a couple of weeks, we start having issues with the mutation files.

We have discarded this approach in favor of using inserts for updates and deletes and handling mutations the same way ITX does.

@ygnuss ygnuss merged commit 10e9c15 into master Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant