Skip to content

Fix tb upd #201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 196 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
196 commits
Select commit Hold shift + click to select a range
b012a03
Merge pull request #25 from sunsingerus/master
sunsingerus Oct 30, 2017
e387f42
Merge pull request #26 from sunsingerus/master
sunsingerus Oct 31, 2017
3078311
Merge pull request #27 from sunsingerus/master
sunsingerus Nov 1, 2017
6dd92d0
Merge pull request #28 from sunsingerus/master
sunsingerus Nov 1, 2017
43f1c4a
Merge pull request #29 from sunsingerus/master
sunsingerus Nov 8, 2017
d2eed56
Merge pull request #30 from sunsingerus/master
sunsingerus Nov 14, 2017
8d7b5e0
Merge pull request #31 from sunsingerus/master
sunsingerus Nov 15, 2017
1035103
Merge pull request #32 from sunsingerus/master
sunsingerus Nov 15, 2017
78c84fa
Merge pull request #33 from sunsingerus/master
sunsingerus Nov 15, 2017
4df25f6
Merge pull request #34 from sunsingerus/master
sunsingerus Nov 20, 2017
38d3081
Merge pull request #35 from sunsingerus/master
sunsingerus Nov 20, 2017
dcb7564
Merge pull request #36 from sunsingerus/master
sunsingerus Nov 21, 2017
9845987
Merge pull request #37 from sunsingerus/master
sunsingerus Nov 21, 2017
94b5b9b
Merge pull request #38 from sunsingerus/master
sunsingerus Nov 22, 2017
478ddcd
Merge pull request #39 from sunsingerus/master
sunsingerus Nov 23, 2017
4a53fca
Merge pull request #40 from sunsingerus/master
sunsingerus Nov 27, 2017
03532b7
Merge pull request #41 from sunsingerus/master
sunsingerus Nov 27, 2017
db6da75
Merge pull request #42 from sunsingerus/master
sunsingerus Nov 27, 2017
e71b2d4
Merge pull request #43 from sunsingerus/master
sunsingerus Nov 28, 2017
c06c823
Merge pull request #44 from sunsingerus/master
sunsingerus Nov 29, 2017
23f8732
Merge pull request #45 from sunsingerus/master
sunsingerus Nov 30, 2017
ffa6c78
Merge pull request #46 from sunsingerus/master
sunsingerus Nov 30, 2017
928d67f
Merge pull request #47 from sunsingerus/master
sunsingerus Nov 30, 2017
866218a
Merge pull request #48 from sunsingerus/master
sunsingerus Nov 30, 2017
429ef32
Merge pull request #49 from sunsingerus/master
sunsingerus Dec 4, 2017
432287f
Merge pull request #50 from sunsingerus/master
sunsingerus Dec 5, 2017
56ddf7d
Merge pull request #51 from sunsingerus/master
sunsingerus Dec 6, 2017
0584a32
Merge pull request #52 from sunsingerus/master
sunsingerus Dec 11, 2017
1118bd2
Merge pull request #53 from sunsingerus/master
sunsingerus Dec 12, 2017
6a19f35
Merge pull request #54 from sunsingerus/master
sunsingerus Dec 13, 2017
ce1b36f
Merge pull request #55 from sunsingerus/master
sunsingerus Dec 13, 2017
0af6951
Merge pull request #56 from sunsingerus/master
sunsingerus Dec 21, 2017
237304d
Merge pull request #58 from sunsingerus/master
sunsingerus Dec 26, 2017
e7a5233
Merge pull request #59 from sunsingerus/master
sunsingerus Dec 26, 2017
6399906
Merge pull request #60 from sunsingerus/master
sunsingerus Dec 26, 2017
f4d98ff
Merge pull request #61 from sunsingerus/master
sunsingerus Dec 26, 2017
0e6f842
Merge pull request #62 from sunsingerus/master
sunsingerus Dec 26, 2017
a4baa67
Merge pull request #63 from sunsingerus/master
sunsingerus Dec 27, 2017
6ed38ce
Merge pull request #64 from sunsingerus/master
sunsingerus Dec 27, 2017
b517adb
Merge pull request #65 from sunsingerus/master
sunsingerus Dec 27, 2017
31e8d41
Merge pull request #66 from sunsingerus/master
sunsingerus Jan 1, 2018
613b4ba
Merge pull request #67 from sunsingerus/master
sunsingerus Jan 1, 2018
8e36f44
Merge pull request #68 from sunsingerus/master
sunsingerus Jan 1, 2018
2888d75
Merge pull request #69 from sunsingerus/master
sunsingerus Jan 1, 2018
8d175de
Merge pull request #70 from sunsingerus/master
sunsingerus Jan 1, 2018
0b1c7eb
Merge pull request #71 from sunsingerus/master
sunsingerus Jan 1, 2018
065582e
Merge pull request #72 from sunsingerus/master
sunsingerus Jan 1, 2018
6dde759
Merge pull request #73 from sunsingerus/master
sunsingerus Jan 1, 2018
06088f4
Merge pull request #74 from sunsingerus/master
sunsingerus Jan 1, 2018
bd5504a
Merge pull request #75 from sunsingerus/master
sunsingerus Jan 1, 2018
43ca43b
Merge pull request #76 from sunsingerus/master
sunsingerus Jan 1, 2018
ba0c4e4
Merge pull request #77 from sunsingerus/master
sunsingerus Jan 1, 2018
fcc814a
Merge pull request #78 from sunsingerus/master
sunsingerus Jan 2, 2018
97b3c29
Merge pull request #79 from sunsingerus/master
sunsingerus Jan 2, 2018
bddb1b2
Merge pull request #80 from sunsingerus/master
sunsingerus Jan 4, 2018
aeff523
Merge pull request #81 from sunsingerus/master
sunsingerus Jan 4, 2018
a68c489
Merge pull request #82 from sunsingerus/master
sunsingerus Jan 4, 2018
ebf1009
Merge pull request #83 from sunsingerus/master
sunsingerus Jan 4, 2018
3aa751d
Merge pull request #85 from sunsingerus/master
sunsingerus Jan 10, 2018
a650dba
Merge pull request #86 from sunsingerus/master
sunsingerus Jan 10, 2018
5c68399
Merge pull request #87 from sunsingerus/master
sunsingerus Jan 11, 2018
ccda3d2
Merge pull request #88 from sunsingerus/master
sunsingerus Jan 14, 2018
0d441ae
Merge pull request #89 from sunsingerus/master
sunsingerus Jan 15, 2018
0f8ecae
Merge pull request #90 from sunsingerus/master
sunsingerus Jan 15, 2018
c7ff975
Merge pull request #91 from sunsingerus/master
sunsingerus Jan 15, 2018
ab69f9c
Merge pull request #92 from sunsingerus/master
sunsingerus Jan 15, 2018
d28ca55
Merge pull request #93 from sunsingerus/master
sunsingerus Jan 16, 2018
a437fdc
Merge pull request #94 from sunsingerus/master
sunsingerus Jan 17, 2018
51189f7
Merge pull request #95 from sunsingerus/master
sunsingerus Jan 17, 2018
557d9aa
Merge pull request #96 from sunsingerus/master
sunsingerus Jan 17, 2018
4a19dc9
Merge pull request #98 from sunsingerus/master
sunsingerus Jan 24, 2018
d61609a
Merge pull request #99 from sunsingerus/master
sunsingerus Jan 24, 2018
33f78d7
Merge pull request #100 from sunsingerus/master
sunsingerus Jan 26, 2018
a743f75
Merge pull request #101 from sunsingerus/master
sunsingerus Jan 26, 2018
b3d6ce0
Merge pull request #102 from sunsingerus/master
sunsingerus Jan 26, 2018
090ca5c
Merge pull request #103 from sunsingerus/master
sunsingerus Jan 26, 2018
7c8f1b0
Merge pull request #104 from sunsingerus/master
sunsingerus Feb 16, 2018
3303718
Merge pull request #105 from sunsingerus/master
sunsingerus Feb 16, 2018
1c671f0
Merge pull request #106 from sunsingerus/master
sunsingerus Feb 16, 2018
0b9a62a
Merge pull request #107 from sunsingerus/master
sunsingerus Feb 19, 2018
34610c2
Merge pull request #108 from sunsingerus/master
sunsingerus Feb 20, 2018
5d2bfd2
Merge pull request #109 from sunsingerus/master
sunsingerus Feb 20, 2018
91721d1
Merge pull request #110 from sunsingerus/master
sunsingerus Feb 21, 2018
3f5fa52
Merge pull request #111 from sunsingerus/master
sunsingerus Feb 21, 2018
163df52
Merge pull request #112 from sunsingerus/master
sunsingerus Feb 21, 2018
4c36c37
Merge pull request #113 from sunsingerus/master
sunsingerus Feb 21, 2018
ab64229
Merge pull request #114 from sunsingerus/master
sunsingerus Feb 21, 2018
c602c69
Merge pull request #115 from sunsingerus/master
sunsingerus Feb 27, 2018
8b37982
Merge pull request #116 from sunsingerus/master
sunsingerus Feb 27, 2018
95c9746
Merge pull request #117 from sunsingerus/master
sunsingerus Mar 1, 2018
cb8be8b
Merge pull request #118 from sunsingerus/master
sunsingerus Mar 5, 2018
bfdcc37
Merge pull request #119 from sunsingerus/master
sunsingerus Mar 5, 2018
6ef4483
Merge pull request #120 from sunsingerus/master
sunsingerus Mar 7, 2018
db6c3b2
Merge pull request #121 from sunsingerus/master
sunsingerus Mar 7, 2018
488d236
Merge pull request #122 from sunsingerus/master
sunsingerus Mar 7, 2018
f9237dc
Merge pull request #123 from sunsingerus/master
sunsingerus Mar 8, 2018
008a47f
Merge pull request #124 from sunsingerus/master
sunsingerus Mar 13, 2018
f88555e
Merge pull request #125 from sunsingerus/master
sunsingerus Mar 14, 2018
2857384
Merge pull request #126 from sunsingerus/master
sunsingerus Mar 14, 2018
c234d51
Merge pull request #127 from sunsingerus/master
sunsingerus Mar 16, 2018
65d4aae
Merge pull request #128 from sunsingerus/master
sunsingerus Mar 16, 2018
cfc1381
Merge pull request #129 from sunsingerus/master
sunsingerus Mar 19, 2018
d01d9ae
Merge pull request #130 from sunsingerus/master
sunsingerus Mar 20, 2018
3ea8227
Merge pull request #131 from sunsingerus/master
sunsingerus Mar 21, 2018
51b9583
Merge pull request #132 from sunsingerus/master
sunsingerus Mar 21, 2018
0fc9561
Merge pull request #133 from sunsingerus/master
sunsingerus Mar 21, 2018
1d7edad
Merge pull request #134 from sunsingerus/master
sunsingerus Mar 22, 2018
602444c
Merge pull request #135 from sunsingerus/master
sunsingerus Mar 23, 2018
e12dda0
fix a bug in csvwriter if number of events if equal to mempool-max-ev…
Dec 20, 2018
d36e3e0
Merge pull request #142 from MaxFedotov/master
sunsingerus Dec 20, 2018
3422bb4
repair create sql error
aquarius-wing Mar 22, 2019
d00a0db
传入dst-schema的时候生成的table名改变
aquarius-wing Mar 23, 2019
bd15f43
cluster测试输出语句
aquarius-wing Mar 23, 2019
99c5966
增加cluster参数成功
aquarius-wing Mar 23, 2019
d57a803
支持在创建sql、创建表时添加到指定的dst-cluster
aquarius-wing Mar 25, 2019
46bceae
传入dst-distribute参数时,会把schema后加_all,table后加_all
aquarius-wing Mar 26, 2019
8a97a1e
update CHANGELOG.md
aquarius-wing Mar 26, 2019
a2067ce
传dst-distribute时不新增on cluster语句,传dst-distribute支持批量写入表数据,同时修复Decimal不…
aquarius-wing Apr 3, 2019
f321c3d
传column_skip时支持创建表、导入数据时,跳过指定列
aquarius-wing Apr 12, 2019
f69f038
删除不必要的logging
aquarius-wing Apr 12, 2019
abe969d
column_skip默认值修改为空数组
aquarius-wing Apr 23, 2019
fb16a2b
删除不必要的logging
aquarius-wing Apr 30, 2019
ec3aa80
bug-实时导入表时会出现的错误
aquarius-wing May 15, 2019
da6e608
Merge pull request #148 from Wing-Lo/master
sunsingerus May 18, 2019
0f4b91a
Fix error File not found
bookin Jul 17, 2019
da31d2b
Merge pull request #156 from bookin/patch-1
sunsingerus Jul 18, 2019
0d4f66a
dev: review UTF8
sunsingerus Aug 21, 2019
c560ba3
Merge pull request #158 from sunsingerus/master
sunsingerus Aug 21, 2019
4be266d
env: enhance dev scripts
sunsingerus Aug 26, 2019
aa08f2b
Merge pull request #159 from sunsingerus/master
sunsingerus Aug 26, 2019
5c5148a
env: enhance RPM and deb packager
sunsingerus Aug 26, 2019
79df309
Merge pull request #160 from sunsingerus/master
sunsingerus Aug 26, 2019
80bd93c
Port (--src-port) cli/config setting was being ignored
Sep 1, 2019
35d7802
Merge pull request #161 from flyrdm/src-port-config-fix
sunsingerus Sep 1, 2019
6867044
bump version
sunsingerus Sep 2, 2019
00f35f7
add sysadmins to audience
sunsingerus Sep 2, 2019
fbc5b56
bump version
sunsingerus Sep 2, 2019
e54734f
write --src-tables-where-clauses explanation
sunsingerus Sep 2, 2019
1743942
fix tables migration
sunsingerus Oct 9, 2019
fa53ed2
Merge pull request #163 from sunsingerus/master
sunsingerus Oct 9, 2019
1a0f7cf
Escape column names used in MySQL select statements
sorin-omni Nov 4, 2019
7dad57d
Merge pull request #164 from sfdumi/master
sunsingerus Nov 4, 2019
6100f20
set exit code 1 for run too
Jan 22, 2020
b98b6e7
Merge pull request #167 from mihaichitic/master
sunsingerus Jan 23, 2020
a194979
fix wheres -> where_clauses
rootzltd Jan 27, 2020
a25bb39
Merge pull request #169 from kimmom/master
sunsingerus Jan 29, 2020
811c59e
dev: add shell quotation
sunsingerus Jan 29, 2020
6eb6db9
Merge pull request #170 from sunsingerus/master
sunsingerus Jan 29, 2020
6ffc91a
new release
sunsingerus Jan 29, 2020
5557f73
Merge pull request #171 from sunsingerus/master
sunsingerus Jan 29, 2020
1c6b52b
docs: fix src-tables
sunsingerus Jan 29, 2020
a1ab743
Merge pull request #172 from sunsingerus/master
sunsingerus Jan 29, 2020
d14e556
docs: add references
sunsingerus Feb 24, 2020
b4ec875
docs: move manual to docs and add references section
sunsingerus Feb 24, 2020
2614ad3
Merge pull request #173 from sunsingerus/master
sunsingerus Feb 24, 2020
e05ce7e
docs: fix links
sunsingerus Feb 24, 2020
2f1244c
Merge pull request #174 from sunsingerus/master
sunsingerus Feb 24, 2020
26a60e6
docs: remove RPM sections
sunsingerus Apr 16, 2020
2c6d1d5
Merge pull request #177 from sunsingerus/master
sunsingerus Apr 16, 2020
020ffbc
Initial support for updates
ygnuss Mar 9, 2021
5212148
Fixed some errors with null values (None) for update events
ygnuss Mar 10, 2021
cc0a593
Added requirements file
ygnuss Mar 10, 2021
e68ff22
Fixed handling delete events
ygnuss Mar 10, 2021
09a2f92
Added audit column for tracking changes in Clickhouse
ygnuss Mar 10, 2021
826b568
Added scripts for running listeners
ygnuss Mar 16, 2021
6683cd9
Added dumper script and moved all init scripts to init folder
ygnuss Mar 16, 2021
a2f19a4
Updated gitignore to ignore out log files and binlog checkpoint files
ygnuss Mar 16, 2021
d7bdeee
Fix to handle multivaluated keys in asset table. This needs to be fix…
ygnuss Mar 16, 2021
22cca2a
Added first-processing script to initialize database without loosing …
ygnuss Mar 16, 2021
9d67f60
fix: Updated run script to be more flexible and support running just …
ygnuss Apr 12, 2021
f58e6b8
Changed update to include in alter table command just those columns w…
ygnuss May 20, 2021
a8e3c92
Merge pull request #1 from tinybirdco/feature/add-update-delete-support
ygnuss Jun 7, 2021
081b632
Added support to insert/update and CSV
alexon1234 Aug 10, 2021
4218b4a
Fix missing config
alexon1234 Aug 10, 2021
6050892
Added shutdown gracefully
alexon1234 Sep 7, 2021
8070102
Moved signal handler to main
alexon1234 Sep 7, 2021
5ac5a1b
Fix issue with MySQL graceful exit
alexon1234 Sep 8, 2021
1912301
Add retry in case of 429
alexon1234 Sep 8, 2021
a2f3884
Graceful exit only stops reader
alexon1234 Sep 8, 2021
19e3d2c
cast retry-after to int
alexon1234 Sep 8, 2021
da57a75
Improve retry in TB CSV
alexon1234 Sep 8, 2021
abc31b1
If received SIGINT break the loop
alexon1234 Sep 8, 2021
f88c5a0
Improve logging when reading binlog pos
alexon1234 Sep 8, 2021
3c18a0d
Validate that binlog file exist before reading it
alexon1234 Sep 8, 2021
5ce22ae
remove local and test script
alexon1234 Sep 13, 2021
c92f49c
Added try/catch in upload to retry in case of error
alexon1234 Sep 14, 2021
80d8816
Changed CSV writer to use QUOTE_ALL
alexon1234 Sep 23, 2021
79323ed
Undo last change and add QUOTE_ALL to DictWriter
alexon1234 Sep 23, 2021
17123e4
Avoiding flushing with every round
alexon1234 Sep 25, 2021
5dc884d
Disabled Verify to avoid SSL checking
alexon1234 Oct 6, 2021
9fbc586
Merge pull request #2 from tinybirdco/feature/update-delete-with-inserts
alexon1234 Jan 16, 2023
89ad6a6
Generate CSV with QUOTE_MINIMAL
alexon1234 Jan 16, 2023
512f82a
Merge pull request #3 from tinybirdco/use_quote_minimal
alexon1234 Jan 16, 2023
99d627d
Use QUOTE_MINIMAL everywhere
alexon1234 Jan 16, 2023
10040a2
Merge pull request #5 from tinybirdco/use_quote_minimal
alexon1234 Jan 16, 2023
afeee29
change format for tb_upd column
Jan 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[flake8]
ignore =
; except
E722,
; inline regex
W605,
; long lines
E501,
; too complex
C901
max-complexity = 10
max-line-length = 120
application-import-names = flake8
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@
*.egg
*.egg-info
dist
sdist
deb_dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg

Expand Down Expand Up @@ -42,3 +43,8 @@ _build

# Pyenv
.python-version

# Tinibird
bl-*
out-*
.e
18 changes: 17 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
# clickhouse-mysql 2019-09-03

## improvements
* fix --src-tables-where-clauses to accept both filenames (for long where-clauses) and where-cluases themselves (for shorted clauses)

## bugfixes
* fix --src-port CLI option
* ensure UTF8 for source migration

# clickhouse-mysql 2019-03-25

## new features
* added new CLI option `--dst-schema` - make table full name change to `schema`.`db__table`
* added new CLI option `--dst-cluster` - support table create on cluster
* added new CLI option `--dst-distribute` - make table full name change to `schema_all`.`db__table_all`,and engine change to Distributed

# clickhouse-mysql 2018-03-14

## new features
Expand All @@ -14,4 +30,4 @@

## bugfixes
* config files vs CLI options order fixed


Loading