Skip to content

Commit 9b7feb6

Browse files
authored
Merge pull request #37 from sunsingerus/master
docs
2 parents ec769a3 + ca11390 commit 9b7feb6

File tree

1 file changed

+31
-7
lines changed

1 file changed

+31
-7
lines changed

README.md

Lines changed: 31 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ pip install clickhouse-driver
4545

4646
Also the following (at least one of) MySQL privileges are required for this operation: `SUPER`, `REPLICATION CLIENT`
4747

48-
```sql
48+
```mysql
4949
CREATE USER 'reader'@'localhost' IDENTIFIED BY 'qwerty';
5050
CREATE USER 'reader'@'127.0.0.1' IDENTIFIED BY 'qwerty';
5151
CREATE USER 'reader'@'*' IDENTIFIED BY 'qwerty';
@@ -65,7 +65,7 @@ Also the following MySQL config options are required:
6565
```ini
6666
[mysqld]
6767
server-id = 1
68-
log_bin = /var/log/mysql/mysql-bin.log
68+
log_bin = /var/lib/mysql/bin.log
6969
expire_logs_days = 10
7070
max_binlog_size = 100M
7171
binlog-format = row #Very important if you want to receive write, update and delete row events
@@ -293,7 +293,7 @@ We have to separate test table into several ones because of this error, produced
293293
ERROR 1118 (42000): Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
294294
```
295295

296-
```sql
296+
```mysql
297297
CREATE TABLE datatypes(
298298

299299
bit_1 BIT(1),
@@ -397,12 +397,14 @@ CREATE TABLE long_varbinary_datatypes(
397397
```
398398

399399

400-
```sql
400+
```mysql
401401
-- in order to be able to set timestamp = '1970-01-01 00:00:01'
402402
set time_zone='+00:00';
403403
```
404404

405-
```sql
405+
Insert minimal acceptable values into the test table:
406+
407+
```mysql
406408
-- MIN values
407409
INSERT INTO datatypes SET
408410

@@ -500,7 +502,9 @@ INSERT INTO long_varbinary_datatypes SET
500502
;
501503
```
502504

503-
```sql
505+
Insert maximum acceptable values into the test table:
506+
507+
```mysql
504508
-- MAX values
505509
INSERT INTO datatypes SET
506510

@@ -722,12 +726,26 @@ Main Steps
722726

723727
#### airline.ontime Data Set in CSV files
724728
Run [download script](run_airline_ontime_data_download.sh)
729+
725730
You may want to adjust dirs where to keep `ZIP` and `CSV` file
731+
726732
In `run_airline_ontime_data_download.sh` edit these lines:
727733
```bash
734+
...
728735
ZIP_FILES_DIR="zip"
729736
CSV_FILES_DIR="csv"
737+
...
730738
```
739+
You may want to adjust number of files to download (In case downloading all it may take some time).
740+
741+
Specify year and months range as you wish:
742+
```bash
743+
...
744+
echo "Download files into $ZIP_FILES_DIR"
745+
for year in `seq 1987 2017`; do
746+
for month in `seq 1 12`; do
747+
...
748+
```
731749
732750
```bash
733751
./run_airline_ontime_data_download.sh
@@ -737,7 +755,7 @@ Downloading can take some time.
737755
#### airline.ontime MySQL Table
738756
Create MySQL table of the following structure:
739757
740-
```sql
758+
```mysql
741759
CREATE DATABASE IF NOT EXISTS `airline`;
742760
CREATE TABLE IF NOT EXISTS `airline`.`ontime` (
743761
`Year` SMALLINT UNSIGNED, -- maps to UInt16,
@@ -971,10 +989,13 @@ CREATE TABLE IF NOT EXISTS `airline`.`ontime` (
971989
972990
#### airline.ontime Data Reader
973991
Run [datareader script](run_airline_ontime_data_reader.sh)
992+
974993
You may want to adjust `PYTHON` path and source and target hosts and usernames
975994
```bash
995+
...
976996
PYTHON=python3.6
977997
PYTHON=/home/user/pypy3.5-5.9-beta-linux_x86_64-portable/bin/pypy
998+
...
978999
```
9791000
```bash
9801001
...
@@ -989,13 +1010,16 @@ PYTHON=/home/user/pypy3.5-5.9-beta-linux_x86_64-portable/bin/pypy
9891010
9901011
#### airline.ontime Data Importer
9911012
Run [data importer script](run_airline_ontime_import.sh)
1013+
9921014
You may want to adjust `CSV` files location, number of imported files and MySQL user/password used for import
9931015
```bash
1016+
...
9941017
# looking for csv files in this dir
9951018
FILES_TO_IMPORT_DIR="/mnt/nas/work/ontime"
9961019

9971020
# limit import to this number of files
9981021
FILES_TO_IMPORT_NUM=3
1022+
...
9991023
```
10001024
```bash
10011025
...

0 commit comments

Comments
 (0)