Skip to content

Distinguish ambiguous column value of None #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 20 commits into from

Conversation

heehehe
Copy link
Member

@heehehe heehehe commented Sep 10, 2023

Resolve #39

Overview

In RowsEvent, there are three cases in which the column value is None.

  1. when the value is NULL
  2. when the datatype is related to time (datetime, time, ...) and the value is not supported format
    (ex: '0000-00-00 00:00:00')
  3. when the datatype is set and the value is empty

Since we cannot distinguish these cases, we're going to provide detail information for None value.

Changes

  • In RowsEvent, we added __none_sources variable.
  • When we execute __read_values_name,
    we check detail information for None cases and save it to __none_sources.
  • When dump() executes, we check for none_sources and show them together with column name and value.

Example

  • SQL query
    CREATE TABLE test_table (col0 int, col1 varchar(10), col2 datetime);
    INSERT INTO test_table VALUES (1, 'abc', '2023-09-09 00:00:00');
    UPDATE test_table SET col1=NULL, col2='0000-00-00 00:00:00' WHERE col0=1;
  • dump result for UPDATE
      === UpdateRowsEvent ===
      Date: 2023-09-09T05:23:14
      Log position: 1360
      Event size: 37
      Read bytes: 13
      Table: test.test_table
      Affected columns: 3
      Changed rows: 1
      Affected columns: 3
      Values:
      --
      *col0:1=>1
    + *col1:abc=>None(null)
    + *col2:2023-09-09 00:00:00=>None(out of datetime range)

@heehehe heehehe changed the title 중의적 column None 구분 Differentiate the ambiguous column value None Sep 10, 2023
@heehehe heehehe changed the title Differentiate the ambiguous column value None Distinguish the ambiguous column value None Sep 16, 2023
@heehehe heehehe changed the title Distinguish the ambiguous column value None Distinguish ambiguous column value of None Sep 16, 2023
@mjs1995 mjs1995 added the 3조 3조 만세 label Sep 16, 2023
Copy link
Member Author

@heehehe heehehe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@starcat37 @mikaniz
변경하면 좋을 부분 A / B 파트로 나눠서 코멘트 남겨뒀는데요, (코멘트 앞의 (A) / (B))
진아님 수인님 상의하셔서 각각 파트 정하시고 진행해주시면 좋을 것 같아요!
(A파트 맡으시는 분께서 먼저 진행하신 뒤 B파트 맡으시는 분께서 진행해주시면 될 것 같습니다🙂)

주로 변수명 변경 위주이긴 한데요,
구조를 파악하고 있어야 변경하기 쉬운 부분들도 있을 것 같아서
작업하시다가 어려운 부분 있으시면 말씀해주세요! :)

@heehehe heehehe marked this pull request as ready for review September 22, 2023 05:50
Since column schema has deleted, we cannot get any none sources when optional_meta_data is False.
@heehehe
Copy link
Member Author

heehehe commented Sep 22, 2023

@mikaniz @starcat37
수인님 진아님 작업해주시느라 고생 많으셨습니다!👏🏻🙇🏻‍♀️


추가로 최신 main 브랜치 pull 받아보니
column schema가 제거되면서 optional metadata를 사용하지 않을 때 문제가 발생하고 있었는데요,
_read_column_data 에서 column_name을 None으로 가져오면서 none_sources도 제대로 받아지지 않는 문제가 있었습니다..!

그래서 self.__optional_meta_data = True로 변경해서 정상 동작하는 것 확인했고,
none_sources는 column_name이 있을 때만 받아오도록 하고
test 파일에서는 none_sources가 비어있지 않을 때만 column별 none_sources 값들 체크하도록 변경했습니다

한번 같이 리뷰해주시면 반영해서 upstream PR에도 올려두겠습니다 :)


@heehehe heehehe requested a review from mikaniz September 22, 2023 23:30
@heehehe heehehe force-pushed the feature/categorize-none-column branch from 3011ef9 to 4c797a1 Compare September 23, 2023 15:58
heehehe and others added 5 commits September 29, 2023 23:07
When we do not know columnInformation
values dictionary key is always null
so overwrite value Object.
Thus always values dictionary has last order column object value

To avoid user confusion, it is marked and displayed as UNKNOWN_COL.
…-column-only

fix last order column only when DML
@heehehe heehehe closed this Oct 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3조 3조 만세
Projects
None yet
Development

Successfully merging this pull request may close these issues.

중의적으로 쓰이는 컬럼값 None🤔
5 participants