Add Intl.Segmenter support #539
Conversation
| // 2. "Mei (梅) has (有) many (很多) sons (儿子)" | ||
| // We want to see that diffWords will get the word counts right and won't try to treat the | ||
| // trailing 子 as common to both texts (since it's part of a different word each time). | ||
| // TODO: Check with a Chinese speaker that this example is correct Chinese. |
There was a problem hiding this comment.
Hello, @ExplodingCabbage, I'm Chinese, I can confirm that the meaning of these two sentences are correct.
But I'm not sure about the test purpose here.
I can see that
> [...chineseSegmenter.segment('我有很多桌子。')].map(({segment})=>segment)
[ '我有', '很多', '桌子', '。' ]
> [...chineseSegmenter.segment('梅有很多儿子。')].map(({segment})=>segment)
[ '梅', '有', '很多', '儿子', '。' ]
Not sure why '我有' are together, but '梅', '有' are separated, if you want test something similar you can change 梅有 to 他有(He has) or 她有(She has).
There was a problem hiding this comment.
It's quite strange...
> [...chineseSegmenter.segment('她有很多桌子。')].map(({segment})=>segment)
[ '她', '有', '很多', '桌子', '。' ]
> [...chineseSegmenter.segment('他有很多桌子。')].map(({segment})=>segment)
[ '他有', '很多', '桌子', '。' ]
'她' and '他' are the same, but one for male, one for female.
There was a problem hiding this comment.
But I'm not sure about the test purpose here.
Huh. What I wanted to demonstrate was that the "tokens" we split the text into are what a Chinese speaker or linguist would consider to be single words. But from what you write, it sounds like the tokens in this case aren't what you'd consider words, though, and Intl.Segmenter is giving an incorrect result? i.e. if I understand you right, you would say "我有" is two words ("我" and "有"), not a single word made up of two characters? Is there any kind of ambiguity about this - like, is there any reason a native speaker might argue that 我有 is a single word - or is Intl.Segmenter just completely unambiguously wrong here?
If so, it's probably worth a bug report to... whatever the underlying source of the segmentation rules is here. (ICU, I think.)
Anyway, for clarity I'll tweak this test to show an example where Intl.Segmenter manages to get the tokenization right. Thank you for commenting!
There was a problem hiding this comment.
not a single word made up of two characters?
It's not something really "incorrect", people have different opinions. Just feels odd, it's inconsistent for things similar.
# I have money
> [...chineseSegmenter.segment('我有钱')].map(({segment})=>segment)
[ '我有', '钱' ]
# You have money
> [...chineseSegmenter.segment('你有钱')].map(({segment})=>segment)
[ '你有', '钱' ]
# He have money
> [...chineseSegmenter.segment('他有钱')].map(({segment})=>segment)
[ '他有', '钱' ]
# She have money
> [...chineseSegmenter.segment('她有钱')].map(({segment})=>segment)
[ '她', '有', '钱' ]
# It have money
> [...chineseSegmenter.segment('它有钱')].map(({segment})=>segment)
[ '它', '有', '钱' ]
# I love flowers
> [...chineseSegmenter.segment('我爱花')].map(({segment})=>segment)
[ '我', '爱', '花' ]
# You love flowers
> [...chineseSegmenter.segment('你爱花')].map(({segment})=>segment)
[ '你爱', '花' ]
# He love flowers
> [...chineseSegmenter.segment('他爱花')].map(({segment})=>segment)
[ '他', '爱', '花' ]
# She love flowers
> [...chineseSegmenter.segment('她爱花')].map(({segment})=>segment)
[ '她', '爱', '花' ]
# It love flowers
> [...chineseSegmenter.segment('它爱花')].map(({segment})=>segment)
[ '它', '爱', '花' ]
…enter segments in a way that seems more correct (or at least more self-consistent) to a native Chinese speaker See discussion at #539 (comment) for explanation
…enter segments in a way that seems more correct (or at least more self-consistent) to a native Chinese speaker (#613) See discussion at #539 (comment) for explanation
Resolves #438; probably also adequately resolves #214 even though it's not quite the solution that was asked for in that issue.