Fix ECMA 262 regex whitespace tests.

[ChALkeR]

Looks like there was a bug at the time of introduction due to #285, which was introduced in #282 and finalized in #286.

1. At the time of introduction (Tests for ECMA 262 regex dialect #282), those tests followed a false premise but passed because of a mistype.
2. When the mistype was noticed (Wrong tests in ecmascript-regex.json  #285), \\ were replaced with \ (Fix data - escape unicode, not regex pattern #286).
3. But now the test is invalid as is failing in all implementations.

---

Per ECMA 262 specification:
https://www.ecma-international.org/ecma-262/10.0/index.html#sec-characterclassescape

\S refers to \s:

Section 21.2.2.12:

The production CharacterClassEscape::S evaluates as follows:

1. Return the set of all characters not included in the set returned by CharacterClassEscape::s.

\s refers to WhiteSpace:

The production CharacterClassEscape::s evaluates as follows:

1. Return the set of characters containing the characters that are on the right-hand side of the WhiteSpace or LineTerminator productions.

WhiteSpace includes NBSP (U00A0):
https://www.ecma-international.org/ecma-262/10.0/index.html#prod-WhiteSpace

That is the exact opposite of what the current test was checking for.

Also, WhiteSpace includes any other Unicode "Space_Separator" code points.

That's not a new addition, same was true in version 6.0, for example:

• https://www.ecma-international.org/ecma-262/6.0/index.html#sec-characterclassescape
• https://www.ecma-international.org/ecma-262/6.0/index.html#sec-white-space

[Julian]

Hi -- will review this carefully, thanks for the PR, it's definitely appreciated, although saying something is "blindly corrected" is unnecessary snark.

[ChALkeR]

@Julian Hi! Sorry, I didn't mean it to have that tone, I just tried to describe the probable cause of the bug. I amended the text, hope it's better now. Thanks!

[karenetheridge]

side note: a lot of these tests don't even use the "format" keyword, so they should be moved into a separate file in optional/, not optional/format. (Looks like it was me, in commit 8388f27, oops! I will fix tomorrow.)

[ChALkeR]

Two reasons why I think that the existing test is a mistake are:

1. The test name, ECMA 262 \s matches ascii whitespace only, directly contradicts ECMA 262 specification, cited above
2. The way how this was introduced with a mistype at first

[karenetheridge]

@ChALkeR some of the test files you modified were just moved in master; sorry! could you possibly rebase?

[ChALkeR]

@karenetheridge Rebased!

[karenetheridge]

While you're in these files, could you please fix this errors also? (line 157 in your version of the file) -- \\w should be \\W:

    "description": "ECMA 262 \\w matches everything but ascii letters",

[karenetheridge]

I find it very troubling that in ECMA 262, \w and \d have ascii semantics, while \s is expected to have unicode semantics -- in my implementation I get to choose between one or the other. On balance I think unicode semantics will be more useful, and I can document that if only ascii digits or word characters are desired, then the pattern should instead use [0-9] and [A-Za-z0-9_].

[karenetheridge]

Are the tests for characters \u2029 and \u2003 correct? That doesn't seem to match the table at https://www.ecma-international.org/ecma-262/10.0/index.html#sec-white-space.

[ChALkeR]

While you're in these files, could you please fix this errors also? (line 157 in your version of the file) -- \\w should be \\W:

Done.

---

Are the tests for characters \u2029 and \u2003 correct?
That doesn't seem to match the table at https://www.ecma-international.org/ecma-262/10.0/index.html#sec-white-space.

Yes, they are correct, their descriptions mention the reason why they are treated as whitespace:

            {
                "description": "paragraph separator matches (line terminator)",
                "data": "\u2029",
                "valid": true
            },
            {
                "description": "EM SPACE matches (Space_Separator)",
                "data": "\u2003",
                "valid": true
            },

https://www.ecma-international.org/ecma-262/10.0/index.html#sec-white-space (table 32) says, in the last row:

Other category “Zs” Any other Unicode “Space_Separator” code point <USP>

\u2003 is in the Space_Separator category.
To confirm that, we can use this list of space separators (Zs):

> curl http://www.unicode.org/Public/UNIDATA/UnicodeData.txt -s | grep ';Zs;'
0020;SPACE;Zs;0;WS;;;;;N;;;;;
00A0;NO-BREAK SPACE;Zs;0;CS;<noBreak> 0020;;;;N;NON-BREAKING SPACE;;;;
1680;OGHAM SPACE MARK;Zs;0;WS;;;;;N;;;;;
2000;EN QUAD;Zs;0;WS;2002;;;;N;;;;;
2001;EM QUAD;Zs;0;WS;2003;;;;N;;;;;
2002;EN SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
2003;EM SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
2004;THREE-PER-EM SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
2005;FOUR-PER-EM SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
2006;SIX-PER-EM SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
2007;FIGURE SPACE;Zs;0;WS;<noBreak> 0020;;;;N;;;;;
2008;PUNCTUATION SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
2009;THIN SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
200A;HAIR SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
202F;NARROW NO-BREAK SPACE;Zs;0;CS;<noBreak> 0020;;;;N;;;;;
205F;MEDIUM MATHEMATICAL SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
3000;IDEOGRAPHIC SPACE;Zs;0;WS;<wide> 0020;;;;N;;;;;

Note how e.g. 200B;ZERO WIDTH SPACE is not in Zs, so:

> /\s/.test('\u200a') // 200A;HAIR SPACE;Zs;0;WS;<compat> 0020;;;;N;;;;;
true
> /\s/.test('\u200b') // 200B;ZERO WIDTH SPACE;Cf;0;BN;;;;;N;;;;;
false

\u2029 matches because it's listed in the next table, "Line Terminators": https://www.ecma-international.org/ecma-262/10.0/index.html#sec-line-terminators (table 33)

And \s is defined to match both:

Return the set of characters containing the characters that are on the right-hand side of the WhiteSpace or LineTerminator productions.

---

I find it very troubling that in ECMA 262, \w and \d have ascii semantics, while \s is expected to have unicode semantics

That's in the ECMA 262 spec and it's how it works for a long time. Breaking that would break a lot of existing js code.

If these tests are designed to follow ECMA 262 (like they are documented to do), those rules should be followed too.

JSON Schema specification documents that it uses ECMA 262 regexp dialect:

• https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.6.4
• https://json-schema.org/draft/2019-09/json-schema-core.html#rfc.section.9.3.2.2
• https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.6.3.3

[ChALkeR]

In total, that's 6 explicitly listed white-space + 4 line terminators + 17 Zs (pulled above from the unicode data) - 2 Zs already explicitly listed (0020 and 00A0) = 25 code points.

Which exactly matches the following test:

const matched = []
for (let i = 0; i < 0x110000; i++) {
  if (/\s/.test(String.fromCodePoint(i)))
    matched.push(i)
}
console.log(matched.map(x => x.toString(16).padStart(4, '0')))
console.log(matched.length)

Outputs:

[
  '0009', '000a', '000b', '000c', '000d', '0020', '00a0', '1680', '2000',
  '2001', '2002', '2003', '2004', '2005', '2006', '2007', '2008', '2009',
  '200a', '2028', '2029', '202f', '205f', '3000', 'feff'
]
25

[karenetheridge]

I'm good with this PR. Does anyone else want to review/weigh in?

[Julian]

OK I think this is good -- I took a random sample and checked and yeah at this point I think it's had enough eyes on it -- I'm sure it's still possible we've still not gotten this spot on but I'm confident enough it's better than what's here, so thanks for your expertise! (Merging)