Merge remote-tracking branch 'origin/master' into fix-diffWords

Conflicts:
	release-notes.md

Author: ExplodingCabbage

README.md

@@ -29,6 +29,8 @@ Broadly, jsdiff's diff functions all take an old text and a new text and perform
 
 * `Diff.diffChars(oldStr, newStr[, options])` - diffs two blocks of text, treating each character as a token.
 
+    ("Characters" here means Unicode code points - the elements you get when you loop over a string with a `for ... of ...` loop.)
+
     Returns a list of [change objects](#change-objects).
 
     Options

release-notes.md

@@ -12,6 +12,7 @@
 
   **`diffLines` with `ignoreWhitespace: true` will no longer ignore the insertion or deletion of entire extra lines of whitespace at the end of the text**. Previously, these would not show up as insertions or deletions, as a side effect of a hack in the base diffing algorithm meant to help ignore whitespace in `diffWords`. More generally, **the undocumented special handling in the core algorithm for ignored terminals has been removed entirely.** (This special case behavior used to rewrite the final two change objects in a scenario where the final change object was an addition or deletion and its `value` was treated as equal to the empty string when compared using the diff object's `.equals` method.)
 
+- [#500](https://github.com/kpdecker/jsdiff/pull/500) **`diffChars` now diffs Unicode code points** instead of UTF-16 code units.
 - [#435](https://github.com/kpdecker/jsdiff/pull/435) **Fix `parsePatch` handling of control characters.** `parsePatch` used to interpret various unusual control characters - namely vertical tabs, form feeds, lone carriage returns without a line feed, and EBCDIC NELs - as line breaks when parsing a patch file. This was inconsistent with the behavior of both JsDiff's own `diffLines` method and also the Unix `diff` and `patch` utils, which all simply treat those control characters as ordinary characters. The result of this discrepancy was that some well-formed patches - produced either by `diff` or by JsDiff itself and handled properly by the `patch` util - would be wrongly parsed by `parsePatch`, with the effect that it would disregard the remainder of a hunk after encountering one of these control characters.
 - [#439](https://github.com/kpdecker/jsdiff/pull/439) **Prefer diffs that order deletions before insertions.** When faced with a choice between two diffs with an equal total edit distance, the Myers diff algorithm generally prefers one that does deletions before insertions rather than insertions before deletions. For instance, when diffing `abcd` against `acbd`, it will prefer a diff that says to delete the `b` and then insert a new `b` after the `c`, over a diff that says to insert a `c` before the `b` and then delete the existing `c`. JsDiff deviated from the published Myers algorithm in a way that led to it having the opposite preference in many cases, including that example. This is now fixed, meaning diffs output by JsDiff will more accurately reflect what the published Myers diff algorithm would output.
 - [#455](https://github.com/kpdecker/jsdiff/pull/455) **The `added` and `removed` properties of change objects are now guaranteed to be set to a boolean value.** (Previously, they would be set to `undefined` or omitted entirely instead of setting them to false.)

src/diff/base.js

@@ -206,7 +206,7 @@ Diff.prototype = {
     return value;
   },
   tokenize(value) {
-    return value.split('');
+    return Array.from(value);
   },
   join(chars) {
     return chars.join('');

test/diff/character.js

@@ -27,6 +27,13 @@ describe('diff/character', function() {
       });
     });
 
+    it('should treat a code point that consists of two UTF-16 code units as a single character, not two', function() {
+      const diffResult = diffChars('𝟘𝟙𝟚𝟛', '𝟘𝟙𝟚𝟜𝟝𝟞');
+      expect(diffResult.length).to.equal(3);
+      expect(diffResult[2].count).to.equal(3);
+      expect(convertChangesToXML(diffResult)).to.equal('𝟘𝟙𝟚<del>𝟛</del><ins>𝟜𝟝𝟞</ins>');
+    });
+
     describe('case insensitivity', function() {
       it("is considered when there's no difference", function() {
         const diffResult = diffChars('New Value.', 'New value.', {ignoreCase: true});