Documentation should specify the required unicode character encoding

[philipashlock]

The use of unicode for character encoding is part of the JSON standard with a strong emphasis on UTF-8.

"The character encoding of JSON text is always Unicode. UTF-8 is the only encoding that makes sense on the wire, but UTF-16 and UTF-32 are also permitted."
source: http://www.json.org/fatfree.html

This should be articulated in the documentation. Since this is part of the JSON standard, this could be described as part of the "Metadata File Format – JSON" section on schema.md but it might also make sense to mention in the JSON section or it's own section on catalog.md

cc: @Raseman

[gbinal]

Makes sense - Are you game to suggest a quick edit to /catalog to that effect?

[JoshData]

+1, and, to be more specific, that required encoding should be UTF-8 specifically, not just any of the encodings mentioned by the JSON spec

[waldoj]

Interesting! I had thought this was an unnecessary specification (I seem to recall writing that in response to a ticket here, many months ago), because I thought that UTF-8 was the only acceptable JSON encoding. I had no idea that UTF-16- and UTF-32 were permitted.

[konklone]

And I had no idea what the difference between UTF-8, UTF-16, and UTF-32 were until reading those Wikipedia articles just now. Now I understand. And now I would like to forget. UTF-8 forever.

[JoshData]

@waldoj That's what I thought too. Thanks @philipashlock for upping our games! :)

[waldoj]

And I had no idea what the difference between UTF-8, UTF-16, and UTF-32 were until reading those Wikipedia articles was just now.

Oh, you sweet, innocent boy.

;)

[gbinal]

Just to be clear, then, this would take the form of clarification in the documentation of Project Open Data, not necc. in anything being needed in the public data listing itself, right?

If so, does the following work and if not, would anyone be game to take a swing at it?

The json catalog files should only use UTF-8 character encoding.

[gbinal]

I've proposed #304 for this.

[gbinal]

#304 resolved this.