UTF8 characters on version slugging -- or slugging in general

[agjohnson]

This ticket came up as part of #1407. We should make sure version slugging is handling UTF8 characters in a sane way.

[gregmuellegger]

Is there any algorithm you can suggest? Currently if a all-non-ascii name is provided the algorithm will return unknown. Partly non-ascii words like Straße will look "chunked" like stra-e.

[ericholscher]

This might be a good solution: https://pypi.python.org/pypi/unicode-slugify
-- It convert utf8 chars to their "equivalent" in ascii. We could also just
generate utf-8 slugs. I don't know a lot about this, but in general I think
we support UTF-8, but we might run into issues with nginx/etc downstream,
but perhaps that is the best solution.

On Mon, Jul 6, 2015 at 11:30 AM, Gregor Müllegger [email protected]
wrote:

Is there any algorithm you can suggest? Currently if a all-non-ascii name
is provided the algorithm will return unknown. Partly non-ascii words
like Straße will look "chunked" like stra-e.

—
Reply to this email directly or view it on GitHub
#1410 (comment)
.

Eric Holscher
Maker of the internet residing in Portland, Oregon
http://ericholscher.com

[gregmuellegger]

I think using UTF-8 in the URL is no good as there are just too many tools that do not properly handle UTF-8. However Wikipedia is doing it. The problem I see is with subdomains that contain project slugs. They definitely will make problems when they contain non-ascii characters.

[humitos]

We've been using the std library to handle this (convert all the unicode to its ascii representation):

>>> import unicodedata
>>> unicodedata.normalize('NFKD', u'camión').encode('ascii', 'ignore')
'camion'
>>> 

[humitos]

think using UTF-8 in the URL is no good as there are just too many tools that do not properly handle UTF-8. However Wikipedia is doing it.

I think it's not a problem using unicode chars on the URL.

Actually, we do support this for filenames: https://test-builds.readthedocs.io/en/unicode-filename/

but we do replace the unicode chars when they are in the version's name/identifier/slug: https://test-builds.readthedocs.io/en/d--branch/

The problem I see is with subdomains that contain project slugs. They definitely will make problems when they contain non-ascii characters.

We should probably keep the project's slug as ASCII.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

I want to reopen it and apply the solution proposed by Eric: use unicode-slugify.

I also considered django.utils.text.slugify but it does not work as good as unicode-slugify:

django.utils.text.slugify:

In [1]: from django.utils.text import slugify                                                                                                                                                 

In [2]: slugify('北京 (capital of China)')                                                                                                                                                    
Out[2]: 'capital-of-china'

unicode-slugify:

from slugify import slugify
slugify(u'北京 (capital of China)', only_ascii=True)
# u'bei-jing-capital-of-china'

(there are more examples on its docs)

[dojutsu-user]

@humitos
Just a doubt.
I think to solve this, these lines
https://github.com/rtfd/readthedocs.org/blob/05cc76b89a72cc963e091854b38adf28ab19ae3e/readthedocs/builds/version_slug.py#L85-L87
are to be replaced by using unicode-slugify.

Something like:

from slugify import slugify
slugified = slugify(content)

[humitos]

@dojutsu-user yes, it is that line. Although, I've already created a for this issue: #5186

[dojutsu-user]

@humitos
Okay. 👍