URLConf: fix support for subprojects

[stsewd]

We were always exposing the subproject part in the custom URLs, that means that we are only serving subprojects,
in our proxito URLs we get around that by marking that part as optional with regex.

Here I split the patter into one url patter for the main project and the other for subprojects (we could also maybe hack this to mark it as optional in the regex pattern).

Other couple of fixes I found along the way

• The urlpattern will be used to generate both, subprojects and the main project url, we could end up generating urls like /subprojects/None/en/latest.
• If the urlconf would start with $subproject/foo the proxied api endpoint would be generated like //_

There are still more things to fix, like:

• Users would need to serve the main project from the subproject path (like if we have /sub/{subproject}, the main project would be served from /sub/en/latest), we could have two urlconfs, one for the main project and other for subprojects for further customization.
• Infinite redirects
• missing updates in our code to make use of the custom urlconf to generate the urls (like the ones from "view docs")
• Possible collitions, a path could match something that shouldn't

I was also thinking we could go for another implementation.
Instead of relying on the url patterns and their regexes, we could have a catch all paths url pattern (/anything/that/isn't/the/proxied/api), then at the view level split the components (anything, that, 'isnt', 'the', 'proxied', 'api'), and use the information from the current project being served to identify each component (if the project is single version, what part is the version, the subproject, the language, etc). Like if the project doesn't have subprojects, we know that the projects part is part of the path, or if the project doesn't have a translation, etc (avoiding collisions like #8399). This is just an idea, and we wouldn't need to hack our middleware to make it work.

[ericholscher]

I was also thinking we could go for another implementation.
Instead of relying on the url patterns and their regexes, we could have a catch all paths url pattern (/anything/that/isn't/the/proxied/api), then at the view level split the components (anything, that, 'isnt', 'the', 'proxied', 'api'), and use the information from the current project being served to identify each component (if the project is single version, what part is the version, the subproject, the language, etc). Like if the project doesn't have subprojects, we know that the projects part is part of the path, or if the project doesn't have a translation, etc (avoiding collisions like #8399). This is just an idea, and we wouldn't need to hack our middleware to make it work.

This sounds interesting, but it's non-trivial. We ran into a ton of issues with this in the past, but we might be able to create some better solutions here than we did previously. Knowing the list of languages, versions, and subprojects would make the list of possible URL's workable, but it definitely gets complex pretty fast.

We should probably think a bit more how to handle this, because I definitely don't love the current implementation, either :/

[ericholscher]

@stsewd lets discuss this on a call? Perhaps tomorrow?

[stsewd]

@stsewd lets discuss this on a call? Perhaps tomorrow?

just saw this p: but yeah, we can schedule a call

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ericholscher]

@stsewd Adding this one to the roadmap, but let's discuss other possible approaches here, as you noted in the PR description.

[humitos]

This sounds interesting, but it's non-trivial. We ran into a ton of issues with this in the past, but we might be able to create some better solutions here than we did previously. Knowing the list of languages, versions, and subprojects would make the list of possible URL's workable, but it definitely gets complex pretty fast.

@stsewd @ericholscher isn't this exactly what the new unresolver does?

readthedocs.org/readthedocs/core/unresolver.py

Line 19 in efdb633

class UnresolverBase:

[stsewd]

isn't this exactly what the new unresolver does?

The unresolver still uses django's urlconf, i.e, still tries to guess the project given the URL.

[stsewd]

I'm closing this PR since we are moving forward with a new implementation for proxito.