Proxito: Next steps on Improve URL Handling

[ericholscher]

Design doc: https://dev.readthedocs.io/en/latest/design/better-doc-urls-handling.html

First work: #9534 #9500 #9462

What are the next steps here?

[ericholscher]

@stsewd Thoughts on next steps for this work? Wanting to make sure we don't lose it.

[stsewd]

So, the last thing I did related to this is #9726. There I'm refactoring the code so introducing the new resolver is easier.

I was also playing with how the new implementation will look like at #9710 (DO NOT REVIEW, I'm only playing around with the new implementation).

Next steps could be checking what we can remove/refactor from the middleware, so it is easy to integrate with the new implementation. I was thinking of introducing the new implementation incrementally, this is, serving some of our docs with the old implementation and other with the new one, this could be archived by splitting the traffic to each view, like serving all subprojects with the new implementation or projects that start with x or similar.

Oh, we also need to migrate projects using the custom URLs feature to the new implementation

[ericholscher]

Next steps could be checking what we can remove/refactor from the middleware, so it is easy to integrate with the new implementation. I was thinking of introducing the new implementation incrementally, this is, serving some of our docs with the old implementation and other with the new one, this could be archived by splitting the traffic to each view, like serving all subprojects with the new implementation or projects that start with x or similar.

This sounds good. We could also just feature flag it?

Oh, we also need to migrate projects using the custom URLs feature to the new implementation

Yea, that should be a very small number though at least. But a couple big ones on .com we want to be careful with.

[ericholscher]

#10044

[stsewd]

Hmm, looks like the linked PRs will close this issue when merged.

[ericholscher]

Yea... silly.

[ericholscher]

Fighting the bots :D

[stsewd]

Forgot to re-open this :D

[humitos]

@stsewd can you expand here with an update for this issue? What's missing? What's blocked? What was done already?

[stsewd]

• We are using the new implementation to serve 200 and redirects on some projects
• Next week after deploy, we will be able to serve 404s with the new implementation
• We are still missing implementing support for custom URLs
• We won't see all benefits of the new implementation till we put all serving into a single view, this is migrating
  

  readthedocs.org/readthedocs/proxito/urls.py

  Lines 141 to 188 in dd07a3e

  	# (Sub)project w/ translation and versions
  	re_path(
  	(
  	r'^(?:projects/(?P<subproject_slug>{project_slug})/)?'
  	r'(?P<lang_slug>{lang_slug})/'
  	r'(?P<version_slug>{version_slug})/'
  	r'(?P<filename>{filename_slug})$'.format(**pattern_opts)
  	),
  	ServeDocs.as_view(),
  	name='docs_detail',
  	),
  	# Hack /en/latest so it redirects properly
  	# We don't want to serve the docs here,
  	# because it's at a different level of serving so relative links break.
  	re_path(
  	(
  	r'^(?:projects/(?P<subproject_slug>{project_slug})/)?'
  	r'(?P<lang_slug>{lang_slug})/'
  	r'(?P<version_slug>{version_slug})$'.format(**pattern_opts)
  	),
  	fast_404,
  	name='docs_detail_directory_indexing',
  	),
  	# # TODO: Support this?
  	# # (Sub)project translation and single version
  	# re_path(
  	# (
  	# r'^(?:|projects/(?P<subproject_slug>{project_slug})/)'
  	# r'(?P<lang_slug>{lang_slug})/'
  	# r'(?P<filename>{filename_slug})$'.format(**pattern_opts)
  	# ),
  	# serve_docs,
  	# name='docs_detail',
  	# ),
  	# (Sub)project single version
  	re_path(
  	(
  	# subproject_slash variable at the end of this regex is for ``/projects/subproject``
  	# so that it will get captured here and redirect properly.
  	r'^(?:projects/(?P<subproject_slug>{project_slug})(?P<subproject_slash>/?))?'
  	r'(?P<filename>{filename_slug})$'.format(**pattern_opts)
  	),
  	ServeDocs.as_view(),
  	name='docs_detail_singleversion_subproject',
  	),

into a single entry point

[ericholscher]

We are still missing implementing support for custom URLs

Excited for this!

[ericholscher]

• We are still missing implementing support for custom URLs
• We won't see all benefits of the new implementation till we put all serving into a single view, this is migrating
  

  readthedocs.org/readthedocs/proxito/urls.py

  Lines 141 to 188 in dd07a3e

  	# (Sub)project w/ translation and versions
  	re_path(
  	(
  	r'^(?:projects/(?P<subproject_slug>{project_slug})/)?'
  	r'(?P<lang_slug>{lang_slug})/'
  	r'(?P<version_slug>{version_slug})/'
  	r'(?P<filename>{filename_slug})$'.format(**pattern_opts)
  	),
  	ServeDocs.as_view(),
  	name='docs_detail',
  	),
  	# Hack /en/latest so it redirects properly
  	# We don't want to serve the docs here,
  	# because it's at a different level of serving so relative links break.
  	re_path(
  	(
  	r'^(?:projects/(?P<subproject_slug>{project_slug})/)?'
  	r'(?P<lang_slug>{lang_slug})/'
  	r'(?P<version_slug>{version_slug})$'.format(**pattern_opts)
  	),
  	fast_404,
  	name='docs_detail_directory_indexing',
  	),
  	# # TODO: Support this?
  	# # (Sub)project translation and single version
  	# re_path(
  	# (
  	# r'^(?:|projects/(?P<subproject_slug>{project_slug})/)'
  	# r'(?P<lang_slug>{lang_slug})/'
  	# r'(?P<filename>{filename_slug})$'.format(**pattern_opts)
  	# ),
  	# serve_docs,
  	# name='docs_detail',
  	# ),
  	# (Sub)project single version
  	re_path(
  	(
  	# subproject_slash variable at the end of this regex is for ``/projects/subproject``
  	# so that it will get captured here and redirect properly.
  	r'^(?:projects/(?P<subproject_slug>{project_slug})(?P<subproject_slash>/?))?'
  	r'(?P<filename>{filename_slug})$'.format(**pattern_opts)
  	),
  	ServeDocs.as_view(),
  	name='docs_detail_singleversion_subproject',
  	),

into a single entry point

@stsewd are either of these good to plan on for next sprint? We need to roll out the proxito serving via feature flag more widely. Seems like the single view refactor will be easier, I'm guessing?

[stsewd]

Seems like the single view refactor will be easier, I'm guessing?

That would be the final step, since it would be difficult to do that with a feature flag.

[stsewd]

Next steps after #10156 is merged:

• Decide what to do with paths that aren't docs Proxito v2: forcing a prefix on all proxito URLs, including /_/ #10181, should we support prefixing everything under /_/ or just tell people to proxy those paths as is to our servers?
• Depending on the decision, migrate existing projects to the new proxito implementation using custom prefixes.
• Roll out the new implementation to everyone (.com, since .org is already using the new implementation)
• Remove old code and use one single entry point for the serve docs view.
• Implement Proxito v2: support hiding the language from the URL #10307.

[ericholscher]

I'm going to migrate the above comment into a new issue, since we're now done with the initial work here!

The finishing work now lives in #10408