Proxito v2: forcing a prefix on all proxito URLs, including /_/

[stsewd]

With #10156, we can use custom paths when serving documentation, all other paths are left as is (since they are well-defined).

But the old implementation allows you to serve those under a prefix (by creating a custom urlconf at runtime). Should we support that as well? For me, changing the pattern of how docs are served and adding a prefix to everything are two different things. This is since a user may want to just change the paths of its documentation, but still have all other things at the top level, like robots.txt, sitemap.xml, proxied APIs, etc. The main addition of supporting path prefixes in our application is that all generated links will have the proper suffix, so they always point to the correct path from where the documentation is served, there is no need for that for paths that aren't docs.

So users can proxy all docs pages as is, and all other paths that aren't docs, like /robots.txt, /sitemap.xml, /_/ proxy to our servers (or choose not to, but /_/ is kind of important), instead of allowing them to have /path-prefix/robots.txt, /path-prefix/sitemap.xml, /path-prefix/_/. I think it makes sense to have this restriction.

The need to prefix everything could be needed if they really want to scope everything under a path prefix, or if the /_/ path collides with something else they have already defined.

So, the question, should we support serving non-docs paths under a path prefix? If so, should this be an option people can opt-in instead of the default?

[humitos]

allowing them to have /path-prefix/robots.txt, /path-prefix/sitemap.xml

I understand that these files don't work in other paths that are not the root one.

/path-prefix/_/

This looks to be the important path to me. It's handy to proxy just one path and call it a day, instead of two (/path-prefix/ and _/). However, I would consider this until we have the requirement unless the work required is trivial, but I doubt it 😄

[ericholscher]

allowing them to have /path-prefix/robots.txt, /path-prefix/sitemap.xml

I understand that these files don't work in other paths that are not the root one.

I know that's true by default. I think some of them you can define a path to with a meta tag, but generally they won't work, and users just won't get them.

/path-prefix/_/

This looks to be the important path to me. It's handy to proxy just one path and call it a day, instead of two (/path-prefix/ and _/). However, I would consider this until we have the requirement unless the work required is trivial, but I doubt it 😄

Yea, I think proxying our internal stuff at the prefix sounds reasonable, if it's not hard to implement. I think it would just be somehow getting the api_host set to that prefix, and having that data make its way into our client somehow on build? We couldn't pass the data via API because the API required being hit at /_/..

[stsewd]

I think it would just be somehow getting the api_host set to that prefix, and having that data make its way into our client somehow on build? We couldn't pass the data via API because the API required being hit at /_/..

I think changing the prefix for /_/ will complicate things a little now that we are moving everything into /_/rtd-config/ to get the initial data, what we currently do is set the custom prefix in the RTD_DATA field that we inject on each page.

The only thing that worries me is if the /_/ path collides with an existing path of the user that's proxying to us, maybe can just ask the existing customers making use of this feature if they have any problems with proxying /_/ paths to us?

[stsewd]

Also, restricting /_/ to the root is easier, since we don't need to support custom pre-fixes for those paths in our code :) (which will require or override the urlconf a runtime, but I have an idea to avoid that, so, hopefully not that complicated)

[ericholscher]

@stsewd Gotcha... I think asking users to also proxy that path to us makes sense, but I can imagine they might not love that idea... But I agree it would be great if all our code didn't need to understand a prefix, but I feel like we already have an API host concept in a number of places, and adding the prefix in there wouldn't be a ton of work, but it would be really nice if we could just simplify things.

[stsewd]

We have the api_host concept, but this will conflict with our plans for the /_/rtd-config/ endpoint I believe, unless we keep injecting some data in all pages (the api_host in this case), or alternative we can ask users to pass /_/rtd-config/ as is, but then we could just ask to do it for all paths under /_/.

[ericholscher]

Yea, I think if anyone is special-casing anything, they could do the entire /_/ path... I don't have a great sense for what the right answer here is..

That said, I don't think this is a super common use case, and we should talk to our customers who are doing subpaths to see if we can get them to proxy /_/ back to us... It's a bit of a shame we didn't namespace it like /_readthedocs/ since that would be easier to special case I think.

[stsewd]

It's a bit of a shame we didn't namespace it like /_readthedocs/ since that would be easier to special case I think.

Yeah, I was thinking the same, having a namespace would be great, maybe we aren't too late for that, we can serve old paths under both prefixes, and add new paths under the new one only.

[ericholscher]

@stsewd I don't want to do too many changes for these edge cases, at this point. We can imagine lots of issues, but overall we've only had a couple customers ask about them, so we should hopefully be able to find a way to work with them to get something reasonable without changing how out system works on custom domains.

[ericholscher]

@stsewd I wonder if we can hack what nvidia needs in nginx or CF, so we can remove it from the application?

[ericholscher]

We're moving forward with the nginx hack for now, and can do this for other customers as needed.