Install sphinx-notfound-page by default for nice 404 pages

[humitos]

We have built an amazing feature on Read the Docs to be able to serve custom 404 pages for projects together with a Sphinx extension that automatically builds a nice 404 page without any effort and almost not configuration.

However, people have to install it by themselves on their projects and configure it.

Benefits

1. all 404 pages will be nice and link to all the other sections of the project
2. no more Maze Found page with no reference to the original documentation project we are browsing
3. reduce amount of checks in our 404 handler (404.html will always exists and be served immediately)

Downsides

1. we are adding another external dependency that we need to maintain over time (see Docs: policy for supported tools and dependencies #7859)
2. would this override an existent custom 404.html file?
3. disparity between Sphinx and MkDocs features by default

Work required

• install the pip package
• configure the extension in our custom conf.py if not installed by user already
• QA there are no incompatibilities
• add it under a feature flag as initial test

[stsewd]

I think we should stop installing things that aren't needed for building docs, users should explicitly use those tools. This is the same discussion we had about installing the search extension. More adequate would be to write guides on how to integrate/use these tools.

[stsewd]

I think we can close this since we are investing more time on the new builders that don't install anything instead?

[humitos]

Yeah. We should write a guide with the "Read the Docs recommended way" or even in the "Getting started with" to mention the extensions we recommend.

Also, we could create a cookiecutter template with our recommended settings.