diff --git a/readthedocs/embed/tests/data/mkdocs/latest/index.json b/readthedocs/embed/tests/data/mkdocs/latest/index.json
deleted file mode 100644
index bdc20985c85..00000000000
--- a/readthedocs/embed/tests/data/mkdocs/latest/index.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
- "content": "MkDocs
\nProject documentation with Markdown.
\n
\nOverview
\nMkDocs is a fast, simple and downright gorgeous static site\ngenerator that's geared towards building project documentation. Documentation\nsource files are written in Markdown, and configured with a single YAML\nconfiguration file.
\nHost anywhere
\nMkDocs builds completely static HTML sites that you can host on GitHub pages,\nAmazon S3, or anywhere else you choose.
\nGreat themes available
\nThere's a stack of good looking themes available for MkDocs. Choose between\nthe built in themes: mkdocs and readthedocs, select one of the 3rd\nparty themes in the MkDocs wiki, or build your own.
\nPreview your site as you work
\nThe built-in dev-server allows you to preview your documentation as you're\nwriting it. It will even auto-reload and refresh your browser whenever you save\nyour changes.
\nEasy to customize
\nGet your project documentation looking just the way you want it by customizing\nthe theme.
\n
\nInstallation
\nInstall with a Package Manager
\nIf you have and use a package manager (such as apt-get, dnf, homebrew,\nyum, chocolatey, etc.) to install packages on your system, then you may\nwant to search for a \"MkDocs\" package and, if a recent version is available,\ninstall it with your package manager (check your system's documentation for\ndetails). That's it, you're done! Skip down to Getting Started.
\nIf your package manager does not have a recent \"MkDocs\" package, you can still\nuse your package manager to install \"Python\" and \"pip\". Then you can use pip to\ninstall MkDocs.
\nManual Installation
\nIn order to manually install MkDocs you'll need Python installed on your\nsystem, as well as the Python package manager, pip. You can check if you have\nthese already installed from the command line:
\n$ python --version\nPython 2.7.2\n$ pip --version\npip 1.5.2\n
\n\nMkDocs supports Python versions 2.7, 3.3, 3.4, 3.5 and pypy.
\nInstalling Python
\nInstall Python by downloading an installer appropriate for your system from\npython.org and running it.
\n\n
Note
\n
If you are installing Python on Windows, be sure to check the box to have\nPython added to your PATH if the installer offers such an option (it's\nnormally off by default).
\n
![\"Add](\"./img/win-py-install.png\")
\n
Installing pip
\nIf you're using a recent version of Python, the Python package manager, pip,\nis most likely installed by default. However, you may need to upgrade pip to the\nlasted version:
\npip install --upgrade pip\n
\n\nIf you need to install pip for the first time, download get-pip.py.\nThen run the following command to install it:
\npython get-pip.py\n
\n\nInstalling MkDocs
\nInstall the mkdocs package using pip:
\npip install mkdocs\n
\n\nYou should now have the mkdocs command installed on your system. Run mkdocs\n--version to check that everything worked okay.
\n$ mkdocs --version\nmkdocs, version 0.15.3\n
\n\n\n
Note
\n
If you are using Windows, some of the above commands may not work\nout-of-the-box.
\n
A quick solution may be to preface every Python command with python -m\nlike this:
\n
python -m pip install mkdocs\npython -m mkdocs\n
\n
For a more permanent solution, you may need to edit your PATH environment\nvariable to include the Scripts directory of your Python installation.\nRecent versions of Python include a script to do this for you. Navigate to\nyour Python installation directory (for example C:\\Python34\\), open the\nTools, then Scripts folder, and run the win_add2path.py file by double\nclicking on it. Alternatively, you can download the script and run it\n(python win_add2path.py).
\n
\nGetting Started
\nGetting started is super easy.
\nmkdocs new my-project\ncd my-project\n
\n\nTake a moment to review the initial project that has been created for you.
\n![\"The](\"./img/initial-layout.png\")
\nThere's a single configuration file named mkdocs.yml, and a folder named\ndocs that will contain your documentation source files. Right now the docs\nfolder just contains a single documentation page, named index.md.
\nMkDocs comes with a built-in dev-server that lets you preview your documentation\nas you work on it. Make sure you're in the same directory as the mkdocs.yml\nconfiguration file, and then start the server by running the mkdocs serve\ncommand:
\n$ mkdocs serve\nINFO - Building documentation...\nINFO - Cleaning site directory\n[I 160402 15:50:43 server:271] Serving on http://127.0.0.1:8000\n[I 160402 15:50:43 handlers:58] Start watching changes\n[I 160402 15:50:43 handlers:60] Start detecting changes\n
\n\nOpen up http://127.0.0.1:8000/ in your browser, and you'll see the default\nhome page being displayed:
\n![\"The](\"./img/screenshot.png\")
\nThe dev-server also supports auto-reloading, and will rebuild your documentation\nwhenever anything in the configuration file, documentation directory, or theme\ndirectory changes.
\nOpen the docs/index.md document in your text editor of choice, change the\ninitial heading to MkLorum, and save your changes. Your browser will\nauto-reload and you should see your updated documentation immediately.
\nNow try editing the configuration file: mkdocs.yml. Change the\nsite_name setting to MkLorum and save the file.
\nsite_name: MkLorum\n
\n\nYour browser should immediately reload, and you'll see your new site name take\neffect.
\n![\"The](\"./img/site-name.png\")
\nAdding pages
\nNow add a second page to your documentation:
\ncurl 'https://jaspervdj.be/lorem-markdownum/markdown.txt' > docs/about.md\n
\n\nAs our documentation site will include some navigation headers, you may want to\nedit the configuration file and add some information about the order, title, and\nnesting of each page in the navigation header by adding a pages\nsetting:
\nsite_name: MkLorum\npages:\n - Home: index.md\n - About: about.md\n
\n\nSave your changes and you'll now see a navigation bar with Home and About\nitems on the left as well as Search, Previous, and Next items on the\nright.
\n![\"Screenshot\"](\"./img/multipage.png\")
\nTry the menu items and navigate back and forth between pages. Then click on\nSearch. A search dialog will appear, allowing you to search for any text on\nany page. Notice that the search results include every occurrence of the search\nterm on the site and links directly to the section of the page in which the\nsearch term appears. You get of all that with no effort or configuration on your\npart!
\n![\"Screenshot\"](\"./img/search.png\")
\nTheming our documentation
\nNow change the configuration file to alter how the documentation is displayed by\nchanging the theme. Edit the mkdocs.yml file and add a theme setting:
\nsite_name: MkLorum\npages:\n - Home: index.md\n - About: about.md\ntheme: readthedocs\n
\n\nSave your changes, and you'll see the ReadTheDocs theme being used.
\n![\"Screenshot\"](\"./img/readthedocs.png\")
\nChanging the Favicon Icon
\nBy default, MkDocs uses the MkDocs favicon icon. To use a different icon, create\nan img subdirectory in your docs_dir and copy your custom favicon.ico file\nto that directory. MkDocs will automatically detect and use that file as your\nfavicon icon.
\nBuilding the site
\nThat's looking good. You're ready to deploy the first pass of your MkLorum\ndocumentation. First build the documentation:
\nmkdocs build\n
\n\nThis will create a new directory, named site. Take a look inside the\ndirectory:
\n$ ls site\nabout fonts index.html license search.html\ncss img js mkdocs sitemap.xml\n
\n\nNotice that your source documentation has been output as two HTML files named\nindex.html and about/index.html. You also have various other media that's\nbeen copied into the site directory as part of the documentation theme. You\neven have a sitemap.xml file and mkdocs/search_index.json.
\nIf you're using source code control such as git you probably don't want to\ncheck your documentation builds into the repository. Add a line containing\nsite/ to your .gitignore file.
\necho "site/" >> .gitignore\n
\n\nIf you're using another source code control tool you'll want to check its\ndocumentation on how to ignore specific directories.
\nAfter some time, files may be removed from the documentation but they will still\nreside in the site directory. To remove those stale files, just run mkdocs\nwith the --clean switch.
\nmkdocs build --clean\n
\n\nOther Commands and Options
\nThere are various other commands and options available. For a complete list of\ncommands, use the --help flag:
\nmkdocs --help\n
\n\nTo view a list of options available on a given command, use the --help flag\nwith that command. For example, to get a list of all options available for the\nbuild command run the following:
\nmkdocs build --help\n
\n\nDeploying
\nThe documentation site that you just built only uses static files so you'll be\nable to host it from pretty much anywhere. GitHub project pages and Amazon\nS3 may be good hosting options, depending upon your needs. Upload the contents\nof the entire site directory to wherever you're hosting your website from and\nyou're done. For specific instructions on a number of common hosts, see the\nDeploying your Docs page.
\nGetting help
\nTo get help with MkDocs, please use the discussion group, GitHub issues or\nthe MkDocs IRC channel #mkdocs on freenode.
",
- "url": "/",
- "language": "en",
- "title": "Home"
-}
diff --git a/readthedocs/embed/tests/test_api.py b/readthedocs/embed/tests/test_api.py
index 31ff8cc61c2..3d84bb7757e 100644
--- a/readthedocs/embed/tests/test_api.py
+++ b/readthedocs/embed/tests/test_api.py
@@ -281,14 +281,8 @@ def test_embed_sphinx(self, storage_mock, section, client):
assert response.data == expected
assert response['Cache-tag'] == 'project,project:latest'
- @mock.patch('readthedocs.embed.views.build_media_storage')
- def test_embed_mkdocs(self, storage_mock, client):
- json_file = data_path / 'mkdocs/latest/index.json'
- storage_mock.exists.return_value = True
- storage_mock.open.side_effect = self._mock_open(
- json_file.open().read()
- )
-
+ def test_embed_mkdocs(self, client):
+ """API v2 doesn't support mkdocs."""
self.version.documentation_type = MKDOCS
self.version.save()
@@ -303,31 +297,7 @@ def test_embed_mkdocs(self, storage_mock, client):
}
)
- expected = {
- 'content': mock.ANY, # too long to compare here
- 'headers': [
- {'Overview': 'overview'},
- {'Installation': 'installation'},
- {'Getting Started': 'getting-started'},
- {'Adding pages': 'adding-pages'},
- {'Theming our documentation': 'theming-our-documentation'},
- {'Changing the Favicon Icon': 'changing-the-favicon-icon'},
- {'Building the site': 'building-the-site'},
- {'Other Commands and Options': 'other-commands-and-options'},
- {'Deploying': 'deploying'},
- {'Getting help': 'getting-help'},
- ],
- 'url': 'http://project.readthedocs.io/en/latest/index.html',
- 'meta': {
- 'project': 'project',
- 'version': 'latest',
- 'doc': 'index',
- 'section': 'Installation',
- },
- }
-
- assert response.status_code == status.HTTP_200_OK
- assert response.data == expected
+ assert response.status_code == status.HTTP_404_NOT_FOUND
def test_no_access(self, client, settings):
settings.RTD_DEFAULT_FEATURES = {}
diff --git a/readthedocs/embed/views.py b/readthedocs/embed/views.py
index 98c3535881b..f60d53eeee5 100644
--- a/readthedocs/embed/views.py
+++ b/readthedocs/embed/views.py
@@ -145,6 +145,7 @@ def do_embed(*, project, version, doc=None, path=None, section=None, url=None):
content = None
headers = None
+ # Embed API v2 supports Sphinx only.
if version.is_sphinx_type:
file_content = _get_doc_content(
project=project,
@@ -160,18 +161,8 @@ def do_embed(*, project, version, doc=None, path=None, section=None, url=None):
url=url,
)
else:
- # TODO: this should read from the html file itself,
- # we don't have fjson files for mkdocs.
- file_content = _get_doc_content(
- project=project,
- version=version,
- doc=doc,
- )
- content, headers, section = parse_mkdocs(
- content=file_content,
- section=section,
- url=url,
- )
+ log.info("Using EmbedAPIv2 for a non Sphinx project.")
+ return None
if content is None:
return None
@@ -310,47 +301,3 @@ def dump(obj):
ret = [dump(clean_references(obj, url)) for obj in query_result]
return ret, headers, section
-
-
-def parse_mkdocs(content, section, url): # pylint: disable=unused-argument
- """Get the embed content for the section."""
- ret = []
- headers = []
-
- if not content or not content.get('content'):
- return (None, None, section)
-
- body = content['content']
- for element in PQ(body)('h2'):
- headers.append(recurse_while_none(element))
-
- if not section and headers:
- # If no section is sent, return the content of the first one
- section = list(headers[0].keys())[0].lower()
-
- if section:
- body_obj = PQ(body)
- escaped_section = escape_selector(section)
- section_list = body_obj(
- ':header:contains("{title}")'.format(title=str(escaped_section)))
- for num in range(len(section_list)):
- header2 = section_list.eq(num)
- # h2_title = h2.text().strip()
- # section_id = h2.attr('id')
- h2_content = ""
- next_p = header2.next()
- while next_p:
- if next_p[0].tag == 'h2':
- break
- h2_html = next_p.outerHtml()
- if h2_html:
- h2_content += "\n%s\n" % h2_html
- next_p = next_p.next()
- if h2_content:
- ret.append(h2_content)
- # ret.append({
- # 'id': section_id,
- # 'title': h2_title,
- # 'content': h2_content,
- # })
- return (ret, headers, section)
diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py
index e0e6aab5086..05dc67f98c3 100644
--- a/readthedocs/projects/models.py
+++ b/readthedocs/projects/models.py
@@ -48,7 +48,7 @@
validate_repository_url,
)
from readthedocs.projects.version_handling import determine_stable_version
-from readthedocs.search.parsers import GenericParser, SphinxParser
+from readthedocs.search.parsers import GenericParser
from readthedocs.storage import build_media_storage
from readthedocs.vcs_support.backends import backend_cls
@@ -1427,23 +1427,7 @@ class Meta:
objects = HTMLFileManager()
def get_processed_json(self):
- if (
- self.version.documentation_type == constants.GENERIC
- or self.version.is_mkdocs_type
- or self.project.has_feature(Feature.INDEX_FROM_HTML_FILES)
- ):
- parser_class = GenericParser
- elif self.version.is_sphinx_type:
- parser_class = SphinxParser
- else:
- log.warning(
- "Invalid documentation type",
- documentation_type=self.version.documentation_type,
- version_slug=self.version.slug,
- project_slug=self.project.slug,
- )
- return {}
- parser = parser_class(self.version)
+ parser = GenericParser(self.version)
return parser.parse(self.path)
@cached_property
@@ -1832,7 +1816,6 @@ def add_features(sender, **kwargs):
DISABLE_SERVER_SIDE_SEARCH = 'disable_server_side_search'
ENABLE_MKDOCS_SERVER_SIDE_SEARCH = 'enable_mkdocs_server_side_search'
DEFAULT_TO_FUZZY_SEARCH = 'default_to_fuzzy_search'
- INDEX_FROM_HTML_FILES = 'index_from_html_files'
# Build related features
SCALE_IN_PROTECTION = "scale_in_prtection"
@@ -1959,13 +1942,6 @@ def add_features(sender, **kwargs):
DEFAULT_TO_FUZZY_SEARCH,
_("Search: Default to fuzzy search for simple search queries"),
),
- (
- INDEX_FROM_HTML_FILES,
- _(
- "Search: Index content directly from html files instead or relying in other "
- "sources"
- ),
- ),
# Build related features.
(
SCALE_IN_PROTECTION,
diff --git a/readthedocs/search/parsers.py b/readthedocs/search/parsers.py
index 3e000c62562..f5a99294b79 100644
--- a/readthedocs/search/parsers.py
+++ b/readthedocs/search/parsers.py
@@ -1,10 +1,8 @@
"""JSON/HTML parsers for search indexing."""
import itertools
-import os
import re
-import orjson as json
import structlog
from selectolax.parser import HTMLParser
@@ -429,89 +427,3 @@ def _process_content(self, page, content):
"title": title,
"sections": sections,
}
-
-
-class SphinxParser(GenericParser):
-
- """
- Parser for Sphinx generated html pages.
-
- This parser relies on the fjson file generated by Sphinx.
- It checks for two paths for each html file,
- this is because the HTMLDir builder can generate the same html file from two different places:
-
- - foo.rst
- - foo/index.rst
-
- Both lead to foo/index.html.
- """
-
- def parse(self, page):
- basename = os.path.splitext(page)[0]
- fjson_paths = [f"{basename}.fjson"]
- if basename.endswith("/index"):
- new_basename = re.sub(r"\/index$", "", basename)
- fjson_paths.append(f"{new_basename}.fjson")
-
- storage_path = self.project.get_storage_path(
- type_="json",
- version_slug=self.version.slug,
- include_file=False,
- )
-
- for fjson_path in fjson_paths:
- try:
- fjson_path = self.storage.join(storage_path, fjson_path)
- if self.storage.exists(fjson_path):
- return self._process_fjson(fjson_path)
- except Exception:
- log.warning(
- "Unhandled exception during search processing file.",
- path=fjson_path,
- )
-
- return {
- "path": page,
- "title": "",
- "sections": [],
- }
-
- def _process_fjson(self, fjson_path):
- """Reads the fjson file from storage and parses it into a structured dict."""
- try:
- with self.storage.open(fjson_path, mode="r") as f:
- file_contents = f.read()
- except IOError:
- log.info("Unable to read file.", path=fjson_path)
- raise
-
- data = json.loads(file_contents)
- sections = []
- path = ""
- title = ""
-
- if "current_page_name" in data:
- path = data["current_page_name"]
- else:
- log.info("Unable to index file due to no name.", path=fjson_path)
-
- if "title" in data:
- title = data["title"]
- title = HTMLParser(title).text().strip()
- else:
- log.info("Unable to index title.", path=fjson_path)
-
- if "body" in data:
- try:
- body = self._clean_body(HTMLParser(data["body"]))
- sections = self._get_sections(title=title, body=body.body)
- except Exception as e:
- log.info("Unable to index sections.", path=fjson_path, exception=e)
- else:
- log.info("Unable to index content.", path=fjson_path)
-
- return {
- "path": path,
- "title": title,
- "sections": sections,
- }
diff --git a/readthedocs/search/tests/data/sphinx/in/autodoc.json b/readthedocs/search/tests/data/sphinx/in/autodoc.json
deleted file mode 100644
index 0967ef424bc..00000000000
--- a/readthedocs/search/tests/data/sphinx/in/autodoc.json
+++ /dev/null
@@ -1 +0,0 @@
-{}
diff --git a/readthedocs/search/tests/data/sphinx/in/httpdomain.json b/readthedocs/search/tests/data/sphinx/in/httpdomain.json
deleted file mode 100644
index 0967ef424bc..00000000000
--- a/readthedocs/search/tests/data/sphinx/in/httpdomain.json
+++ /dev/null
@@ -1 +0,0 @@
-{}
diff --git a/readthedocs/search/tests/data/sphinx/in/no-title.html b/readthedocs/search/tests/data/sphinx/in/no-title.html
index 5af77ac2e7f..6301e9cddea 100644
--- a/readthedocs/search/tests/data/sphinx/in/no-title.html
+++ b/readthedocs/search/tests/data/sphinx/in/no-title.html
@@ -1,3 +1,13 @@
+
+
+
+
+
+ <no title>
+
+
+
+
A page without a title.
Only content.
@@ -5,3 +15,6 @@
- Two
- Three
+
+
+
diff --git a/readthedocs/search/tests/data/sphinx/in/no-title.json b/readthedocs/search/tests/data/sphinx/in/no-title.json
deleted file mode 100644
index ae78faddcef..00000000000
--- a/readthedocs/search/tests/data/sphinx/in/no-title.json
+++ /dev/null
@@ -1,59 +0,0 @@
-{
- "parents": [
- {
- "link": "../",
- "title": "Guides"
- }
- ],
- "prev": {
- "link": "../conda/",
- "title": "Conda Support"
- },
- "next": {
- "link": "../feature-flags/",
- "title": "Feature Flags"
- },
- "title": "<no title>",
- "meta": {},
- "body": "",
- "metatags": "",
- "rellinks": [
- [
- "genindex",
- "General Index",
- "I",
- "index"
- ],
- [
- "http-routingtable",
- "HTTP Routing Table",
- "",
- "routing table"
- ],
- [
- "guides/feature-flags",
- "Feature Flags",
- "N",
- "next"
- ],
- [
- "guides/conda",
- "Conda Support",
- "P",
- "previous"
- ]
- ],
- "sourcename": "guides/environment-variables.rst.txt",
- "toc": "\n",
- "display_toc": false,
- "page_source_suffix": ".rst",
- "current_page_name": "guides/environment-variables",
- "sidebars": [
- "localtoc.html",
- "relations.html",
- "sourcelink.html",
- "searchbox.html"
- ],
- "customsidebar": null,
- "alabaster_version": "0.7.12"
-}
diff --git a/readthedocs/search/tests/data/sphinx/in/page.html b/readthedocs/search/tests/data/sphinx/in/page.html
index c2c051bd821..224d82846c9 100644
--- a/readthedocs/search/tests/data/sphinx/in/page.html
+++ b/readthedocs/search/tests/data/sphinx/in/page.html
@@ -1,3 +1,12 @@
+
+
+
+
+
+ I Need Secrets (or Environment Variables) in my Build
+
+
+
Content at the beginning.
@@ -158,3 +167,6 @@
Footnotes and domains
+
+
+
diff --git a/readthedocs/search/tests/data/sphinx/in/page.json b/readthedocs/search/tests/data/sphinx/in/page.json
deleted file mode 100644
index 464200077c7..00000000000
--- a/readthedocs/search/tests/data/sphinx/in/page.json
+++ /dev/null
@@ -1,59 +0,0 @@
-{
- "parents": [
- {
- "link": "../",
- "title": "Guides"
- }
- ],
- "prev": {
- "link": "../conda/",
- "title": "Conda Support"
- },
- "next": {
- "link": "../feature-flags/",
- "title": "Feature Flags"
- },
- "title": "I Need Secrets (or Environment Variables) in my Build",
- "meta": {},
- "body": "",
- "metatags": "",
- "rellinks": [
- [
- "genindex",
- "General Index",
- "I",
- "index"
- ],
- [
- "http-routingtable",
- "HTTP Routing Table",
- "",
- "routing table"
- ],
- [
- "guides/feature-flags",
- "Feature Flags",
- "N",
- "next"
- ],
- [
- "guides/conda",
- "Conda Support",
- "P",
- "previous"
- ]
- ],
- "sourcename": "guides/environment-variables.rst.txt",
- "toc": "\n",
- "display_toc": true,
- "page_source_suffix": ".rst",
- "current_page_name": "guides/environment-variables",
- "sidebars": [
- "localtoc.html",
- "relations.html",
- "sourcelink.html",
- "searchbox.html"
- ],
- "customsidebar": null,
- "alabaster_version": "0.7.12"
-}
diff --git a/readthedocs/search/tests/data/sphinx/in/requests.json b/readthedocs/search/tests/data/sphinx/in/requests.json
deleted file mode 100644
index 0967ef424bc..00000000000
--- a/readthedocs/search/tests/data/sphinx/in/requests.json
+++ /dev/null
@@ -1 +0,0 @@
-{}
diff --git a/readthedocs/search/tests/data/sphinx/in/toctree.html b/readthedocs/search/tests/data/sphinx/in/toctree.html
index 0d1fb99b542..435ff53e77c 100644
--- a/readthedocs/search/tests/data/sphinx/in/toctree.html
+++ b/readthedocs/search/tests/data/sphinx/in/toctree.html
@@ -1,3 +1,12 @@
+
+
+
+
+
+
+
+
+
Public API
This section of the documentation details the public API
@@ -87,3 +96,6 @@
Public API