Skip to content

Reapply the Elastic Search upgrade to master #4722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Jan 24, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6a451ec
Reapply search upgrade to master.
ericholscher Dec 20, 2018
14a1c66
Remove hacked return value for testing
ericholscher Dec 20, 2018
2e36570
index project asynchronously
safwanrahman Dec 20, 2018
d2607e0
Merge pull request #5023 from safwanrahman/async_project
ericholscher Dec 20, 2018
ab023f7
Update docembed
ericholscher Dec 21, 2018
dbbd240
Merge remote-tracking branch 'origin/master' into search-reapply
ericholscher Jan 21, 2019
7308daf
Fix merge syntax
ericholscher Jan 21, 2019
12de58b
Don't do two searches
ericholscher Jan 21, 2019
46c4f20
Merge remote-tracking branch 'origin/master' into search-reapply
ericholscher Jan 22, 2019
60714be
Lint search branch
ericholscher Jan 22, 2019
c8adfb4
Fix env
ericholscher Jan 22, 2019
2d28826
Fix lint issues
ericholscher Jan 22, 2019
bb4db91
Update migration name
ericholscher Jan 22, 2019
61f3d3b
Merge remote-tracking branch 'origin/master' into search-reapply
ericholscher Jan 24, 2019
e289926
Adjust shards & replicas to use less memory
ericholscher Jan 24, 2019
9370e20
Remove test that was testing deleted code
ericholscher Jan 24, 2019
1ebe494
Properly use the HTML encoder on searches.
ericholscher Jan 24, 2019
2063d2d
Add an XSS test
ericholscher Jan 24, 2019
75c4ae3
Index HTMLDir projects properly
ericholscher Jan 24, 2019
46bc58f
HTMLDir excluded files
ericholscher Jan 24, 2019
f6523d4
Don't exclude all pages based on name
ericholscher Jan 24, 2019
02741ba
Remove hack doing only OR
ericholscher Jan 24, 2019
7d0e58c
Fix test
ericholscher Jan 24, 2019
444da14
New object syntax
ericholscher Jan 24, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion readthedocs/search/tests/data/docs/story.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"content": "ReadtheDocsPhilosophy\nRead the Docs is Open Source software. We have licensed the code base as MIT, which provides almost no restrictions on the use of the code.\nHowever, as a project there are things that we care about more than others. We built Read the Docs to support in the Open Source community. The code is open for people to contribute to, so that they may build features into https://readthedocs.org that they want. We also believe sharing the code openly is a valuable learning tool, especially for demonsrating how to collaborate and maintain an enormous website.\nOfficial website Support\nThe time of the core developers of Read the Docs is limited. We provide official developers support for the following things:\nLocal development on the Python code base\nUsage of https://readthedocs.org for Open Source projects\nBug fixes in the code base, as it applies to running it on https://readthedocs.org\nUnsupported\nThere are use cases that we don\u2019t support, because it doesn\u2019t further our goal of promoting in the Open Source Community.\nWe do not support:\nSpecific usage of Sphinx and Mkdocs, that don\u2019t affect our hosting\nCustom s of Read the Docs at your company\n of Read the Docs on other platforms\nAny issues outside of the Read the Docs Python Code\nRationale\nRead the Docs was founded to improve in the Open Source Community. We fully recognize and allow the code to be used for internal installs at companies, but we will not spend our time supporting it. Our time is limited, and we want to spend it on the mission that we set out to originally support.\nIf you feel strongly about installing Read the Docs internal to a company, we will happily link to third party resources on this topic. Please open an issue with a proposal if you want to take on this task.",
"content": "ReadtheDocsPhilosophy\nRead the Docs is Open Source software. We have licensed the code base as MIT, which provides almost no restrictions on the use of the code.\nHowever, as a project there are things that we care about more than others. We built Read the Docs to support in the Open Source community. The code is open for people to contribute to, so that they may build features into https://readthedocs.org that they want. <h3>XSS exploit</h3> We also believe sharing the code openly is a valuable learning tool, especially for demonsrating how to collaborate and maintain an enormous website.\nOfficial website Support\nThe time of the core developers of Read the Docs is limited. We provide official developers support for the following things:\nLocal development on the Python code base\nUsage of https://readthedocs.org for Open Source projects\nBug fixes in the code base, as it applies to running it on https://readthedocs.org\nUnsupported\nThere are use cases that we don\u2019t support, because it doesn\u2019t further our goal of promoting in the Open Source Community.\nWe do not support:\nSpecific usage of Sphinx and Mkdocs, that don\u2019t affect our hosting\nCustom s of Read the Docs at your company\n of Read the Docs on other platforms\nAny issues outside of the Read the Docs Python Code\nRationale\nRead the Docs was founded to improve in the Open Source Community. We fully recognize and allow the code to be used for internal installs at companies, but we will not spend our time supporting it. Our time is limited, and we want to spend it on the mission that we set out to originally support.\nIf you feel strongly about installing Read the Docs internal to a company, we will happily link to third party resources on this topic. Please open an issue with a proposal if you want to take on this task.",
"headers": [
"Unsupported",
"Rationale"
Expand Down
26 changes: 26 additions & 0 deletions readthedocs/search/tests/test_xss.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import pytest

from readthedocs.search.documents import PageDocument


@pytest.mark.django_db
@pytest.mark.search
class TestXSS(object):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to inheritance from object anymore


def test_facted_page_xss(self, client, project):
query = 'XSS'
page_search = PageDocument.faceted_search(query=query)
results = page_search.execute()
expected = """
&lt;h3&gt;<em>XSS</em> exploit&lt;&#x2F;h3&gt;
""".strip()
assert results[0].meta.highlight.content[0][:len(expected)] == expected

def test_simple_page_xss(self, client, project):
query = 'XSS'
page_search = PageDocument.simple_search(query=query)
results = page_search.execute()
expected = """
'&lt;h3&gt;<em>XSS</em> exploit&lt;&#x2F;h3&gt;
""".strip()
assert results[0].meta.highlight.content[0][:len(expected)] == expected