-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Refactor search code #5197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Refactor search code #5197
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit
Hold shift + click to select a range
47e3c1a
Reactor search code
ericholscher a84bb15
Show more results in JS search results
ericholscher 6f36acd
Fix docstring
ericholscher 0bfa998
Fix lint
ericholscher 936465d
Fix tests
ericholscher 0496cf6
Small fixes for search
ericholscher ab28491
Fix comment.
ericholscher 617f47f
Move to pformat and real logging.
ericholscher d0f6082
Standardize search result listings
ericholscher af8a28f
Only request 3 fragments, as that’s all we display
ericholscher 5f197ec
Fix pytest test logic.
ericholscher 465628e
Handle changing newlines to periods.
ericholscher e3e245c
Fix tests.
ericholscher e909759
Check for content to highlight
ericholscher 49098d7
Attempt to fix tests agian.
ericholscher 07dc26b
Delete some old code, and remove single function/class files.
ericholscher e9f3f3f
Fix lint error.
ericholscher d6d02da
Keep all search views in the search app
ericholscher 336e674
remove need to pass around an index_name
ericholscher eb65817
Properly filter project search
ericholscher 144ed0e
Refactor the Document and FacetedSearch classes
ericholscher d314f58
Merge remote-tracking branch 'origin/master' into readd-search-signals
ericholscher 03ad482
Fix lint
ericholscher 5dfc17a
Add TODO for signal handling
ericholscher fcc9620
Use old value for ELASTICSEARCH_DSL_AUTOSYNC in tests
ericholscher 701ecdb
Update JS file
ericholscher f636737
Add logging to our search JS to debug
ericholscher 7583f9a
Make log a bit better
ericholscher 747da90
Show the actual result length instead of the API count, since it’s wr…
ericholscher 6c317f1
Update bundle with this change.
ericholscher e2655de
Remove use of signals in favor of SettingsOverrideObject
ericholscher f457ff7
Simpler super's
ericholscher 7b2013c
Docstring for some methods/functions and small linting
humitos 0741a25
Fix search lint
ericholscher 85956e0
Comment the proper place
ericholscher 6ac2f35
Merge branch 'readd-search-signals' of github.com:rtfd/readthedocs.or…
humitos e062191
Merge remote-tracking branch 'origin/master' into readd-search-signals
ericholscher 8e4cc2b
Nicer highlight replacement syntax
ericholscher 8c7bda4
Remove search debug logging.
ericholscher 5b9f460
Use normal user object everywhere.
ericholscher e2e271b
Merge remote-tracking branch 'origin/readd-search-signals' into readd…
ericholscher a993f08
Fix typo
ericholscher d52b968
Use classic JS loop
ericholscher fc277fa
Update docs
ericholscher 417ea45
Cap operators
ericholscher 80c58c7
Fix lint again
ericholscher 72d867f
Once more with the linting
ericholscher 5f118ff
Change API queryset filter to `public(user)`
ericholscher e263e69
Small doc fixup
ericholscher 1a3e146
Support filter_user argument for not filtering users in corporate sea…
ericholscher fab9f42
Address review feedback
ericholscher 0a06726
More cleanup.
ericholscher File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,18 +3,21 @@ | |
from elasticsearch_dsl import FacetedSearch, TermsFacet | ||
from elasticsearch_dsl.query import Bool, SimpleQueryString | ||
|
||
from readthedocs.core.utils.extend import SettingsOverrideObject | ||
from readthedocs.search.documents import PageDocument, ProjectDocument | ||
from readthedocs.search.signals import before_file_search, before_project_search | ||
|
||
log = logging.getLogger(__name__) | ||
|
||
|
||
class RTDFacetedSearch(FacetedSearch): | ||
""" | ||
Pass in a user in order to filter search results by privacy. | ||
|
||
"""Overwrite the initialization in order too meet our needs.""" | ||
.. warning:: | ||
|
||
# TODO: Remove the overwrite when the elastic/elasticsearch-dsl-py#916 | ||
# See more: https://github.com/elastic/elasticsearch-dsl-py/issues/916 | ||
This isn't currently used on the .org, | ||
but is used on the .com | ||
""" | ||
|
||
def __init__(self, user, **kwargs): | ||
self.user = user | ||
|
@@ -27,20 +30,8 @@ def search(self): | |
This was causing all of the indexed content to be returned, which was | ||
never used on the client side. | ||
""" | ||
s = super().search() | ||
s = super(RTDFacetedSearch, self).search() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think just |
||
s = s.source(exclude=['content', 'headers']) | ||
resp = self.signal.send(sender=self, user=self.user, search=s) | ||
if resp: | ||
# Signal return a search object | ||
# TODO: Find a better way to handle having the signal modify the search object | ||
# This is currently depending on signal processing order, | ||
# which is brittle. | ||
try: | ||
s = resp[0][1] | ||
except AttributeError: | ||
log.exception( | ||
'Failed to return a search object from search signals' | ||
) | ||
return s | ||
|
||
def query(self, search, query): | ||
|
@@ -49,25 +40,23 @@ def query(self, search, query): | |
|
||
Also does HTML encoding of results to avoid XSS issues. | ||
""" | ||
search = super().query(search, query) | ||
search = super(RTDFacetedSearch, self).query(search, query) | ||
search = search.highlight_options(encoder='html', number_of_fragments=3) | ||
return search | ||
|
||
|
||
class ProjectSearch(RTDFacetedSearch): | ||
class ProjectSearchBase(RTDFacetedSearch): | ||
facets = {'language': TermsFacet(field='language')} | ||
signal = before_project_search | ||
doc_types = [ProjectDocument] | ||
index = ProjectDocument._doc_type.index | ||
fields = ('name^10', 'slug^5', 'description') | ||
|
||
|
||
class PageSearch(RTDFacetedSearch): | ||
class PageSearchBase(RTDFacetedSearch): | ||
facets = { | ||
'project': TermsFacet(field='project'), | ||
'version': TermsFacet(field='version') | ||
} | ||
signal = before_file_search | ||
doc_types = [PageDocument] | ||
index = PageDocument._doc_type.index | ||
fields = ['title^10', 'headers^5', 'content'] | ||
|
@@ -92,3 +81,21 @@ def query(self, search, query): | |
|
||
search = search.query(bool_query) | ||
return search | ||
|
||
|
||
class PageSearch(SettingsOverrideObject): | ||
""" | ||
Allow this class to be overridden based on CLASS_OVERRIDES setting. | ||
|
||
This is primary used on the .com to adjust how we filter our search queries | ||
""" | ||
_default_class = PageSearchBase | ||
|
||
|
||
class ProjectSearch(SettingsOverrideObject): | ||
""" | ||
Allow this class to be overridden based on CLASS_OVERRIDES setting. | ||
|
||
This is primary used on the .com to adjust how we filter our search queries | ||
""" | ||
_default_class = ProjectSearchBase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class if it's not used here, should probably lives on corporate repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Eh? It's definitely used on the .org.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed the comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nevermind. Got confused by the comment.