Skip to content

Refactor search code #5197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 52 commits into from
Feb 6, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
47e3c1a
Reactor search code
ericholscher Jan 29, 2019
a84bb15
Show more results in JS search results
ericholscher Jan 29, 2019
6f36acd
Fix docstring
ericholscher Jan 29, 2019
0bfa998
Fix lint
ericholscher Jan 29, 2019
936465d
Fix tests
ericholscher Jan 29, 2019
0496cf6
Small fixes for search
ericholscher Jan 29, 2019
ab28491
Fix comment.
ericholscher Jan 29, 2019
617f47f
Move to pformat and real logging.
ericholscher Jan 29, 2019
d0f6082
Standardize search result listings
ericholscher Jan 29, 2019
af8a28f
Only request 3 fragments, as that’s all we display
ericholscher Jan 29, 2019
5f197ec
Fix pytest test logic.
ericholscher Jan 29, 2019
465628e
Handle changing newlines to periods.
ericholscher Jan 30, 2019
e3e245c
Fix tests.
ericholscher Jan 30, 2019
e909759
Check for content to highlight
ericholscher Jan 30, 2019
49098d7
Attempt to fix tests agian.
ericholscher Jan 30, 2019
07dc26b
Delete some old code, and remove single function/class files.
ericholscher Jan 30, 2019
e9f3f3f
Fix lint error.
ericholscher Jan 30, 2019
d6d02da
Keep all search views in the search app
ericholscher Jan 30, 2019
336e674
remove need to pass around an index_name
ericholscher Jan 30, 2019
eb65817
Properly filter project search
ericholscher Jan 30, 2019
144ed0e
Refactor the Document and FacetedSearch classes
ericholscher Jan 31, 2019
d314f58
Merge remote-tracking branch 'origin/master' into readd-search-signals
ericholscher Jan 31, 2019
03ad482
Fix lint
ericholscher Jan 31, 2019
5dfc17a
Add TODO for signal handling
ericholscher Feb 4, 2019
fcc9620
Use old value for ELASTICSEARCH_DSL_AUTOSYNC in tests
ericholscher Feb 4, 2019
701ecdb
Update JS file
ericholscher Feb 5, 2019
f636737
Add logging to our search JS to debug
ericholscher Feb 5, 2019
7583f9a
Make log a bit better
ericholscher Feb 5, 2019
747da90
Show the actual result length instead of the API count, since it’s wr…
ericholscher Feb 5, 2019
6c317f1
Update bundle with this change.
ericholscher Feb 5, 2019
e2655de
Remove use of signals in favor of SettingsOverrideObject
ericholscher Feb 5, 2019
f457ff7
Simpler super's
ericholscher Feb 5, 2019
7b2013c
Docstring for some methods/functions and small linting
humitos Feb 5, 2019
0741a25
Fix search lint
ericholscher Feb 5, 2019
85956e0
Comment the proper place
ericholscher Feb 5, 2019
6ac2f35
Merge branch 'readd-search-signals' of github.com:rtfd/readthedocs.or…
humitos Feb 5, 2019
e062191
Merge remote-tracking branch 'origin/master' into readd-search-signals
ericholscher Feb 5, 2019
8e4cc2b
Nicer highlight replacement syntax
ericholscher Feb 5, 2019
8c7bda4
Remove search debug logging.
ericholscher Feb 5, 2019
5b9f460
Use normal user object everywhere.
ericholscher Feb 5, 2019
e2e271b
Merge remote-tracking branch 'origin/readd-search-signals' into readd…
ericholscher Feb 5, 2019
a993f08
Fix typo
ericholscher Feb 5, 2019
d52b968
Use classic JS loop
ericholscher Feb 5, 2019
fc277fa
Update docs
ericholscher Feb 5, 2019
417ea45
Cap operators
ericholscher Feb 5, 2019
80c58c7
Fix lint again
ericholscher Feb 5, 2019
72d867f
Once more with the linting
ericholscher Feb 5, 2019
5f118ff
Change API queryset filter to `public(user)`
ericholscher Feb 6, 2019
e263e69
Small doc fixup
ericholscher Feb 6, 2019
1a3e146
Support filter_user argument for not filtering users in corporate sea…
ericholscher Feb 6, 2019
fab9f42
Address review feedback
ericholscher Feb 6, 2019
0a06726
More cleanup.
ericholscher Feb 6, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 29 additions & 22 deletions readthedocs/search/faceted_search.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,21 @@
from elasticsearch_dsl import FacetedSearch, TermsFacet
from elasticsearch_dsl.query import Bool, SimpleQueryString

from readthedocs.core.utils.extend import SettingsOverrideObject
from readthedocs.search.documents import PageDocument, ProjectDocument
from readthedocs.search.signals import before_file_search, before_project_search

log = logging.getLogger(__name__)


class RTDFacetedSearch(FacetedSearch):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class if it's not used here, should probably lives on corporate repo.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh? It's definitely used on the .org.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nevermind. Got confused by the comment.

"""
Pass in a user in order to filter search results by privacy.

"""Overwrite the initialization in order too meet our needs."""
.. warning::

# TODO: Remove the overwrite when the elastic/elasticsearch-dsl-py#916
# See more: https://github.com/elastic/elasticsearch-dsl-py/issues/916
This isn't currently used on the .org,
but is used on the .com
"""

def __init__(self, user, **kwargs):
self.user = user
Expand All @@ -27,20 +30,8 @@ def search(self):
This was causing all of the indexed content to be returned, which was
never used on the client side.
"""
s = super().search()
s = super(RTDFacetedSearch, self).search()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think just super() should be enough now that we are in Py3.

s = s.source(exclude=['content', 'headers'])
resp = self.signal.send(sender=self, user=self.user, search=s)
if resp:
# Signal return a search object
# TODO: Find a better way to handle having the signal modify the search object
# This is currently depending on signal processing order,
# which is brittle.
try:
s = resp[0][1]
except AttributeError:
log.exception(
'Failed to return a search object from search signals'
)
return s

def query(self, search, query):
Expand All @@ -49,25 +40,23 @@ def query(self, search, query):

Also does HTML encoding of results to avoid XSS issues.
"""
search = super().query(search, query)
search = super(RTDFacetedSearch, self).query(search, query)
search = search.highlight_options(encoder='html', number_of_fragments=3)
return search


class ProjectSearch(RTDFacetedSearch):
class ProjectSearchBase(RTDFacetedSearch):
facets = {'language': TermsFacet(field='language')}
signal = before_project_search
doc_types = [ProjectDocument]
index = ProjectDocument._doc_type.index
fields = ('name^10', 'slug^5', 'description')


class PageSearch(RTDFacetedSearch):
class PageSearchBase(RTDFacetedSearch):
facets = {
'project': TermsFacet(field='project'),
'version': TermsFacet(field='version')
}
signal = before_file_search
doc_types = [PageDocument]
index = PageDocument._doc_type.index
fields = ['title^10', 'headers^5', 'content']
Expand All @@ -92,3 +81,21 @@ def query(self, search, query):

search = search.query(bool_query)
return search


class PageSearch(SettingsOverrideObject):
"""
Allow this class to be overridden based on CLASS_OVERRIDES setting.

This is primary used on the .com to adjust how we filter our search queries
"""
_default_class = PageSearchBase


class ProjectSearch(SettingsOverrideObject):
"""
Allow this class to be overridden based on CLASS_OVERRIDES setting.

This is primary used on the .com to adjust how we filter our search queries
"""
_default_class = ProjectSearchBase
6 changes: 0 additions & 6 deletions readthedocs/search/signals.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# -*- coding: utf-8 -*-

"""We define custom Django signals to trigger before executing searches."""
import django.dispatch
from django.db.models.signals import post_save, pre_delete
from django.dispatch import receiver
from django_elasticsearch_dsl.apps import DEDConfig
Expand All @@ -10,11 +9,6 @@
from readthedocs.projects.signals import bulk_post_create, bulk_post_delete
from readthedocs.search.tasks import delete_objects_in_es, index_objects_to_es

before_project_search = django.dispatch.Signal(
providing_args=['user', 'search']
)
before_file_search = django.dispatch.Signal(providing_args=['user', 'search'])


@receiver(bulk_post_create, sender=HTMLFile)
def index_html_file(instance_list, **_):
Expand Down