[GSoC 2018] All Search Improvements #4636

safwanrahman · 2018-09-15T01:59:23Z

Here is the all code related to search upgrade. its rebased upon master and squashed the migration.

Things need to do

remove old code [Fix #4247] deleting old search code #4635
Docs of settings

This need to be merged after #4635 get merged.

TODO: integrate it with view and template

Upgrade Elasticsearch to version 6.x

[Fix #2328 #2013] Refresh search index and test for case insensitive search

[Fix #2457] Implement exact match search

[Fix #4247] deleting old search code

ericholscher

A few quick comments. I haven't reviewed it in full, but I think we should be pretty close to getting it shipped. 👍

ericholscher · 2018-09-17T15:33:24Z

.travis.yml

@@ -43,3 +43,4 @@ notifications:
 branches:
  only:
  - master
+  - search_upgrade


Can remove this.

ericholscher · 2018-09-17T15:39:32Z

readthedocs/projects/utils.py

@@ -32,18 +32,21 @@ def version_from_slug(slug, version):
    return v


-def find_file(filename):
+def find_file(basename, pattern, path):


Are we using this anywhere? Might be able to be deleted.

ericholscher · 2018-09-17T15:40:21Z

readthedocs/search/conf.py

@@ -0,0 +1 @@
+SEARCH_EXCLUDED_FILE = ['search.html', 'genindex.html', 'py-modindex.html']


I think we can move this into the search.documents file, instead of needing another file for it.

ericholscher · 2018-09-17T15:41:08Z

readthedocs/search/pagination.py

+
+
+class SearchPagination(PageNumberPagination):
+    page_size = 10


We might want to default this higher, since we aren't paginating results on the search page. We probably want 25 here perhaps.

safwanrahman · 2018-09-20T20:54:49Z

@ericholscher Fixed as per you reviewed! I think it can be merged now!

stsewd

I think you need to update the common submodule to pass the linter error https://github.com/rtfd/common/blob/52dfe7ab1dbda9d2a05f2d63bedac63e30a4e5ad/prospector.yml#L33-L33

stsewd

Just noted some changes in the docs

stsewd · 2018-09-21T15:54:07Z

docs/development/search.rst

@@ -0,0 +1,110 @@
+Search
+============


nitpick: ====== must be the same length of the title

stsewd · 2018-09-21T15:54:40Z

docs/development/search.rst

+
+Read The Docs uses Elasticsearch_ instead of the built in Sphinx search for providing better search
+results. Documents are indexed in the Elasticsearch index and the search is made through the API.
+All the Search Code is open source and lives in the `Github Repository`_.


np: Github -> GitHub

stsewd · 2018-09-21T15:56:08Z

docs/development/search.rst

+
+Indexing into Elasticsearch
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For using search, you need to index data to the Elasticsearch Index. Run `reindex_elasticsearch`


reindex_elasticsearch should be enclosed in double `

stsewd · 2018-09-21T15:57:23Z

docs/development/search.rst

+Auto Indexing
+^^^^^^^^^^^^^
+By default, Auto Indexing is turned off in development mode. To turn it on, change the
+`ELASTICSEARCH_DSL_AUTOSYNC` settings to `True` in the `readthedocs/settings/dev.py` file.


Same here, double `

stsewd · 2018-09-21T15:58:56Z

docs/development/search.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After any build is successfully finished, `HTMLFile` objects are created for each of the
+`HTML` files and the old version's `HTMLFile` object is deleted. By default,


Same here, double `

This reverts commit 182ab8d, reversing changes made to 6261427.

…4716)" This reverts commit 183b176.

Squashed commit of the following: commit b5007bb Author: Eric Holscher <[email protected]> Date: Tue Nov 20 11:41:40 2018 -0500 Clean up HTMLFile Admin commit 27e3dd4 Author: Eric Holscher <[email protected]> Date: Mon Nov 19 15:39:20 2018 -0500 Set replicas to 2 so we have 3 total copies commit ee13ed6 Merge: f221f97 ef0cd7b Author: Eric Holscher <[email protected]> Date: Tue Nov 20 11:54:57 2018 -0500 Merge remote-tracking branch 'origin/rel' into search-reapply commit f221f97 Merge: 57c7b57 7bea6d8 Author: Eric Holscher <[email protected]> Date: Mon Nov 19 15:21:00 2018 -0500 Merge pull request #4909 from safwanrahman/search_fix Tuning Elasticsearch for search improvements commit 7bea6d8 Author: Safwan Rahman <[email protected]> Date: Sat Nov 17 02:17:22 2018 +0600 more fix commit 10a6590 Author: Safwan Rahman <[email protected]> Date: Sat Nov 17 01:50:48 2018 +0600 adding comments commit 40865ae Author: Safwan Rahman <[email protected]> Date: Sat Nov 17 00:54:28 2018 +0600 fixing test commit 536874e Author: Safwan Rahman <[email protected]> Date: Sat Nov 17 00:29:34 2018 +0600 optimize for elasticsearch commit 57c7b57 Author: Eric Holscher <[email protected]> Date: Tue Nov 6 12:10:08 2018 -0600 Update migration commit a1bd6a4 Merge: 95b23c2 d2137df Author: Eric Holscher <[email protected]> Date: Tue Nov 6 12:08:38 2018 -0600 Merge branch 'master' of github.com:rtfd/readthedocs.org into search-reapply commit d2137df Merge: b63ef59 953a4ee Author: Eric Holscher <[email protected]> Date: Tue Nov 6 11:52:42 2018 -0600 Merge pull request #4545 from rtfd/davidfischer/enable-timezone-support Enable timezone support and set timezone to UTC commit 95b23c2 Merge: 75445f1 bc248aa Author: Eric Holscher <[email protected]> Date: Thu Nov 1 14:48:13 2018 -0500 Merge branch 'master' into search-reapply commit 953a4ee Merge: 0a01b96 104c3a6 Author: Eric Holscher <[email protected]> Date: Thu Nov 1 14:21:37 2018 -0500 Merge remote-tracking branch 'origin/master' into davidfischer/enable-timezone-support commit 75445f1 Author: Eric Holscher <[email protected]> Date: Wed Oct 10 15:45:29 2018 +0200 One more test commit fae116c Author: Eric Holscher <[email protected]> Date: Wed Oct 10 15:45:03 2018 +0200 Fix tests commit 5b823ad Author: Eric Holscher <[email protected]> Date: Wed Oct 10 15:43:53 2018 +0200 Don't change the query param, keep it q commit 621e14f Author: Eric Holscher <[email protected]> Date: Wed Oct 10 14:12:13 2018 +0200 Fix logic around search processing commit 5af8307 Merge: 97e3ad8 c38efc8 Author: Eric Holscher <[email protected]> Date: Wed Oct 10 12:35:16 2018 +0200 Merge remote-tracking branch 'origin/master' into search-reapply commit 97e3ad8 Author: Eric Holscher <[email protected]> Date: Thu Oct 4 09:49:59 2018 +0200 Revert "Revert "Merge pull request #4636 from rtfd/search_upgrade" (#4716)" This reverts commit 183b176. commit 0a01b96 Author: David Fischer <[email protected]> Date: Tue Sep 18 15:38:00 2018 -0700 Fix a merge fail commit 73ad35c Merge: ad7c76c ba34164 Author: David Fischer <[email protected]> Date: Tue Sep 18 15:11:48 2018 -0700 Merge branch 'master' into davidfischer/enable-timezone-support commit ad7c76c Author: David Fischer <[email protected]> Date: Tue Aug 21 12:54:05 2018 -0700 Found one more naive datetime commit 46db5d9 Author: David Fischer <[email protected]> Date: Mon Aug 20 12:09:31 2018 -0700 Enable timezone support and set timezone to UTC

safwanrahman and others added 30 commits June 9, 2018 01:42

first phase to elasticsearch 6.2.x

3c41b42

adding requirements

6410495

implementing project search, test and travis fix

272b50a

fixing travis

b8f1a06

fixing search install plugin

6c430e5

fixing up tests

035c312

fixing lint

746b378

first phase file search

de47978

indexing the file objects

ab6fffb

File searching basic backend task has been implemented

3523fab

TODO: integrate it with view and template

integrate the new search with view and template

9a5b0ed

fixing highlighting

e9b1c03

fixing up tests

37f6936

adding migration

f730556

lint fix

05f5e05

renameing

0965a94

Merge pull request #4211 from safwanrahman/search

d4f6708

Upgrade Elasticsearch to version 6.x

adding tests for case insensitive search

c13798b

fixup

26e21de

[fix #2013] Delete search index after file get removed

a464ddc

fixup according to comments

10b7f36

Implement exact match search and rewrite for operator ordering

54f0106

Merge pull request #4277 from safwanrahman/insensitive

3cdac0c

[Fix #2328 #2013] Refresh search index and test for case insensitive search

adding test for faceted search

12d7f9b

fixing linter

dd18370

[fix #4265] Port Document search API for Elasticsearch 6.x

e8ac769

Merge pull request #4292 from safwanrahman/exact_match

e9bfeee

[Fix #2457] Implement exact match search

fixup and adding test

ccf2382

more fixup

a82f006

adding more tests

044565b

safwanrahman requested review from ericholscher, agjohnson, davidfischer and a team September 15, 2018 02:16

safwanrahman added this to the Search improvements milestone Sep 15, 2018

safwanrahman and others added 3 commits September 15, 2018 20:15

fixing lint

21fed3a

deleting old search code

bf6ccbe

Merge pull request #4635 from safwanrahman/delete_old

1417f86

[Fix #4247] deleting old search code

ericholscher reviewed Sep 17, 2018

View reviewed changes

fixup as per review

295f91a

safwanrahman force-pushed the search_upgrade branch from 9b905ac to 295f91a Compare September 20, 2018 20:39

fixing test

87691fa

ericholscher added 3 commits September 21, 2018 10:15

Add ability to specify queue for indexing

bcbdd13

Fix test length

d5d7f7d

Merge remote-tracking branch 'origin/master' into search_upgrade

83db570

stsewd reviewed Sep 21, 2018

View reviewed changes

ericholscher and others added 3 commits September 21, 2018 18:51

Update common

4b7c88d

update documentation

669fe22

adding documentations for settings

d1bba06

safwanrahman mentioned this pull request Sep 21, 2018

Add options to index celery tasks in the host queue #4459

Closed

Merge branch 'master' into search_upgrade

bf71fb4

ericholscher merged commit 182ab8d into master Sep 28, 2018

ericholscher added a commit that referenced this pull request Oct 3, 2018

Revert "Merge pull request #4636 from rtfd/search_upgrade"

634b53a

This reverts commit 182ab8d, reversing changes made to 6261427.

ericholscher added a commit that referenced this pull request Oct 3, 2018

Revert "Merge pull request #4636 from rtfd/search_upgrade"

7c89ab3

This reverts commit 182ab8d, reversing changes made to 6261427.

agjohnson pushed a commit that referenced this pull request Oct 3, 2018

Revert "Merge pull request #4636 from rtfd/search_upgrade" (#4716)

183b176

This reverts commit 182ab8d, reversing changes made to 6261427.

ericholscher added a commit that referenced this pull request Oct 4, 2018

Revert "Revert "Merge pull request #4636 from rtfd/search_upgrade" (#…

97e3ad8

…4716)" This reverts commit 183b176.

stsewd deleted the search_upgrade branch February 27, 2019 02:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[GSoC 2018] All Search Improvements #4636

[GSoC 2018] All Search Improvements #4636

Uh oh!

safwanrahman commented Sep 15, 2018 •

edited

Loading

Uh oh!

ericholscher left a comment

Uh oh!

ericholscher Sep 17, 2018

Uh oh!

ericholscher Sep 17, 2018

Uh oh!

ericholscher Sep 17, 2018

Uh oh!

ericholscher Sep 17, 2018

Uh oh!

safwanrahman commented Sep 20, 2018

Uh oh!

stsewd left a comment

Uh oh!

stsewd left a comment

Uh oh!

stsewd Sep 21, 2018

Uh oh!

stsewd Sep 21, 2018

Uh oh!

stsewd Sep 21, 2018

Uh oh!

stsewd Sep 21, 2018

Uh oh!

stsewd Sep 21, 2018

Uh oh!

Uh oh!

		@@ -0,0 +1 @@
		SEARCH_EXCLUDED_FILE = ['search.html', 'genindex.html', 'py-modindex.html']

Uh oh!

[GSoC 2018] All Search Improvements #4636

[GSoC 2018] All Search Improvements #4636

Uh oh!

Conversation

safwanrahman commented Sep 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericholscher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

safwanrahman commented Sep 20, 2018

Uh oh!

stsewd left a comment

Choose a reason for hiding this comment

Uh oh!

stsewd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

safwanrahman commented Sep 15, 2018 •

edited

Loading