Development Installation guide / search feature

[g-chauvel]

I tried to setup "Development Installation" to look at the search feature

So I followed instructions from https://docs.readthedocs.io/en/stable/development/install.html

In prior execution where I encountered the error described in this issue, I cleaned everything (docker containers/volumes)
and set readthedocs/settings/dev.py: ELASTICSEARCH_DSL_AUTOSYNC to True

To check that everything works, step 3, I hit "Build version" to "latest", it starts 2 builds:

build latest (4d83138) on 18385d475a3e
stuck on "Installing", log:

git clone --no-single-branch --depth 50 https://github.com/rtfd/readthedocs.org .
git checkout --force origin/master
git clean -d -f -f

build stable (c49fac9) on 18385d475a3e
stuck on "Installing", log:

git clone --no-single-branch --depth 50 https://github.com/rtfd/readthedocs.org .
git checkout --force c49fac97345916c46c747d233c992230637759a0
git clean -d -f -f 

25min after, still the same thing, looked at "ps fauxwww", "top": it seems stuck

I cliked on "Build latest" again, build #3 started and

git remote set-url origin https://github.com/rtfd/readthedocs.org
git fetch origin --force --tags --prune --prune-tags --depth 50
git checkout --force origin/master
git clean -d -f -f
python3.7 -mvirtualenv /usr/src/app/checkouts/readthedocs.org/user_builds/read-the-docs/envs/latest
/usr/src/app/checkouts/readthedocs.org/user_builds/read-the-docs/envs/latest/bin/python -m pip install --upgrade --no-cache-dir pip setuptools
/usr/src/app/checkouts/readthedocs.org/user_builds/read-the-docs/envs/latest/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2 
...

build #3 is completed in 256sec, #1 and #2 are still installing (stange, but it's not what i am trying to solve)

view docs, seach, popup, typing for example: "subversion", I have this error : "There was an error. Please try again."

docker containers log :

proxito_1   | [17/May/2021 13:43:54] readthedocs.proxito.middleware:62[16]: DEBUG Proxito Public Domain: host=read-the-docs.community.dev.readthedocs.io
proxito_1   | [17/May/2021 13:43:54] readthedocs.proxito.middleware:193[16]: DEBUG Proxito Project: slug=read-the-docs
proxito_1   | [17/May/2021 13:43:54] readthedocs.search.faceted_search:73[16]: INFO Hacking Elastic to fix search connection pooling
proxito_1   | [17/May/2021 13:43:54] elasticsearch:291[16]: WARNING POST http://search:9200/page_index/_search [status:400 request:0.006s]
proxito_1   | Internal Server Error: /_/api/v2/search/
proxito_1   | Traceback (most recent call last):
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 34, in inner
proxito_1   |     response = get_response(request)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 115, in _get_response
proxito_1   |     response = self.process_exception_by_middleware(e, request)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 113, in _get_response
proxito_1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
proxito_1   |     return view_func(*args, **kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/views/generic/base.py", line 71, in view
proxito_1   |     return self.dispatch(request, *args, **kwargs)
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/api/v2/mixins.py", line 20, in dispatch
proxito_1   |     response = super().dispatch(request, *args, **kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 509, in dispatch
proxito_1   |     response = self.handle_exception(exc)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 469, in handle_exception
proxito_1   |     self.raise_uncaught_exception(exc)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
proxito_1   |     raise exc
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 506, in dispatch
proxito_1   |     response = handler(request, *args, **kwargs)
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/api.py", line 333, in get
proxito_1   |     result = self.list()
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/api.py", line 341, in list
proxito_1   |     queryset, self.request, view=self,
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/api.py", line 85, in paginate_queryset
proxito_1   |     total_count = queryset.total_count()
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/faceted_search.py", line 277, in total_count
proxito_1   |     s = s.execute()
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch_dsl/search.py", line 715, in execute
proxito_1   |     self, es.search(index=self._index, body=self.to_dict(), **self._params)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 153, in _wrapped
proxito_1   |     return func(*args, params=params, headers=headers, **kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 1669, in search
proxito_1   |     body=body,
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 415, in perform_request
proxito_1   |     raise e
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 388, in perform_request
proxito_1   |     timeout=timeout,
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 275, in perform_request
proxito_1   |     self._raise_error(response.status, raw_data)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 323, in _raise_error
proxito_1   |     status_code, error_message, additional_info
proxito_1   | elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: [nested] nested object under path [sections] is not of nested type')
proxito_1   | [17/May/2021 13:43:54] django.request:228[16]: ERROR Internal Server Error: /_/api/v2/search/
proxito_1   | Traceback (most recent call last):
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 34, in inner
proxito_1   |     response = get_response(request)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 115, in _get_response
proxito_1   |     response = self.process_exception_by_middleware(e, request)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 113, in _get_response
proxito_1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
proxito_1   |     return view_func(*args, **kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/django/views/generic/base.py", line 71, in view
proxito_1   |     return self.dispatch(request, *args, **kwargs)
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/api/v2/mixins.py", line 20, in dispatch
proxito_1   |     response = super().dispatch(request, *args, **kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 509, in dispatch
proxito_1   |     response = self.handle_exception(exc)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 469, in handle_exception
proxito_1   |     self.raise_uncaught_exception(exc)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
proxito_1   |     raise exc
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 506, in dispatch
proxito_1   |     response = handler(request, *args, **kwargs)
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/api.py", line 333, in get
proxito_1   |     result = self.list()
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/api.py", line 341, in list
proxito_1   |     queryset, self.request, view=self,
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/api.py", line 85, in paginate_queryset
proxito_1   |     total_count = queryset.total_count()
proxito_1   |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/faceted_search.py", line 277, in total_count
proxito_1   |     s = s.execute()
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch_dsl/search.py", line 715, in execute
proxito_1   |     self, es.search(index=self._index, body=self.to_dict(), **self._params)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 153, in _wrapped
proxito_1   |     return func(*args, params=params, headers=headers, **kwargs)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 1669, in search
proxito_1   |     body=body,
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 415, in perform_request
proxito_1   |     raise e
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 388, in perform_request
proxito_1   |     timeout=timeout,
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 275, in perform_request
proxito_1   |     self._raise_error(response.status, raw_data)
proxito_1   |   File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 323, in _raise_error
proxito_1   |     status_code, error_message, additional_info
proxito_1   | elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: [nested] nested object under path [sections] is not of nested type')
proxito_1   | [17/May/2021 13:43:54] "GET /_/api/v2/search/?q=subversio&project=read-the-docs&version=latest&language=en HTTP/1.0" 500 27350
nginx_1     | 10.10.0.1 - - [17/May/2021:13:43:54 +0000] "GET /_/api/v2/search/?q=subversio&project=read-the-docs&version=latest&language=en HTTP/1.1" 500 27350 "http://read-the-docs.community.dev.readthedocs.io/en/latest/?rtd_search=subversio" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0" "-"

As written in https://docs.readthedocs.io/en/stable/development/search.html#troubleshooting
inv docker.manage 'search_index --delete'

then
inv docker.manage reindex_elasticsearch

[17/May/2021 13:51:04] readthedocs.api.v2.client:68[2819]: DEBUG Using slumber v2 with user admin, pointed at http://web:8000
[17/May/2021 13:51:04] readthedocs.builds.apps:24[2819]: INFO ShutdownBuilder/StopBuilder task could not be imported.
[17/May/2021 13:51:04] readthedocs.builds.apps:30[2819]: INFO AutoscaleBuildersTask task could not be imported.
[17/May/2021 13:51:04] readthedocs.builds.apps:44[2819]: INFO Metrics tasks could not be imported.
usage: manage.py reindex_elasticsearch [-h] --queue QUEUE
                                       [--change-index CHANGE_INDEX]
                                       [--update-from UPDATE_FROM]
                                       [--models [MODELS [MODELS ...]]]
                                       [--version] [-v {0,1,2,3}]
                                       [--settings SETTINGS]
                                       [--pythonpath PYTHONPATH] [--traceback]
                                       [--no-color] [--force-color]
manage.py reindex_elasticsearch: error: the following arguments are required: --queue

I looked at common/dockerfiles/tasks.py, "def manage" to check what is performed
and ran docker exec -it community_web_1 bash, python3 manage.py reindex_elasticsearch --queue default (queue name "default" identified from container community_build_1)
Re-indexing tasks have been created.

and got the following trace from docker container:

build_1     | [17/May/2021 13:55:14] celery.app.trace:254[1033]: ERROR Task readthedocs.search.tasks.index_objects_to_es[c7d33f7d-6b01-42cf-8416-331119a86be4] raised unexpected: ImproperlyConfigured('settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.',)
build_1     | Traceback (most recent call last):
build_1     |   File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 405, in trace_task
build_1     |     R = retval = fun(*args, **kwargs)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 697, in __protected_call__
build_1     |     return self.run(*args, **kwargs)
build_1     |   File "/usr/src/app/checkouts/readthedocs.org/readthedocs/search/tasks.py", line 52, in index_objects_to_es
build_1     |     log.info("Indexing model: %s, '%s' objects", model.__name__, queryset.count())
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 392, in count
build_1     |     return self.query.get_count(using=self.db)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/query.py", line 504, in get_count
build_1     |     number = obj.get_aggregation(using, ['__count'])['__count']
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/query.py", line 489, in get_aggregation
build_1     |     result = compiler.execute_sql(SINGLE)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 1129, in execute_sql
build_1     |     sql, params = self.as_sql()
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 488, in as_sql
build_1     |     from_, f_params = self.get_from_clause()
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 760, in get_from_clause
build_1     |     clause_sql, clause_params = self.compile(from_clause)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 405, in compile
build_1     |     sql, params = node.as_sql(self, self.connection)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/datastructures.py", line 159, in as_sql
build_1     |     base_sql = compiler.quote_name_unless_alias(self.table_name)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/models/sql/compiler.py", line 396, in quote_name_unless_alias
build_1     |     r = self.connection.ops.quote_name(name)
build_1     |   File "/usr/local/lib/python3.6/dist-packages/django/db/backends/dummy/base.py", line 20, in complain
build_1     |     raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
build_1     | django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
build_1     | [17/May/2021 13:55:14] celery.worker.strategy:151[1027]: INFO Received task: readthedocs.search.tasks.index_objects_to_es[e21ba45f-dcda-4b25-bcfd-5df391105b65]  

Is there something I am missing/doing wrong to have a working search ?

[stsewd]

Hi, builds usually get stuck when you do any changes to the code, as the server re-loads on changes. Also, make sure you are using the docker-compose version from common/dockerfiles/requirements.txt. You shouldn't need to change anything from the settings to have it working. And we should update the search commands to have the queue set or just make it work on docker.up --init

[g-chauvel]

I created a specific virtualenv for the setup and strictly followed the instructions.
pip list shows that I am using docker-compose 1.25.0 as specified in common/dockerfiles/requirements.txt

appart modification to set ELASTICSEARCH_DSL_AUTOSYNC True prior to start everything, I'm not making any live modifications.

[humitos]

@g-chauvel were you able to solve this problem?

appart modification to set ELASTICSEARCH_DSL_AUTOSYNC True prior to start everything, I'm not making any live modifications.

This is True by default when you run inv docker.up. There is no need to modify it. Are you doing something different?

[g-chauvel]

I looked a bit more at various files and indeed found that it was not required to modify anything.
So I was running the guide without any modifications after that. (I cleaned everything in docker container/volumes + repositories)

The result is the same: the search-as-you-type feature is not working out of the box.
and when i tried to reindex, I had the error described in first post.

[humitos]

Is this issue still valid? Is there any update here or action from our side required/expected?

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further. Thanks!