Page index fails to build unless PageIndex#put_mapping called explicitly

[jeffmhastings]

On a local install of readthedocs, I must explicitly call PageIndex#put_mapping in order for pages to index properly in elasticsearch. Initially I was only able to see the status code for the bulk_import statement in readthedocs/search/indexes.py:Index#bulk_index:

WARNING [elasticsearch:69] POST /_bulk [status:400 request:0.001s]

After modifying elasticsearch/connection/http_urllib3.py to log the response body, I got this:

{"error":"ElasticsearchIllegalArgumentException[Can't specify parent if no parent field has been configured]","status":400}

ElasticSearch version: 1.3.2

Running the script found in the elastic search setup of the documentation solves the issue. If this index setup is considered a necessary step, it might make sense to add it as an additional management command to the 'core' app. Another option might be to put the index mappings automatically, but I'm not sure where the best place to do that would be.

[ericholscher]

Aye. This should be better documented. Would accept a PR adding some info about it to the install docs.

[humitos]

Not sure how to fix this yet, but I found that my local installation of RTD fails when trying to build the elasticsearch index for LATEST versions.

Running this command fails since it build all the LATEST versions for all the existing projects:

python manage.py reindex_elasticsearch

If I select manually what project I want to build I see that all of the versions are properly indexed by elasticsearch but the LATEST fails (in the default installation it's the 20):

python manage.py reindex_elasticsearch -p pip

Output log:

[22/Feb/2017 14:26:21] INFO [readthedocs.core.management.commands.reindex_elasticsearch:40] Reindexing Version latest of Pip (20)
[22/Feb/2017 14:26:21] INFO [readthedocs.vcs_support.base:57] VCS[Pip:latest]: git rev-parse HEAD
[22/Feb/2017 14:26:21] INFO [readthedocs.vcs_support.base:65] VCS[Pip:latest]: 5b7060a20d6a4d0ce46a20809d3723d718182b81

[22/Feb/2017 14:26:21] INFO [readthedocs.projects.tasks:627] (Search Index) Sending Data: pip [quickstart logic user_guide cookbook development index configuration usage news installing reference/pip reference/pip_show reference/pip_hash reference/pip_install reference/pip_download reference/index reference/pip_uninstall reference/pip_check reference/pip_list reference/pip_wheel reference/pip_search reference/pip_freeze]
[22/Feb/2017 14:26:21] INFO [readthedocs.restapi.utils:97] Updating search index: project=pip pages=[quickstart logic user_guide cookbook development index configuration usage news installing reference/pip reference/pip_show reference/pip_hash reference/pip_install reference/pip_download reference/index reference/pip_uninstall reference/pip_check reference/pip_list reference/pip_wheel reference/pip_search reference/pip_freeze]
[22/Feb/2017 14:26:21] INFO [elasticsearch:63] PUT http://127.0.0.1:9200/readthedocs/project/6 [status:200 request:0.002s]
[22/Feb/2017 14:26:21] INFO [elasticsearch.trace:73] curl -XPUT 'http://localhost:9200/readthedocs/project/6?pretty' -d '{
  "author": [
    "humitos"
  ],
  "description": "",
  "id": 6,
  "lang": "en",
  "name": "Pip",
  "slug": "pip",
  "tags": null,
  "url": "/projects/pip/",
  "weight": 0
}'
[22/Feb/2017 14:26:21] INFO [elasticsearch.trace:73] curl -XPUT 'http://localhost:9200/readthedocs/project/6?pretty' -d '{
  "author": [
    "humitos"
  ],
  "description": "",
  "id": 6,
  "lang": "en",
  "name": "Pip",
  "slug": "pip",
  "tags": null,
  "url": "/projects/pip/",
  "weight": 0
}'
[22/Feb/2017 14:26:21] DEBUG [elasticsearch.trace:76] #[200] (0.002s)
#{
#  "_id": "6",
#  "_index": "readthedocs",
#  "_type": "project",
#  "_version": 426,
#  "created": false
#}
[22/Feb/2017 14:26:21] DEBUG [elasticsearch.trace:76] #[200] (0.002s)
#{
#  "_id": "6",
#  "_index": "readthedocs",
#  "_type": "project",
#  "_version": 426,
#  "created": false
#}
[22/Feb/2017 14:26:21] WARNING [elasticsearch:82] POST /_bulk [status:400 request:0.003s]
[22/Feb/2017 14:26:21] ERROR [readthedocs.core.management.commands.reindex_elasticsearch:51] Reindex failed for Version latest of Pip (20)
Traceback (most recent call last):
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/core/management/commands/reindex_elasticsearch.py", line 49, in handle
    version.pk, commit, delete_non_commit_files=False)
  File "/home/humitos/.virtualenvs/rtd-py2/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
    return self.run(*args, **kwargs)
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/projects/tasks.py", line 637, in update_search
    delete=delete_non_commit_files,
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/restapi/utils.py", line 155, in index_search_request
    page_obj.bulk_index(index_list, parent=project.slug, routing=route)
  File "/home/humitos/readthedocs/readthedocs.org/readthedocs/search/indexes.py", line 140, in bulk_index
    bulk_index(self.es, docs, chunk_size=chunk_size)
  File "/home/humitos/.virtualenvs/rtd-py2/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 182, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/home/humitos/.virtualenvs/rtd-py2/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 124, in streaming_bulk
    raise e
RequestError: TransportError(400, u"ElasticsearchIllegalArgumentException[Can't specify parent if no parent field has been configured]")

[humitos]

I'm removing the Good First Issue since elasticsearch is not a good place to start working with RTD :)

[safwanrahman]

Maybe we can close this in favor of #4183