Upgrade Elasticsearch to 6.8.3

[dojutsu-user]

Related #5620

As per the documentation for upgrading, we first need to upgrade elasticsearch to 6.8 before upgrading to 7.4.
https://www.elastic.co/guide/en/elastic-stack/7.4/upgrading-elastic-stack.html#upgrading-elastic-stack

Related PR in readthedocs-sphinx-search: readthedocs/readthedocs-sphinx-search#44

[dojutsu-user]

There are several steps involved in upgrading the ES.
According to the documentation, we need to have Rolling Upgrades when upgrading from 6.x to 6.y (y > x) and then again Rolling Upgrades when upgrading from 6.8 to 7.4.0.

Here is the full path with steps for Rolling Upgrades for upgrading to 7.4: https://www.elastic.co/guide/en/elasticsearch/reference/7.4/setup-upgrade.html

[dojutsu-user]

I think in Elastic Cloud -- Rolling Upgrades for a cluster are just one click. I am not sure though.

[humitos]

I'm always 👍 on upgrading our dependencies.

Although, considering the effort that upgrading ES involves, this may not be the case. I would ask some questions before continuing with this path:

1. are we talking about Python elasticsearch module or the ES server for the rolling updates?
2. are we using some new nice features from ES 7.x?
3. what kind of improvements are we getting in comparison with 6.x?
4. what is the version we are currently using in production?
5. do we know if 7.x is supported in AWS/Azure/ES or whatever we use in production?
6. would the changes done in this PR be compatible with the current version we have running in production?

Reading a bit the upgrading docs, it seems that what is not supported is to use the index from an old version in a newer one. So, can we just spin up an ES 7.x instance and do a full re-index while keep using the previous instance until the full re-index finishes?

I think the original issue was marked as Accepted just to stop the stale bot, but it should have been marked as Needed: design decision IMO.

[dojutsu-user]

@humitos

1. are we talking about Python elasticsearch module or the ES server for the rolling updates?

We are talking about both. Upgrading the python module as well as the ES server. Rolling upgrades will update the ES server.

2. are we using some new nice features from ES 7.x?
3. what kind of improvements are we getting in comparison with 6.x?

No, we are not using any "new" features from ES 7.x. But there are always performance improvement of which few are listed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights-7.0.0.html
Also, Elasticsearch introduces too many breaking changes in every major release. I believe that we should keep it updated as it will make it easy for us to update it in the future also.

4. what is the version we are currently using in production?

According to our docs, we are using Elasticsearch 6.3 in our production. This can also be checked if Elasticsearch endpoint is visited directly.

5. do we know if 7.x is supported in AWS/Azure/ES or whatever we use in production?

This PR upgrades ES to 6.8 and the current latest version is 7.4. So I believe that version 6.8 is available in AWS/Azure/ES. I am not sure though. I believe we are actually using Elastic Cloud.
I think @ericholscher can provide more information here.

6. would the changes done in this PR be compatible with the current version we have running in production?

No. We need to update the ES on the server also.

Reading a bit the upgrading docs, it seems that what is not supported is to use the index from an old version in a newer one. So, can we just spin up an ES 7.x instance and do a full re-index while keep using the previous instance until the full re-index finishes?

Upgrading from version 6.8 to 7.4 does not require a reindex. Just a simple rolling upgrade.
However spinning up the latest ES instance and doing a full reindex is always an option. However, I am -1 on this option. Kibana (Another tool in Elastic Stack) provides a Upgrade Assistant which tells any major/minor problems with the index before upgrading. So if we forget anything, it will tell us. Docs: https://www.elastic.co/guide/en/kibana/6.8/upgrade-assistant.html

[safwanrahman]

Better to explicit the version. otherwise it will break between version