Use PIP instead of Conda in Read the Docs build to speed up (#1387)

For a bit of background:
Currently documentation generation is being failed in Read the Docs due to timeout.
The largest portion of time is being consumed for Conda installation, and seems like Read the Docs maintainers also encourage people to just use PIP instead (see also `https://github.com/readthedocs/readthedocs.org/issues/6254#issuecomment-542288316`).
We use Conda installation in order to install JDK 8 because Read the Docs has JDK 11 by default, and Spark 2.x does not support JDK 11.

For what this PR proposes:
This PR switches Conda to PIP to install Java 8 via using `install-jdk` package. Seems like they use AdoptOpenJDK instead but the jobs looks passing see https://readthedocs.org/projects/koalas/builds/10741596/.
Once Spark 3.0 is released, we can remove all related codes as Spark 3.0 supports JDK 11, and we don't need to install JDK 8 manually.

This PR decreases build time 1000 seconds (timeout) to around 900 seconds.

Author: HyukjinKwon

.github/workflows/master.yml

@@ -68,6 +68,9 @@ jobs:
         # as Black only works with Python 3.6+. This is hacky but we will drop
         # Python 3.5 soon so it's fine.
         sed -i '/black/d' requirements-dev.txt
+        # 'install-jdk' does not support Python 3.5. This is hacky but we will remove
+        # once Spark 3.0 is released. See also https://github.com/databricks/koalas/pull/1387
+        sed -i '/install-jdk/d' requirements-dev.txt
         pip install -r requirements-dev.txt
         pip install pandas==$PANDAS_VERSION pyarrow==$PYARROW_VERSION
         pip list
@@ -142,6 +145,9 @@ jobs:
         conda config --env --add pinned_packages pandas==$PANDAS_VERSION
         conda config --env --add pinned_packages pyarrow==$PYARROW_VERSION
         conda install -c conda-forge --yes pandas==$PANDAS_VERSION pyarrow==$PYARROW_VERSION
+        # 'install-jdk' does not exist in Conda. This is hacky but we will remove
+        # once Spark 3.0 is released. See also https://github.com/databricks/koalas/pull/1387
+        sed -i '/install-jdk/d' requirements-dev.txt
         conda install -c conda-forge --yes --freeze-installed --file requirements-dev.txt
         conda list
     - name: Run tests

.readthedocs.yml

@@ -36,6 +36,4 @@ python:
       path: .
       extra_requirements:
         - spark
-conda:
-  environment: dev/readthedocs-environment.yml
 

dev/readthedocs-environment.yml

@@ -17,6 +17,7 @@
 import errno
 
 from databricks import koalas
+from databricks.koalas import utils
 sys.path.insert(0, os.path.abspath('.'))
 
 # Remove previously generated rst files. Ignore errors just in case it stops generating whole docs.
@@ -28,6 +29,22 @@
         raise
 
 
+if "READTHEDOCS" in os.environ:
+    # For Read the Docs, it has JDK 11 by default. Spark 2.x only supports JDK 8 at most.
+    # Once Spark 3.0 is released, we can remove this as Spark 3.0 supports JDK 11.
+    # For now, we should manually install JDK 8.
+    import jdk
+    if not os.path.isdir(jdk._JRE_DIR):
+        jdk.install("8", jre=True)
+    java_home = "%s/%s" % (jdk._JRE_DIR, os.listdir(jdk._JRE_DIR)[0])
+    os.environ["JAVA_HOME"] = java_home
+    os.environ["PATH"] = "%s/bin:%s" % (java_home, os.environ["PATH"])
+
+
+# Lower the number of partitions to speed up documentation build
+utils.default_session({"spark.sql.shuffle.partitions": "4"})
+
+
 def gendoc():
     """Get releases from Github and generate reStructuredText files for release notes."""
     dev_dir = "%s/../../dev" % os.path.dirname(os.path.abspath(__file__))

docs/source/conf.py

@@ -30,3 +30,6 @@ pytest-cov
 scikit-learn
 openpyxl
 xlrd
+
+# For Read the docs
+install-jdk