From a28f5b8312f97e8b200d027517c6f50c5d511d5f Mon Sep 17 00:00:00 2001 From: Marco Gorelli Date: Tue, 23 Aug 2022 10:56:00 +0100 Subject: [PATCH 1/4] recommend mamba --- .../development/contributing_codebase.rst | 5 + .../development/contributing_environment.rst | 125 ++++++++---------- 2 files changed, 62 insertions(+), 68 deletions(-) diff --git a/doc/source/development/contributing_codebase.rst b/doc/source/development/contributing_codebase.rst index bc85a54e61f22..78b5cf6a193b9 100644 --- a/doc/source/development/contributing_codebase.rst +++ b/doc/source/development/contributing_codebase.rst @@ -75,6 +75,11 @@ If you want to run checks on all recently committed files on upstream/main you c without needing to have done ``pre-commit install`` beforehand. +.. note:: + + You may want to periodically run ``pre-commit gc``, to clean up repos + which are no longer used. + .. note:: If you have conflicting installations of ``virtualenv``, then you may get an diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index c881770aa7584..e98497b006e8d 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -16,53 +16,8 @@ locally before pushing your changes. :local: -Creating an environment using Docker --------------------------------------- - -Instead of manually setting up a development environment, you can use `Docker -`_ to automatically create the environment with just several -commands. pandas provides a ``DockerFile`` in the root directory to build a Docker image -with a full pandas development environment. - -**Docker Commands** - -Build the Docker image:: - - # Build the image pandas-yourname-env - docker build --tag pandas-yourname-env . - # Or build the image by passing your GitHub username to use your own fork - docker build --build-arg gh_username=yourname --tag pandas-yourname-env . - -Run Container:: - - # Run a container and bind your local repo to the container - docker run -it -w /home/pandas --rm -v path-to-local-pandas-repo:/home/pandas pandas-yourname-env - -.. note:: - If you bind your local repo for the first time, you have to build the C extensions afterwards. - Run the following command inside the container:: - - python setup.py build_ext -j 4 - - You need to rebuild the C extensions anytime the Cython code in ``pandas/_libs`` changes. - This most frequently occurs when changing or merging branches. - -*Even easier, you can integrate Docker with the following IDEs:* - -**Visual Studio Code** - -You can use the DockerFile to launch a remote session with Visual Studio Code, -a popular free IDE, using the ``.devcontainer.json`` file. -See https://code.visualstudio.com/docs/remote/containers for details. - -**PyCharm (Professional)** - -Enable Docker support and use the Services tool window to build and manage images as well as -run and interact with containers. -See https://www.jetbrains.com/help/pycharm/docker.html for details. - -Creating an environment without Docker ---------------------------------------- +Option 1: creating an environment without Docker +------------------------------------------------ Installing a C compiler ~~~~~~~~~~~~~~~~~~~~~~~ @@ -142,14 +97,13 @@ compiler installation instructions. Let us know if you have any difficulties by opening an issue or reaching out on `Gitter `_. -Creating a Python environment -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Option 1a: using mamba (recommended) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now create an isolated pandas development environment: -* Install either `Anaconda `_, `miniconda - `_, or `miniforge `_ -* Make sure your conda is up to date (``conda update conda``) +* Install `mamba `_ +* Make sure your mamba is up to date (``mamba update mamba``) * Make sure that you have :any:`cloned the repository ` * ``cd`` to the pandas source directory @@ -162,12 +116,9 @@ We'll now kick off a three-step process: .. code-block:: none # Create and activate the build environment - conda env create -f environment.yml + mamba env create -f environment.yml conda activate pandas-dev - # or with older versions of Anaconda: - source activate pandas-dev - # Build and install pandas python setup.py build_ext -j 4 python -m pip install -e . --no-build-isolation --no-use-pep517 @@ -176,27 +127,20 @@ At this point you should be able to import pandas from your locally built versio $ python >>> import pandas - >>> print(pandas.__version__) - 0.22.0.dev0+29.g4ad6d4d74 + >>> print(pandas.__version__) # note: the exact output may differ + 1.5.0.dev0+1355.ge65a30e3eb.dirty This will create the new environment, and not touch any of your existing environments, nor any existing Python installation. -To view your environments:: - - conda info -e - To return to your root environment:: conda deactivate -See the full conda docs `here `__. +Option 1b: using pip +~~~~~~~~~~~~~~~~~~~~ - -Creating a Python environment (pip) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you aren't using conda for your development environment, follow these instructions. +If you aren't using mamba for your development environment, follow these instructions. You'll need to have at least the :ref:`minimum Python version ` that pandas supports. You also need to have ``setuptools`` 51.0.0 or later to build pandas. @@ -268,3 +212,48 @@ should already exist. # Build and install pandas python setup.py build_ext -j 4 python -m pip install -e . --no-build-isolation --no-use-pep517 + +Option 2: creating an environment using Docker +---------------------------------------------- + +Instead of manually setting up a development environment, you can use `Docker +`_ to automatically create the environment with just several +commands. pandas provides a ``DockerFile`` in the root directory to build a Docker image +with a full pandas development environment. + +**Docker Commands** + +Build the Docker image:: + + # Build the image pandas-yourname-env + docker build --tag pandas-yourname-env . + # Or build the image by passing your GitHub username to use your own fork + docker build --build-arg gh_username=yourname --tag pandas-yourname-env . + +Run Container:: + + # Run a container and bind your local repo to the container + docker run -it -w /home/pandas --rm -v path-to-local-pandas-repo:/home/pandas pandas-yourname-env + +.. note:: + If you bind your local repo for the first time, you have to build the C extensions afterwards. + Run the following command inside the container:: + + python setup.py build_ext -j 4 + + You need to rebuild the C extensions anytime the Cython code in ``pandas/_libs`` changes. + This most frequently occurs when changing or merging branches. + +*Even easier, you can integrate Docker with the following IDEs:* + +**Visual Studio Code** + +You can use the DockerFile to launch a remote session with Visual Studio Code, +a popular free IDE, using the ``.devcontainer.json`` file. +See https://code.visualstudio.com/docs/remote/containers for details. + +**PyCharm (Professional)** + +Enable Docker support and use the Services tool window to build and manage images as well as +run and interact with containers. +See https://www.jetbrains.com/help/pycharm/docker.html for details. From 3a1a26f802f74cab216e32c1047942c9dd662146 Mon Sep 17 00:00:00 2001 From: Marco Gorelli Date: Tue, 23 Aug 2022 21:11:43 +0100 Subject: [PATCH 2/4] replace conda with mamba --- .../development/contributing_environment.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index e98497b006e8d..ef4b51661fe0b 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -27,9 +27,9 @@ operations. To install pandas from source, you need to compile these C extensions, which means you need a C compiler. This process depends on which platform you're using. -If you have setup your environment using ``conda``, the packages ``c-compiler`` +If you have setup your environment using ``mamba``, the packages ``c-compiler`` and ``cxx-compiler`` will install a fitting compiler for your platform that is -compatible with the remaining conda packages. On Windows and macOS, you will +compatible with the remaining mamba packages. On Windows and macOS, you will also need to install the SDKs as they have to be distributed separately. These packages will automatically be installed by using the ``pandas`` ``environment.yml`` file. @@ -62,16 +62,16 @@ To setup the right paths on the commandline, call **macOS** -To use the ``conda``-based compilers, you will need to install the +To use the ``mamba``-based compilers, you will need to install the Developer Tools using ``xcode-select --install``. Otherwise information about compiler installation can be found here: https://devguide.python.org/setup/#macos **Linux** -For Linux-based ``conda`` installations, you won't have to install any -additional components outside of the conda environment. The instructions -below are only needed if your setup isn't based on conda environments. +For Linux-based ``mamba`` installations, you won't have to install any +additional components outside of the mamba environment. The instructions +below are only needed if your setup isn't based on mamba environments. Some Linux distributions will come with a pre-installed C compiler. To find out which compilers (and versions) are installed on your system:: @@ -117,7 +117,7 @@ We'll now kick off a three-step process: # Create and activate the build environment mamba env create -f environment.yml - conda activate pandas-dev + mamba activate pandas-dev # Build and install pandas python setup.py build_ext -j 4 @@ -135,7 +135,7 @@ nor any existing Python installation. To return to your root environment:: - conda deactivate + mamba deactivate Option 1b: using pip ~~~~~~~~~~~~~~~~~~~~ From f3b003d4e7f796a8763b0819943711fc3c6174f5 Mon Sep 17 00:00:00 2001 From: Marco Gorelli Date: Tue, 23 Aug 2022 21:19:45 +0100 Subject: [PATCH 3/4] use mamba reference --- doc/source/development/contributing_environment.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index ef4b51661fe0b..aff751e9c4b2f 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -27,7 +27,7 @@ operations. To install pandas from source, you need to compile these C extensions, which means you need a C compiler. This process depends on which platform you're using. -If you have setup your environment using ``mamba``, the packages ``c-compiler`` +If you have setup your environment using :ref:`mamba `, the packages ``c-compiler`` and ``cxx-compiler`` will install a fitting compiler for your platform that is compatible with the remaining mamba packages. On Windows and macOS, you will also need to install the SDKs as they have to be distributed separately. @@ -62,14 +62,14 @@ To setup the right paths on the commandline, call **macOS** -To use the ``mamba``-based compilers, you will need to install the +To use the :ref:`mamba `-based compilers, you will need to install the Developer Tools using ``xcode-select --install``. Otherwise information about compiler installation can be found here: https://devguide.python.org/setup/#macos **Linux** -For Linux-based ``mamba`` installations, you won't have to install any +For Linux-based :ref:`mamba ` installations, you won't have to install any additional components outside of the mamba environment. The instructions below are only needed if your setup isn't based on mamba environments. @@ -97,6 +97,8 @@ compiler installation instructions. Let us know if you have any difficulties by opening an issue or reaching out on `Gitter `_. +.. _contributing.mamba: + Option 1a: using mamba (recommended) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -191,7 +193,7 @@ Consult the docs for setting up pyenv `here `__. Below is a brief overview on how to set-up a virtual environment with Powershell under Windows. For details please refer to the -`official virtualenv user guide `__ +`official virtualenv user guide `__. Use an ENV_DIR of your choice. We'll use ~\\virtualenvs\\pandas-dev where '~' is the folder pointed to by either $env:USERPROFILE (Powershell) or From b6385c29e1b78d8a12298048de5da6a50c8ca6bc Mon Sep 17 00:00:00 2001 From: Marco Gorelli Date: Mon, 29 Aug 2022 10:20:21 +0100 Subject: [PATCH 4/4] remove unnecessary -f environment.yml --- doc/source/development/contributing_environment.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index aff751e9c4b2f..070e6d2892bc8 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -104,7 +104,7 @@ Option 1a: using mamba (recommended) Now create an isolated pandas development environment: -* Install `mamba `_ +* Install `mamba `_ * Make sure your mamba is up to date (``mamba update mamba``) * Make sure that you have :any:`cloned the repository ` * ``cd`` to the pandas source directory @@ -118,7 +118,7 @@ We'll now kick off a three-step process: .. code-block:: none # Create and activate the build environment - mamba env create -f environment.yml + mamba env create mamba activate pandas-dev # Build and install pandas