From 2b9e661fbbe66bffa225a0a58e80c50d581aa51a Mon Sep 17 00:00:00 2001 From: MarcoGorelli <> Date: Fri, 9 Dec 2022 09:15:37 +0000 Subject: [PATCH 1/4] rewrite contributing environment guide --- .../development/contributing_environment.rst | 169 ++++++------------ 1 file changed, 57 insertions(+), 112 deletions(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index 69f7f054d865d..2ea7eae0b6f2c 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -15,50 +15,25 @@ locally before pushing your changes. It's recommended to also install the :ref:` .. contents:: Table of contents: :local: +.. _contributing.c_compiler: -Option 1: creating an environment without Docker ------------------------------------------------- +Step 1: install a C compiler +---------------------------- -Installing a C compiler -~~~~~~~~~~~~~~~~~~~~~~~ - -pandas uses C extensions (mostly written using Cython) to speed up certain -operations. To install pandas from source, you need to compile these C -extensions, which means you need a C compiler. This process depends on which -platform you're using. - -If you have setup your environment using :ref:`mamba `, the packages ``c-compiler`` -and ``cxx-compiler`` will install a fitting compiler for your platform that is -compatible with the remaining mamba packages. On Windows and macOS, you will -also need to install the SDKs as they have to be distributed separately. -These packages will automatically be installed by using the ``pandas`` -``environment.yml`` file. +How to do this will depend on your platform. If you choose to user ``Docker`` +in the next step, then you can skip this step. **Windows** -You will need `Build Tools for Visual Studio 2019 -`_. - -.. warning:: - You DO NOT need to install Visual Studio 2019. - You only need "Build Tools for Visual Studio 2019" found by - scrolling down to "All downloads" -> "Tools for Visual Studio 2019". - In the installer, select the "C++ build tools" workload. - -You can install the necessary components on the commandline using -`vs_buildtools.exe `_: - -.. code:: +You will need `Build Tools for Visual Studio `_. - vs_buildtools.exe --quiet --wait --norestart --nocache ^ - --installPath C:\BuildTools ^ - --add "Microsoft.VisualStudio.Workload.VCTools;includeRecommended" ^ - --add Microsoft.VisualStudio.Component.VC.v141 ^ - --add Microsoft.VisualStudio.Component.VC.v141.x86.x64 ^ - --add Microsoft.VisualStudio.Component.Windows10SDK.17763 +Note: you do not need to install Visual Studio. +You only need "Build Tools for Visual Studio" found by +scrolling down to "All downloads" -> "Tools for Visual Studio". +In the installer, select the "C++ build tools" workload. -To setup the right paths on the commandline, call -``"C:\BuildTools\VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.16 10.0.17763.0``. +Alternatively, you could use the `WSL `_ +and consult the ``Linux`` instructions below. **macOS** @@ -69,12 +44,7 @@ https://devguide.python.org/setup/#macos **Linux** -For Linux-based :ref:`mamba ` installations, you won't have to install any -additional components outside of the mamba environment. The instructions -below are only needed if your setup isn't based on mamba environments. - -Some Linux distributions will come with a pre-installed C compiler. To find out -which compilers (and versions) are installed on your system:: +You probably already have a C compiler installed. You can check this with:: # for Debian/Ubuntu: dpkg --list | grep compiler @@ -83,38 +53,32 @@ which compilers (and versions) are installed on your system:: `GCC (GNU Compiler Collection) `_, is a widely used compiler, which supports C and a number of other languages. If GCC is listed -as an installed compiler nothing more is required. If no C compiler is -installed (or you wish to install a newer version) you can install a compiler -(GCC in the example code below) with:: - - # for recent Debian/Ubuntu: - sudo apt install build-essential - # for Red Had/RHEL/CentOS/Fedora - yum groupinstall "Development Tools" +as an installed compiler nothing more is required. -For other Linux distributions, consult your favorite search engine for -compiler installation instructions. +If no C compiler is installed, or you wish to upgrade, or you're using a different +Linux distribution, consult your favorite search engine for compiler installation/update +instructions. Let us know if you have any difficulties by opening an issue or reaching out on our contributor community :ref:`Slack `. -.. _contributing.mamba: +.. _contributing.isolated_environment: -Option 1a: using mamba (recommended) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Step 2: create an isolated environment +---------------------------------------- -Now create an isolated pandas development environment: +Before we begin, please: -* Install `mamba `_ -* Make sure your mamba is up to date (``mamba update mamba``) * Make sure that you have :any:`cloned the repository ` * ``cd`` to the pandas source directory -We'll now kick off a three-step process: +.. _contributing.mamba: + +Option 1: using mamba (recommended) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -1. Install the build dependencies -2. Build and install pandas -3. Install the optional dependencies +* Install `mamba `_ +* Make sure your mamba is up to date (``mamba update mamba``) .. code-block:: none @@ -122,28 +86,9 @@ We'll now kick off a three-step process: mamba env create --file environment.yml mamba activate pandas-dev - # Build and install pandas - python setup.py build_ext -j 4 - python -m pip install -e . --no-build-isolation --no-use-pep517 +Option 2: using pip +~~~~~~~~~~~~~~~~~~~ -At this point you should be able to import pandas from your locally built version:: - - $ python - >>> import pandas - >>> print(pandas.__version__) # note: the exact output may differ - 1.5.0.dev0+1355.ge65a30e3eb.dirty - -This will create the new environment, and not touch any of your existing environments, -nor any existing Python installation. - -To return to your root environment:: - - mamba deactivate - -Option 1b: using pip -~~~~~~~~~~~~~~~~~~~~ - -If you aren't using mamba for your development environment, follow these instructions. You'll need to have at least the :ref:`minimum Python version ` that pandas supports. You also need to have ``setuptools`` 51.0.0 or later to build pandas. @@ -162,10 +107,6 @@ You also need to have ``setuptools`` 51.0.0 or later to build pandas. # Install the build dependencies python -m pip install -r requirements-dev.txt - # Build and install pandas - python setup.py build_ext -j 4 - python -m pip install -e . --no-build-isolation --no-use-pep517 - **Unix**/**macOS with pyenv** Consult the docs for setting up pyenv `here `__. @@ -174,7 +115,6 @@ Consult the docs for setting up pyenv `here `__. # Create a virtual environment # Use an ENV_DIR of your choice. We'll use ~/Users//.pyenv/versions/pandas-dev - pyenv virtualenv # For instance: @@ -186,19 +126,15 @@ Consult the docs for setting up pyenv `here `__. # Now install the build dependencies in the cloned pandas repo python -m pip install -r requirements-dev.txt - # Build and install pandas - python setup.py build_ext -j 4 - python -m pip install -e . --no-build-isolation --no-use-pep517 - **Windows** Below is a brief overview on how to set-up a virtual environment with Powershell under Windows. For details please refer to the `official virtualenv user guide `__. -Use an ENV_DIR of your choice. We'll use ~\\virtualenvs\\pandas-dev where -'~' is the folder pointed to by either $env:USERPROFILE (Powershell) or -%USERPROFILE% (cmd.exe) environment variable. Any parent directories +Use an ENV_DIR of your choice. We'll use ``~\\virtualenvs\\pandas-dev`` where +``~`` is the folder pointed to by either ``$env:USERPROFILE`` (Powershell) or +``%USERPROFILE%`` (cmd.exe) environment variable. Any parent directories should already exist. .. code-block:: powershell @@ -212,16 +148,10 @@ should already exist. # Install the build dependencies python -m pip install -r requirements-dev.txt - # Build and install pandas - python setup.py build_ext -j 4 - python -m pip install -e . --no-build-isolation --no-use-pep517 +Option 3: using Docker +~~~~~~~~~~~~~~~~~~~~~~ -Option 2: creating an environment using Docker ----------------------------------------------- - -Instead of manually setting up a development environment, you can use `Docker -`_ to automatically create the environment with just several -commands. pandas provides a ``DockerFile`` in the root directory to build a Docker image +pandas provides a ``DockerFile`` in the root directory to build a Docker image with a full pandas development environment. **Docker Commands** @@ -238,13 +168,6 @@ Run Container:: # but if not alter ${PWD} to match your local repo path docker run -it --rm -v ${PWD}:/home/pandas pandas-dev -When inside the running container you can build and install pandas the same way as the other methods - -.. code-block:: bash - - python setup.py build_ext -j 4 - python -m pip install -e . --no-build-isolation --no-use-pep517 - *Even easier, you can integrate Docker with the following IDEs:* **Visual Studio Code** @@ -258,3 +181,25 @@ See https://code.visualstudio.com/docs/remote/containers for details. Enable Docker support and use the Services tool window to build and manage images as well as run and interact with containers. See https://www.jetbrains.com/help/pycharm/docker.html for details. + +Step 3: build and install pandas +-------------------------------- + +You can now run:: + + # Build and install pandas + python setup.py build_ext -j 4 + python -m pip install -e . --no-build-isolation --no-use-pep517 + +At this point you should be able to import pandas from your locally built version:: + + $ python + >>> import pandas + >>> print(pandas.__version__) # note: the exact output may differ + 1.5.0.dev0+1355.ge65a30e3eb.dirty + +This will create the new environment, and not touch any of your existing environments, +nor any existing Python installation. + +Note that you will need to repeat this step each time the C extensions change, for example +if you modified them or if you did a fetch and merge from ``upstream/main``. From c7345e88a67d5e1d8d34746748db07f0e139ffc1 Mon Sep 17 00:00:00 2001 From: MarcoGorelli <> Date: Mon, 12 Dec 2022 15:29:55 +0000 Subject: [PATCH 2/4] clean up --- doc/source/development/contributing_environment.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index 1cd8da777e77f..f84205ae9bcdd 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -15,8 +15,6 @@ locally before pushing your changes. It's recommended to also install the :ref:` .. contents:: Table of contents: :local: -.. _contributing.c_compiler: - Step 1: install a C compiler ---------------------------- @@ -49,7 +47,12 @@ https://devguide.python.org/setup/#macos **Linux** -You probably already have a C compiler installed. You can check this with:: +For Linux-based :ref:`mamba ` installations, you won't have to install any +additional components outside of the mamba environment. The instructions +below are only needed if your setup isn't based on mamba environments. + +Some Linux distributions will come with a pre-installed C compiler. To find out +which compilers (and versions) are installed on your system:: # for Debian/Ubuntu: dpkg --list | grep compiler @@ -67,8 +70,6 @@ instructions. Let us know if you have any difficulties by opening an issue or reaching out on our contributor community :ref:`Slack `. -.. _contributing.isolated_environment: - Step 2: create an isolated environment ---------------------------------------- From e5b8abcd893028e2c89918e4e536ab58d756054b Mon Sep 17 00:00:00 2001 From: MarcoGorelli <> Date: Tue, 13 Dec 2022 09:51:17 +0000 Subject: [PATCH 3/4] use ..note:: --- doc/source/development/contributing_environment.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index f84205ae9bcdd..391f47acb7f69 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -207,5 +207,6 @@ At this point you should be able to import pandas from your locally built versio This will create the new environment, and not touch any of your existing environments, nor any existing Python installation. -Note that you will need to repeat this step each time the C extensions change, for example -if you modified them or if you did a fetch and merge from ``upstream/main``. +.. note:: + You will need to repeat this step each time the C extensions change, for example + if you modified them or if you did a fetch and merge from ``upstream/main``. From 8c7ae06f8f53211ec13fe4eec5f57e456d280253 Mon Sep 17 00:00:00 2001 From: MarcoGorelli <> Date: Tue, 13 Dec 2022 19:25:20 +0000 Subject: [PATCH 4/4] them -> any file in pandas/_libs --- doc/source/development/contributing_environment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/development/contributing_environment.rst b/doc/source/development/contributing_environment.rst index 391f47acb7f69..942edd863a19a 100644 --- a/doc/source/development/contributing_environment.rst +++ b/doc/source/development/contributing_environment.rst @@ -209,4 +209,4 @@ nor any existing Python installation. .. note:: You will need to repeat this step each time the C extensions change, for example - if you modified them or if you did a fetch and merge from ``upstream/main``. + if you modified any file in ``pandas/_libs`` or if you did a fetch and merge from ``upstream/main``.