fix broken Docker image build

[rendner]

• closes BUG: ERROR building development env using Docker  #45807

This PR adds a missing flag to successfully build the pandas Docker image.

[NumberPiOso]

I have not tested the environment itself, but I could build the docker image properly.

docker build --tag pandas-pr45807-env .
Successfully built ba38c2d0969f
Successfully tagged pandas-pr45807-env:latest

I think a test to test the build of this image could avoid the same mistake.

[jreback]

sure would welcome a test of this.

[rendner]

@jreback

sure would welcome a test of this.

I expect that the the test is a shell script which runs only docker build --tag pandas-dev-env . (maybe with --no-cache?)

Can you give me a pointer where I should create such a test?

[mroeschke]

Can you give me a pointer where I should create such a test?

Maybe the easiest thing is to create a "no-op" Github action (in code-checks.yml) that builds the Dockerfile: https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action

[rendner]

@mroeschke I created a new job in the mentioned code-checks.yml.

I have run the new build job twice and both failed:

• first run
• second run

Both failed with the same CondaHTTPError: HTTP 404 NOT FOUND errors:

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/charset-normalizer-2.0.11-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/colorama-0.4.4-pyh9f0ad1d_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/idna-3.3-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/pip-22.0.3-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/pycparser-2.21-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/pyopenssl-22.0.0-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/requests-2.27.1-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/tqdm-4.62.3-pyhd8ed1ab_0.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/urllib3-1.26.8-pyhd8ed1ab_1.conda
CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.conda

Is there something wrong?

Another problem which I currently see is that in the Dockerfile the main branch is used to build the Docker image. Therefore, changes in a PR outside of the Dockerfile are only taken into account after they are merged into the main branch. This means that even if the job runs successfully for a PR, it can fail after the merge into the main branch.

[mroeschke]

For the 404, maybe make sure -c conda-forge is passed when installing the deps from environment.yml.

I'm guessing this Dockerfile shouldn't change much ideally so I think it's a good first pass test for this PR

[rendner]

@mroeschke
For me it looks like that the conda-forge channel is used (as configured). The problem seems to be that it tries to download non-existing conda files instead of the existing tar.bz2 files.

What I tried in the meantime:

• I added conda config --set use_only_tar_bz2 true
  -> flag was added to /root/.condarc: https://github.com/pandas-dev/pandas/runs/5183038474?check_suite_focus=true#step:4:569
  -> results into the same problem: 11 non-downloadable conda files (it seems the flag isn't taken into account)

• I updated conda before installing mamba: conda update -n base conda
  -> results into the same problem: but 10 non-downloadable conda files instead of 11

Since I am not familiar with conda and mamba, I would need help at this point. I have enabled additional debug output. Here is the output of the last failed build:

• conda info + conda config
• mamba env update with debug output

Maybe you can find something. Building the Dockerfile via docker build ... works on my local machine.
I also recognized that the Dockerfile and the ci/setup_env.sh use different steps to configure the setup.

[mroeschke]

For now, I think it's best to just make the Dockerfile changes here and open a new issue about testing the Dockerfile in the CI (i.e. revert the Github Action job changes).

I don't use the Dockerfile for local development so I am not sure where the issue might be

[rendner]

I digged a little bit deeper, to identify why the problem didn't happen locally.

Starting with quay.io/condaforge/miniforge3:4.11.0-2 (released 2022-02-11) I see the strange download non-existant .conda URLs behavior. The older versions work.

Here is a reduced Dockerfile to demo the problem:

# first non working base image
FROM quay.io/condaforge/miniforge3:4.11.0-2

# last working base image
# FROM quay.io/condaforge/miniforge3:4.11.0-0

# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive

# Configure apt and install packages
RUN apt-get update \
    && apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
    #
    # Verify git, process tools, lsb-release (common in install instructions for CLIs) installed
    && apt-get -y install git iproute2 procps iproute2 lsb-release \
    #
    # cleanup
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=dialog

COPY environment.yml environment.yml

RUN conda install -y mamba
RUN mamba env update -n base -f environment.yml

I will pin the base image and check if the test runs successfully.

I also found an issue that describes the problem that occurs when I use the latest base image: Mamba tries to download non-existant .conda URLs from conda-meta

[rendner]

@mroeschke The PR is ready for review from my side.

[mroeschke]

Excellent! Thanks @rendner

[NickCrews]

Thank you @rendner! I was having trouble getting docker to build, but I assumed it was on my end, so just gave up because I didn't want to dig into the weeds. Thank you for doing this, it works now for me!

[rendner]

The latest version quay.io/condaforge/miniforge3:4.11.0-4 (released Feb 26, 2022) works again.

This seems to be the commit which temporary solves the problem: conda-forge/miniforge@5429a0f

@mroeschke
We could unpin the version of the base image or wait until it is fully resolved. What do you think?

Note:
After unpinning the base image, users who have the broken image quay.io/condaforge/miniforge3:4.11.0-2 as the latest version locally must delete the image or use build option --pull to be able to build successfully.

[mroeschke]

Sure. As long as our CI check can replicate a Docker build with an image that works correctly, updating the version should be fine