BLD: include dll in package_data on Windows

[chris-b1]

• closes Pandas 0.23.0 gives ImportError: DLL load failed #21106
• tests added / passed
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Not sure how to test this, but I believe should remove the runtime dependency by static linking against the MSVC++ runtime. @cgohlke any thoughts?

[codecov]

Codecov Report

Merging #21321 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #21321   +/-   ##
=======================================
  Coverage   91.89%   91.89%           
=======================================
  Files         153      153           
  Lines       49596    49596           
=======================================
  Hits        45576    45576           
  Misses       4020     4020

Flag	Coverage Δ
#multiple	90.29% <ø> (ø)	⬆️
#single	41.86% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/core/reshape/merge.py	94.25% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d79203a...7845eb2. Read the comment docs.

[cgohlke]

If I read this correctly, this will not only static link the C++ runtime to one C++ extension, but static link the C multi-threaded runtime library to all Pandas extension modules. That is not a good idea since Python can only load a total of ~127 of such DLLs.

It's probably better to allow distributors to include the C++ runtime DLL,, in the wheels, e.g.

diff --git a/setup.py b/setup.py
index 6febe674f..e161b1534 100755
--- a/setup.py
+++ b/setup.py
@@ -733,7 +733,7 @@ setup(name=DISTNAME,
       maintainer=AUTHOR,
       version=versioneer.get_version(),
       packages=find_packages(include=['pandas', 'pandas.*']),
-      package_data={'': ['data/*', 'templates/*'],
+      package_data={'': ['data/*', 'templates/*', '_libs/*.dll'],
                     'pandas.tests.io': ['data/legacy_hdf/*.h5',
                                         'data/legacy_pickle/*/*.pickle',
                                         'data/legacy_msgpack/*/*.msgpack',

Distributors could then copy (or not) the C++ runtime DLL to the _libs folder before building the wheel.

[jreback]

@chris-b1 is this the fix mpl did?

[chris-b1]

No, mpl packages the whole DLL. I thought static linking might be better in terms of limiting ending binary size, but wasn't aware of issue @cgohlke linked.

@cgohlke - wasn't clear to me looking at mpl setup, do you just manually copy DLL over into that directory? I see the stuff adding it to package_data but couldn't figure out how the DLL gets there in the first place.

https://github.com/matplotlib/matplotlib/blob/6ec80eac66c4828f28540f76734cae811787d626/setupext.py#L1770

[cgohlke]

wasn't clear to me looking at mpl setup, do you just manually copy DLL over into that directory?

Yes, this way it is up to the distributor to copy any required DLL to the _libs folder before building the wheel. IMO it is not worth automating this in setup.py because there are many possible Visual Studio versions and use cases to consider. Canopy and Anaconda distributions won't need to package any C/C++ runtime DLL.

[chris-b1]

@TomAugspurger - is it correct that the Windows wheels you upload to PyPI are from @cgohlke? If that's the case, @cgohlke does this work for you?

[TomAugspurger]

That's correct.

…

On Wed, Jun 6, 2018 at 12:43 PM, chris-b1 ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> - is it correct that the Windows wheels you upload to PyPI are from @cgohlke <https://github.com/cgohlke>? If that's the case, @cgohlke <https://github.com/cgohlke> does this work for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21321 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIiPF0-76XXhgbdfSUie9JWRh288Pks5t6BSzgaJpZM4UZ_5n> .

[cgohlke]

does this work for you?

I'm sure it does for Python 3. The build script will copy MSVCP140.DLL and CONCRT140.DLL from the Visual Studio redist directory to pandas\_libs\ before running setup.py, e.g.:

copy /Y /B "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT\MSVCP140.DLL" pandas\_libs\
copy /Y /B "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT\CONCRT140.DLL" pandas\_libs\

It's probably better not to copy msvcp90.dll for Python 2.7. IIRC the Python installer already installs the C++ runtime unless it's a per user installation. The .pyd file may or may not be missing a manifest to load msvcp90.dll next to it. I do not have any system for testing without the redistributable package installed. I am not aware of any other package that distributes msvcp90.dll, not matplotlib and not pyzmq.

[jreback]

does the MANIFEST.in need updating here?

[jreback]

@chris-b1 can you rebase. ready to go?

[chris-b1]

Just rebased, Should be good, I'm not familiar with MANIFEST.in, but seems to only involve source distributions?

https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute

[jreback]

@chris-b1 ok that sounds good then.