Windows CI #23182

TomAugspurger · 2018-10-16T13:45:42Z

Trying to ensure we get the pyarrow from main, not conda-forge.

TomAugspurger · 2018-10-16T13:46:27Z

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=1261&view=logs

I cancelled the other builds.

h-vetinari · 2018-10-16T13:48:21Z

ci/azure-windows-36.yaml

  - matplotlib
  - numexpr
  - numpy=1.14*
  - openpyxl=2.5.5
-  - pyarrow
+  - pyarrow=0.9.0


I could reproduce #23180 and the first fix I tried (boost-cpp<1.67) worked. Pinning pyarrow is probably not gonna change things, because the offending error happens with 0.9.0 anyway.

https://repo.continuum.io/pkgs/main/win-64/ has a package for pyarrow 0.9.0. I'm hoping that removing feather-format will keep us in main, not not conda-forge for these packages.

But, if this fails I'll try pinning to boost-cpp<1.67

It failed. Trying a boost-cpp pin now.

This reverts commit f55a827.

TomAugspurger · 2018-10-16T15:00:45Z

Welp, the arrow failure seems to be fixed (thanks @h-vetinari)

But now I think our old friend #21786 is back perhaps?

2018-10-16T14:46:56.4427066Z ================================== FAILURES ===================================
2018-10-16T14:46:56.4429803Z __________ TestApi.test_multiple_agg_funcs[rolling-2-expected_vals0] __________
2018-10-16T14:46:56.4430094Z [gw1] win32 -- Python 3.6.6 C:\Miniconda\envs\pandas\python.exe
2018-10-16T14:46:56.4430210Z 
2018-10-16T14:46:56.4430674Z self = <pandas.tests.test_window.TestApi object at 0x000001BEBBE95CC0>
2018-10-16T14:46:56.4431798Z func = 'rolling', window_size = 2
2018-10-16T14:46:56.4432414Z expected_vals = [[nan, nan, nan, nan], [15.0, 20.0, 25.0, 20.0], [25.0, 30.0, 35.0, 30.0], [nan, nan, nan, nan], [20.0, 30.0, 35.0, 30.0], [35.0, 40.0, 60.0, 40.0], ...]
2018-10-16T14:46:56.4433525Z 
2018-10-16T14:46:56.4433684Z     @pytest.mark.parametrize("func,window_size,expected_vals", [
2018-10-16T14:46:56.4433870Z         ('rolling', 2, [[np.nan, np.nan, np.nan, np.nan],
2018-10-16T14:46:56.4434208Z                         [15., 20., 25., 20.],
2018-10-16T14:46:56.4434613Z                         [25., 30., 35., 30.],
2018-10-16T14:46:56.4435474Z                         [np.nan, np.nan, np.nan, np.nan],
2018-10-16T14:46:56.4435544Z                         [20., 30., 35., 30.],
2018-10-16T14:46:56.4435854Z                         [35., 40., 60., 40.],
2018-10-16T14:46:56.4436266Z                         [60., 80., 85., 80]]),
2018-10-16T14:46:56.4436779Z         ('expanding', None, [[10., 10., 20., 20.],
2018-10-16T14:46:56.4437141Z                              [15., 20., 25., 20.],
2018-10-16T14:46:56.4437986Z                              [20., 30., 30., 20.],
2018-10-16T14:46:56.4438067Z                              [10., 10., 30., 30.],
2018-10-16T14:46:56.4438397Z                              [20., 30., 35., 30.],
2018-10-16T14:46:56.4438860Z                              [26.666667, 40., 50., 30.],
2018-10-16T14:46:56.4439321Z                              [40., 80., 60., 30.]])])
2018-10-16T14:46:56.4440299Z     def test_multiple_agg_funcs(self, func, window_size, expected_vals):
2018-10-16T14:46:56.4440597Z         # GH 15072
2018-10-16T14:46:56.4440731Z         df = pd.DataFrame([
2018-10-16T14:46:56.4440875Z             ['A', 10, 20],
2018-10-16T14:46:56.4444532Z             ['A', 20, 30],
2018-10-16T14:46:56.4445692Z             ['A', 30, 40],
2018-10-16T14:46:56.4445965Z             ['B', 10, 30],
2018-10-16T14:46:56.4446254Z             ['B', 30, 40],
2018-10-16T14:46:56.4446547Z             ['B', 40, 80],
2018-10-16T14:46:56.4447045Z             ['B', 80, 90]], columns=['stock', 'low', 'high'])
2018-10-16T14:46:56.4447255Z     
2018-10-16T14:46:56.4447656Z         f = getattr(df.groupby('stock'), func)
2018-10-16T14:46:56.4447932Z         if window_size:
2018-10-16T14:46:56.4448319Z             window = f(window_size)
2018-10-16T14:46:56.4448585Z         else:
2018-10-16T14:46:56.4449275Z             window = f()
2018-10-16T14:46:56.4449401Z     
2018-10-16T14:46:56.4449654Z         index = pd.MultiIndex.from_tuples([
2018-10-16T14:46:56.4449945Z             ('A', 0), ('A', 1), ('A', 2),
2018-10-16T14:46:56.4450512Z             ('B', 3), ('B', 4), ('B', 5), ('B', 6)], names=['stock', None])
2018-10-16T14:46:56.4450921Z         columns = pd.MultiIndex.from_tuples([
2018-10-16T14:46:56.4451754Z             ('low', 'mean'), ('low', 'max'), ('high', 'mean'),
2018-10-16T14:46:56.4451854Z             ('high', 'min')])
2018-10-16T14:46:56.4452236Z         expected = pd.DataFrame(expected_vals, index=index, columns=columns)
2018-10-16T14:46:56.4452829Z     
2018-10-16T14:46:56.4453277Z         result = window.agg(OrderedDict((
2018-10-16T14:46:56.4453663Z             ('low', ['mean', 'max']),
2018-10-16T14:46:56.4454003Z             ('high', ['mean', 'min']),
2018-10-16T14:46:56.4454617Z         )))
2018-10-16T14:46:56.4454928Z     
2018-10-16T14:46:56.4455007Z >       tm.assert_frame_equal(result, expected)
2018-10-16T14:46:56.4455046Z 
2018-10-16T14:46:56.4455480Z pandas\tests\test_window.py:362: 
2018-10-16T14:46:56.4456659Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2018-10-16T14:46:56.4457099Z pandas\util\testing.py:1457: in assert_frame_equal
2018-10-16T14:46:56.4457539Z     obj='DataFrame.iloc[:, {idx}]'.format(idx=i))
2018-10-16T14:46:56.4458491Z pandas\util\testing.py:1292: in assert_series_equal
2018-10-16T14:46:56.4458636Z     obj='{obj}'.format(obj=obj))
2018-10-16T14:46:56.4459084Z pandas\_libs\testing.pyx:66: in pandas._libs.testing.assert_almost_equal
2018-10-16T14:46:56.4459458Z     cpdef assert_almost_equal(a, b,
2018-10-16T14:46:56.4460905Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2018-10-16T14:46:56.4461361Z 
2018-10-16T14:46:56.4461523Z >   raise_assert_detail(obj, msg, lobj, robj)
2018-10-16T14:46:56.4461755Z E   AssertionError: DataFrame.iloc[:, 0] are different
2018-10-16T14:46:56.4461882Z E   
2018-10-16T14:46:56.4462057Z E   DataFrame.iloc[:, 0] values are different (71.42857 %)
2018-10-16T14:46:56.4462334Z E   [left]:  [nan, nan, nan, nan, nan, nan, nan]
2018-10-16T14:46:56.4463072Z E   [right]: [nan, 15.0, 25.0, nan, 20.0, 35.0, 60.0]
2018-10-16T14:46:56.4463235Z

cc @chris-b1.

codecov · 2018-10-16T15:02:01Z

Codecov Report

Merging #23182 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #23182   +/-   ##
=======================================
  Coverage   92.19%   92.19%           
=======================================
  Files         169      169           
  Lines       50956    50956           
=======================================
  Hits        46977    46977           
  Misses       3979     3979

Flag	Coverage Δ
#multiple	`90.61% <ø> (ø)`	⬆️
#single	`42.27% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8d29e7...6af3461. Read the comment docs.

TomAugspurger · 2018-10-16T15:16:13Z

I think https://dev.azure.com/pandas-dev/pandas/_settings/agentqueues?queueId=3&_a=details may have the details of that worker, if that means anything to anyone.

h-vetinari · 2018-10-16T15:59:31Z

So the pyarrow import still wasn't fully sorted, because later on in the log, import pyarrow.parquet failed (also for me locally). Adding parquet-cpp to the list of requirements should get it from main (I ran conda install parquet-cpp and got an alternative to conda-forge), and then the import works again for me.

Regarding the window failures (which I can't reproduce), there's a follow-up to @chris-b1's MVSC bug report here, which recommends - until MVSC is fixed upstream - to bracket _libs.window.{add_var|add_mean|remove_mean} with #pragma optimize("", off) [...] #pragma optimize("", on).

TomAugspurger · 2018-10-16T19:10:23Z

Do pragmas get included in the code generated by Cython? Based on some limited testing / searching, I don't think they do...

Regardless, I thought #21813 that, so maybe it's a different issue.

TomAugspurger · 2018-10-16T19:21:31Z

c512f42 is a bit of a hammer. I don't intend to actually merge it.

h-vetinari · 2018-10-16T19:59:51Z

Remaining failures should be solved with:

So the pyarrow import still wasn't fully sorted, because later on in the log, import pyarrow.parquet failed (also for me locally). Adding parquet-cpp to the list of requirements should get it from main (I ran conda install parquet-cpp and got an alternative to conda-forge), and then the import works again for me.

c512f42 did work, but then that points to the same underlying error as in #21786, no?

TomAugspurger · 2018-10-16T20:02:44Z

c512f42 did work, but then that points to the same underlying error as in #21786, no?

Seems like it. @chris-b1 is there a reason you applied the isnan fix only in those places? Is using it everywhere like I did a bad idea?

chris-b1 · 2018-10-16T20:08:44Z

There's likely a little performance cost, because the call to isnan won't be inlined, but otherwise might need to be applied everywhere.

…

On Tue, Oct 16, 2018, 3:04 PM Tom Augspurger ***@***.***> wrote: c512f42 <c512f42> did work, but then that points to the same underlying error as in #21786 <#21786>, no? Seems like it. @chris-b1 <https://github.com/chris-b1> is there a reason you applied the isnan fix only in those places? Is using it everywhere like I did a bad idea? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1b_GBXGpt4pkQYlWisi0xSguOCcSD2ks5uljvJgaJpZM4XeZIq> .

TomAugspurger · 2018-10-16T20:17:03Z

So, this (possible) perf hit would be on all platforms correct, not just windows with MSVC_VER < 1800?

(How) do we know that the compiler doesn't inline it? I assume you discovered that when debugging originally...

I'll try to be a bit more surgical in where we apply it

TomAugspurger · 2018-10-17T01:37:13Z

Locally, with these changes

       before           after         ratio
     [7ef69bdb]       [c512f420]
     <fix-windows-ci~2>       <fix-windows-ci~1>
+     2.13±0.04ms       2.85±0.3ms     1.34  rolling.Quantile.time_quantile('DataFrame', 1000, 'float', 0, 'lower')
+     1.48±0.01ms       1.92±0.1ms     1.30  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 1, 'lower')
+      18.6±0.3ms         24.1±1ms     1.30  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'int', 'std')
+        73.6±1ms         93.8±3ms     1.27  rolling.Quantile.time_quantile('Series', 1000, 'float', 0.5, 'lower')
+     1.86±0.03ms       2.33±0.3ms     1.25  rolling.Quantile.time_quantile('Series', 1000, 'int', 1, 'midpoint')
+     1.58±0.09ms      1.97±0.07ms     1.25  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 1, 'higher')
+     1.67±0.01ms      2.04±0.05ms     1.23  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 1, 'nearest')
+      72.5±0.7ms         88.5±3ms     1.22  rolling.Quantile.time_quantile('Series', 1000, 'float', 0.5, 'nearest')
+        72.1±1ms         87.5±2ms     1.21  rolling.Quantile.time_quantile('Series', 1000, 'float', 0.5, 'higher')
+     2.14±0.07ms       2.60±0.1ms     1.21  rolling.Quantile.time_quantile('DataFrame', 1000, 'float', 0, 'higher')
+     1.78±0.07ms       2.16±0.2ms     1.21  rolling.Quantile.time_quantile('Series', 1000, 'int', 1, 'linear')
+     2.34±0.03ms      2.81±0.09ms     1.20  rolling.Quantile.time_quantile('Series', 1000, 'float', 0, 'higher')
+     4.60±0.03ms         5.45±1ms     1.19  rolling.VariableWindowMethods.time_rolling('Series', '1h', 'float', 'skew')
+     3.16±0.04ms       3.73±0.5ms     1.18  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'int', 'count')
+     1.62±0.03ms       1.91±0.1ms     1.18  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 0, 'linear')
+     2.84±0.02ms      3.32±0.08ms     1.17  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'float', 'sum')
+     1.87±0.03ms       2.18±0.2ms     1.17  rolling.Quantile.time_quantile('Series', 1000, 'int', 1, 'higher')
+     2.34±0.04ms       2.72±0.1ms     1.16  rolling.Quantile.time_quantile('Series', 1000, 'float', 0, 'lower')
+      56.6±0.8ms         65.7±2ms     1.16  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 0.5, 'lower')
+     1.92±0.03ms       2.22±0.3ms     1.16  rolling.Quantile.time_quantile('Series', 1000, 'int', 0, 'linear')
+        57.6±2ms         66.7±3ms     1.16  rolling.Quantile.time_quantile('Series', 1000, 'int', 0.5, 'lower')
+     1.96±0.02ms      2.26±0.09ms     1.15  rolling.Quantile.time_quantile('DataFrame', 10, 'float', 1, 'linear')
+     2.41±0.02ms       2.78±0.1ms     1.15  rolling.Quantile.time_quantile('Series', 1000, 'float', 0, 'nearest')
+         116±2ms          133±1ms     1.15  rolling.VariableWindowMethods.time_rolling('DataFrame', '1d', 'float', 'median')
+      63.8±0.7ms         73.1±6ms     1.15  rolling.Quantile.time_quantile('Series', 1000, 'int', 0.5, 'linear')
+     2.23±0.03ms       2.56±0.3ms     1.15  rolling.Quantile.time_quantile('Series', 10, 'float', 1, 'midpoint')
+      64.5±0.9ms         73.8±2ms     1.14  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 0.5, 'midpoint')
+      50.8±0.4ms       58.0±0.7ms     1.14  rolling.Quantile.time_quantile('Series', 10, 'float', 0.5, 'lower')
+     1.81±0.03ms      2.06±0.08ms     1.14  rolling.Quantile.time_quantile('Series', 10, 'int', 0, 'nearest')
+      56.6±0.8ms         64.3±1ms     1.14  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 0.5, 'nearest')
+     1.63±0.02ms      1.85±0.04ms     1.14  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 0, 'midpoint')
+        77.3±1ms        87.8±10ms     1.14  rolling.Quantile.time_quantile('Series', 1000, 'float', 0.5, 'linear')
+     2.22±0.02ms      2.52±0.06ms     1.13  rolling.Quantile.time_quantile('Series', 10, 'float', 0, 'nearest')
+      65.0±0.5ms         73.7±4ms     1.13  rolling.Quantile.time_quantile('Series', 1000, 'int', 0.5, 'midpoint')
+     1.91±0.01ms      2.16±0.02ms     1.13  rolling.Quantile.time_quantile('DataFrame', 10, 'float', 0, 'nearest')
+      52.5±0.6ms         59.3±3ms     1.13  rolling.Quantile.time_quantile('Series', 10, 'float', 0.5, 'midpoint')
+      63.4±0.4ms         71.5±4ms     1.13  rolling.Quantile.time_quantile('DataFrame', 1000, 'int', 0.5, 'linear')
+     2.83±0.04ms       3.18±0.2ms     1.12  rolling.VariableWindowMethods.time_rolling('DataFrame', '1h', 'int', 'sum')
+     1.91±0.04ms      2.14±0.05ms     1.12  rolling.Quantile.time_quantile('Series', 1000, 'int', 1, 'nearest')
+      74.9±0.6ms         84.1±3ms     1.12  rolling.VariableWindowMethods.time_rolling('DataFrame', '1h', 'float', 'median')
+     4.43±0.02ms      4.96±0.09ms     1.12  rolling.VariableWindowMethods.time_rolling('DataFrame', '1d', 'float', 'skew')
+     1.53±0.06ms      1.71±0.03ms     1.12  rolling.Quantile.time_quantile('DataFrame', 10, 'int', 1, 'lower')
+     2.41±0.04ms      2.70±0.05ms     1.12  rolling.Quantile.time_quantile('Series', 1000, 'float', 0, 'linear')
+     2.23±0.03ms       2.50±0.2ms     1.12  rolling.Quantile.time_quantile('Series', 10, 'float', 1, 'nearest')
+     2.93±0.06ms       3.28±0.1ms     1.12  rolling.VariableWindowMethods.time_rolling('DataFrame', '1d', 'float', 'count')
+     2.42±0.03ms       2.71±0.1ms     1.12  rolling.Quantile.time_quantile('Series', 1000, 'float', 1, 'linear')
+     3.28±0.06ms      3.67±0.07ms     1.12  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'int', 'count')
+     2.99±0.03ms      3.35±0.05ms     1.12  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'float', 'mean')
+     2.23±0.02ms       2.49±0.2ms     1.12  rolling.Quantile.time_quantile('Series', 10, 'float', 1, 'lower')
+     2.43±0.05ms       2.71±0.1ms     1.12  rolling.Quantile.time_quantile('Series', 1000, 'float', 1, 'lower')
+     2.01±0.05ms      2.25±0.03ms     1.12  rolling.Quantile.time_quantile('DataFrame', 10, 'float', 1, 'lower')
+     4.63±0.06ms      5.17±0.07ms     1.12  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'int', 'max')
+     1.80±0.01ms      2.00±0.07ms     1.11  rolling.Quantile.time_quantile('Series', 10, 'int', 0, 'midpoint')
+     1.80±0.01ms      2.00±0.07ms     1.11  rolling.Quantile.time_quantile('Series', 10, 'int', 0, 'higher')
+     2.20±0.01ms       2.45±0.2ms     1.11  rolling.Quantile.time_quantile('Series', 10, 'float', 1, 'higher')
+     1.54±0.05ms      1.71±0.03ms     1.11  rolling.Quantile.time_quantile('DataFrame', 10, 'int', 1, 'higher')
+     2.96±0.06ms      3.29±0.05ms     1.11  rolling.VariableWindowMethods.time_rolling('Series', '1h', 'int', 'sum')
+     2.90±0.07ms      3.22±0.05ms     1.11  rolling.VariableWindowMethods.time_rolling('DataFrame', '50s', 'int', 'mean')
+     1.82±0.03ms      2.02±0.08ms     1.11  rolling.Quantile.time_quantile('Series', 10, 'int', 1, 'lower')
+     2.42±0.03ms      2.68±0.04ms     1.11  rolling.Quantile.time_quantile('Series', 1000, 'float', 1, 'higher')
+         107±1ms          119±3ms     1.11  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'int', 'median')
+      4.33±0.1ms       4.78±0.1ms     1.11  rolling.VariableWindowMethods.time_rolling('DataFrame', '1d', 'int', 'max')
+     4.37±0.07ms       4.82±0.1ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'int', 'min')
+     3.10±0.03ms      3.42±0.02ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'float', 'count')
+     3.04±0.01ms      3.35±0.06ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'float', 'mean')
+     4.58±0.04ms      5.05±0.08ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '1h', 'float', 'min')
+     2.82±0.03ms      3.11±0.04ms     1.10  rolling.VariableWindowMethods.time_rolling('DataFrame', '1d', 'float', 'mean')
+     3.08±0.04ms      3.40±0.06ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'int', 'mean')
+     2.22±0.06ms      2.45±0.08ms     1.10  rolling.Quantile.time_quantile('Series', 10, 'float', 0, 'lower')
+     3.07±0.02ms      3.38±0.05ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '1h', 'int', 'mean')
+         120±2ms          133±2ms     1.10  rolling.VariableWindowMethods.time_rolling('Series', '1d', 'float', 'median')
+     6.75±0.08ms       7.43±0.2ms     1.10  rolling.VariableWindowMethods.time_rolling('DataFrame', '1h', 'int', 'kurt')
+     1.99±0.03ms      2.20±0.01ms     1.10  rolling.Quantile.time_quantile('DataFrame', 10, 'float', 0, 'lower')
+     2.38±0.02ms      2.62±0.01ms     1.10  rolling.Quantile.time_quantile('Series', 1000, 'float', 1, 'nearest')

so that's not a great solution...

Is there a way to define a function (or macro?) that winds up as

if x == x

on non-windows machines, and as the more complicated if not isnan(x) when we detect MSVC 2017 is being used?

gfyoung · 2018-10-17T03:39:02Z

@TomAugspurger : Can't you leverage the pragma's used in cmath.h?

TomAugspurger · 2018-10-17T10:34:34Z

I'm not sure. And I'm not sure how to convince the compiler that it should be inlined.

TomAugspurger · 2018-10-17T10:49:05Z

Trying out 29f392c, but I don't know C / C++ so thoughts from others would be appreciated.

I'm running ASV over the rolling benchmarks now.

TomAugspurger · 2018-10-17T11:42:19Z

A couple questions:

I see some new warnings in the build logs

2018-10-17T11:02:00.9761682Z     pandas\_libs/window.cpp(9522) : warning C4244: 'argument' : conversion from '__pyx_t_5numpy_int64_t' to 'int', possible loss of data
2018-10-17T11:02:00.9761814Z     pandas\_libs/window.cpp(16928) : warning C4244: '=' : conversion from 'double' to '__pyx_t_5numpy_float32_t', possible loss of data

I suppose it's fine, but should I explicitly cast the int to float first?

TomAugspurger · 2018-10-17T11:42:44Z

ASV with this version looks a bit better maybe. Re-running those slow ones, to see if it's just noise.

       before           after         ratio
     [b5313f92]       [29f392c4]
     <fix-windows-ci~1>       <fix-windows-ci>
+     3.01±0.04ms         4.69±2ms     1.56  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'int', 'sum')
+     4.69±0.03ms         6.47±2ms     1.38  rolling.VariableWindowMethods.time_rolling('Series', '50s', 'float', 'max')
+      18.8±0.2ms       21.8±0.8ms     1.16  rolling.VariableWindowMethods.time_rolling('DataFrame', '1h', 'int', 'std')

h-vetinari · 2018-10-17T11:32:32Z

ci/azure-windows-36.yaml

  - fastparquet
  - feather-format
  - matplotlib
  - numexpr
  - numpy=1.14*
  - openpyxl=2.5.5
+  - parquet-cpp
  - pyarrow
  - pytables


Could you try vc=14.0 (without the pragma stuff)? the last working build had that, while the newer ones have 14.1. Plus, the notna definition still didn't fix the errors in the WinPy36 build.

the notna definition still didn't fix the errors in the WinPy36 build.

29f392c had a bug (not isn't valid C++).

I think this turned up a real bug that's hitting users / will hit users, depending on how things are compiled. Probably best to fix it.

TomAugspurger · 2018-10-17T11:57:54Z

No changes on asv continuous c512f4200..HEAD -b rolling.VariableWindowMethods.time_rolling, so I think things are being inlined now hopefully (at least on linux / Mac).

TomAugspurger · 2018-10-17T12:07:58Z

I think the MSVC versions this is being applied to is incorrect.

The CI failure is on Version: VisualStudio/15.8.5+28010.2036. According to https://stackoverflow.com/a/70630/1889400, that's _MSVC_VER == 1915.

TomAugspurger · 2018-10-17T12:14:01Z

I guess I had a slight different fix in mind than @chris-b1. I'd like to replace all uses of if val == val with if notnan(val).

On the versions affected by the bug, (MSVC 1910 through 1915?), we would need that to be not isnan(val).

On the versions not affected by the bug (other platforms, older MSVC, and in a future patched MSVC), that would be just {return x == x;}

So I'm thinking

#if defined(_MSC_VER) && (_MSC_VER >= 1900) && (_MSC_VER < 2000)
#include <cmath>
namespace std {
  __inline int isnan(double x) { return _isnan(x); }
  __inline int signbit(double num) { return _copysign(1.0, num) < 0; }
  __inline int notnan(double x) { return not isnan(x); }
}
#else
#include <cmath>

namespace std {
  __inline int notnan(double x) { return x == x; }
}

#endif
#endif

I'm not sure we even need the isnan and isgnbit anymore, since IIUC, MSVC 2017 has those defined already?

TomAugspurger · 2018-10-17T12:22:43Z

0b7dc84 attempts that.

Reminder that I don't know what I'm doing here, and I don't have a windows machine, so this should get a close look from the experts :)

TomAugspurger · 2018-10-17T13:25:20Z

Latest tries to work around

2018-10-17T13:14:47.3917605Z     d:\a\1\s\pandas\_libs\src/headers/cmath(11): error C2883: 'std::signbit': function declaration conflicts with 'signbit' introduced by using-declaration
2018-10-17T13:14:47.3917664Z     C:\Program Files (x86)\Windows Kits\10\include\10.0.17134.0\ucrt\corecrt_math.h(310): note: see declaration of 'signbit'

TomAugspurger · 2018-10-17T13:56:08Z

The azure 36 build passed. @chris-b1 do you have time today to take a look at the changes?

chris-b1 · 2018-10-17T17:26:38Z

Thanks for grinding through this, I'll have time to look later this afternoon

…

On Wed, Oct 17, 2018, 8:57 AM Tom Augspurger ***@***.***> wrote: The azure 36 build passed. @chris-b1 <https://github.com/chris-b1> do you have time today to take a look at the changes? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1b_HtfUL7ujKuDrK8RFmc8y8qZ-Pugks5ulzcwgaJpZM4XeZIq> .

chris-b1

Looks good to me, I'll open a f/u issue for the MSVC bug

gfyoung

Nice!

cc @jreback

TomAugspurger · 2018-10-18T01:46:24Z

Merging in a couple hours if no objections (cc @jreback if you haven't had a look yet)

jreback · 2018-10-18T01:57:12Z

this was fine

TomAugspurger added 2 commits October 16, 2018 08:44

Try removing feather

f55a827

pin pyarrow

4097bdc

TomAugspurger added the CI Continuous Integration label Oct 16, 2018

h-vetinari reviewed Oct 16, 2018

View reviewed changes

TomAugspurger added 4 commits October 16, 2018 09:19

Revert "Try removing feather"

40dafbd

This reverts commit f55a827.

pin boost-cpp

0c1e34e

Merge remote-tracking branch 'upstream/master' into fix-windows-ci

a508c2e

remove arrow pin

7ef69bd

test isnan

c512f42

added parquet-cpp

b5313f9

This was referenced Oct 16, 2018

DEPR/CLN: Clean up to_dense signature deprecations #22910

Merged

BUG: Don't parse NaN as 'nan' in Data IO #23162

Merged

jreback added this to the 0.24.0 milestone Oct 17, 2018

Try defining notnan

29f392c

fixup

d4c23f7

h-vetinari reviewed Oct 17, 2018

View reviewed changes

Define notnan, apply change to just MSVC 2017

0b7dc84

TomAugspurger added 4 commits October 17, 2018 07:23

not to !

76c4d06

define isnan, signbit

e6fd311

Merge remote-tracking branch 'upstream/master' into fix-windows-ci

aaa1afc

try again

6af3461

chris-b1 approved these changes Oct 17, 2018

View reviewed changes

chris-b1 mentioned this pull request Oct 17, 2018

PERF: replace use of isnan once MSVC bug is fixed #23209

Closed

gfyoung approved these changes Oct 17, 2018

View reviewed changes

gfyoung mentioned this pull request Oct 17, 2018

TST/CLN: Parameterize test in sparse/test_indexing #23213

Merged

TomAugspurger merged commit 1546c35 into pandas-dev:master Oct 18, 2018

TomAugspurger deleted the fix-windows-ci branch October 18, 2018 02:06

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

Windows CI (pandas-dev#23182)

bc2012a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows CI #23182

Windows CI #23182

TomAugspurger commented Oct 16, 2018

TomAugspurger commented Oct 16, 2018

h-vetinari Oct 16, 2018

TomAugspurger Oct 16, 2018

TomAugspurger Oct 16, 2018

TomAugspurger Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 16, 2018

codecov bot commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 16, 2018

h-vetinari commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 16, 2018

h-vetinari commented Oct 16, 2018

TomAugspurger commented Oct 16, 2018

chris-b1 commented Oct 16, 2018 via email

TomAugspurger commented Oct 16, 2018

TomAugspurger commented Oct 17, 2018 •

edited

Loading

gfyoung commented Oct 17, 2018 •

edited

Loading

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018 •

edited

Loading

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

h-vetinari Oct 17, 2018

TomAugspurger Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

chris-b1 commented Oct 17, 2018 via email

chris-b1 left a comment

gfyoung left a comment

TomAugspurger commented Oct 18, 2018

jreback commented Oct 18, 2018

Windows CI #23182

Windows CI #23182

Conversation

TomAugspurger commented Oct 16, 2018

TomAugspurger commented Oct 16, 2018

h-vetinari Oct 16, 2018

Choose a reason for hiding this comment

TomAugspurger Oct 16, 2018

Choose a reason for hiding this comment

TomAugspurger Oct 16, 2018

Choose a reason for hiding this comment

TomAugspurger Oct 16, 2018 • edited Loading

Choose a reason for hiding this comment

TomAugspurger commented Oct 16, 2018

codecov bot commented Oct 16, 2018 • edited Loading

Codecov Report

TomAugspurger commented Oct 16, 2018

h-vetinari commented Oct 16, 2018 • edited Loading

TomAugspurger commented Oct 16, 2018 • edited Loading

TomAugspurger commented Oct 16, 2018

h-vetinari commented Oct 16, 2018

TomAugspurger commented Oct 16, 2018

chris-b1 commented Oct 16, 2018 via email

TomAugspurger commented Oct 16, 2018

TomAugspurger commented Oct 17, 2018 • edited Loading

gfyoung commented Oct 17, 2018 • edited Loading

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018 • edited Loading

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

h-vetinari Oct 17, 2018

Choose a reason for hiding this comment

TomAugspurger Oct 17, 2018

Choose a reason for hiding this comment

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

TomAugspurger commented Oct 17, 2018

chris-b1 commented Oct 17, 2018 via email

chris-b1 left a comment

Choose a reason for hiding this comment

gfyoung left a comment

Choose a reason for hiding this comment

TomAugspurger commented Oct 18, 2018

jreback commented Oct 18, 2018

TomAugspurger Oct 16, 2018 •

edited

Loading

codecov bot commented Oct 16, 2018 •

edited

Loading

h-vetinari commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 16, 2018 •

edited

Loading

TomAugspurger commented Oct 17, 2018 •

edited

Loading

gfyoung commented Oct 17, 2018 •

edited

Loading

TomAugspurger commented Oct 17, 2018 •

edited

Loading