Skip to content

BUG: repair 'style' kwd handling in DataFrame.plot (#21003) #33821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Sep 5, 2020
Merged

BUG: repair 'style' kwd handling in DataFrame.plot (#21003) #33821

merged 20 commits into from
Sep 5, 2020

Conversation

joooeey
Copy link
Contributor

@joooeey joooeey commented Apr 27, 2020

@pep8speaks
Copy link

pep8speaks commented Apr 27, 2020

Hello @joooeey! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-09-05 10:38:05 UTC

@joooeey
Copy link
Contributor Author

joooeey commented Apr 27, 2020

Looks like there's one issue left but I have no idea how to resolve it. pytest pandas yields the lengthy Access Violation error given below at 70% progress.

Does anyone know what's going on there?

Anaconda prompt output

pandas\tests\io\excel\test_readers.py ........ss........ss........ss.... [ 69%]
............................................ssssssssssss................ [ 69%]
ssss........ss........ss........ss........ss........ss................ss [ 69%]
ss........ss........ss........ss........ss........ss........ss........ss [ 69%]
........ss................ssss........ss........ss........ss........ss.. [ 69%]
......ss........ss...xx...ss...xx...ss........ss........ss........ssssss [ 70%]
ssssss........ss...Windows fatal exception: access violation

Thread 0x000007b8 (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 470 in _handle_results
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Thread 0x00002340 (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 422 in _handle_tasks
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Thread 0x00003404 (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 413 in _handle_workers
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Thread 0x00003c08 (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 110 in worker
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Thread 0x00003920 (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 110 in worker
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Thread 0x0000168c (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 110 in worker
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Thread 0x00000450 (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 110 in worker
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 926 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\lib\threading.py", line 890 in _bootstrap

Current thread 0x000019cc (most recent call first):
  File "C:\ProgramData\Anaconda3\lib\site-packages\psutil\_pswindows.py", line 993 in open_files
  File "C:\ProgramData\Anaconda3\lib\site-packages\psutil\_pswindows.py", line 679 in wrapper
  File "C:\ProgramData\Anaconda3\lib\site-packages\psutil\__init__.py", line 1154 in open_files
  File "C:\Users\Joey\pandas-joooeey\pandas\tests\io\excel\conftest.py", line 62 in check_for_file_leaks
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 788 in call_fixture_func
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 964 in pytest_fixture_setup
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 87 in <lambda>
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 914 in execute
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 584 in _compute_fixture_value
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 503 in _get_active_fixturedef
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 487 in getfixturevalue
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 477 in _fillfixtures
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\fixtures.py", line 297 in fillfixtures
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\python.py", line 1485 in setup
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 373 in prepare
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 123 in pytest_runtest_setup
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 87 in <lambda>
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 217 in <lambda>
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 244 in from_call
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 217 in call_runtest_hook
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 186 in call_and_report
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 94 in runtestprotocol
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\runner.py", line 85 in pytest_runtest_protocol
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 87 in <lambda>
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\main.py", line 272 in pytest_runtestloop
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 87 in <lambda>
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\main.py", line 247 in _main
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\main.py", line 191 in wrap_session
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 87 in <lambda>
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\lib\site-packages\_pytest\config\__init__.py", line 125 in main
  File "C:\ProgramData\Anaconda3\Scripts\pytest-script.py", line 10 in <module>

(base) C:\Users\Joey\pandas-joooeey>

joooeey added 2 commits April 28, 2020 09:32
manually made edits suggested by `black`.
Copy link
Member

@MarcoGorelli MarcoGorelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joooeey !

I'll just make a few comments ahead of core members' reviews

@joooeey
Copy link
Contributor Author

joooeey commented Apr 30, 2020

This is odd. CI reports two test failures in pandas.tests.series:

  • .test_constructors.TestSeriesConstructors.test_constructor_generic_timestamp_no_frequency
  • .test_dtypes.TestSeriesDtypes.test_astype_generic_timestamp_no_frequency

These tests run fine on my machine.

@MarcoGorelli
Copy link
Member

CI pipeline is currently broken, the errors were probably unrelated

@joooeey joooeey requested a review from MarcoGorelli May 9, 2020 15:51
Copy link
Member

@MarcoGorelli MarcoGorelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joooeey - nice work!

So, as far as I can tell, the problem was that before, with style='d', this line

        nocolor_style = style is None or re.match("[a-z]+", style) is None

wasn't setting nocolor_style to True, which is necessary to reach

            if isinstance(colors, dict):
                kwds["color"] = colors[label]
            else:
                kwds["color"] = colors[col_num % len(colors)]

@jreback jreback added the Visualization plotting label May 9, 2020
Copy link
Member

@charlesdong1991 charlesdong1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, and I think it's very close. @joooeey

Since the PR is to repair the style keyword handling which essentially sets the marker for plot. Therefore, it is necessary to also test marker is still set correctly. As mentioned in test, you could use ax.lines.get_marker() == 'd' to achieve it.

And personally I think you could also remove some of your comments in the tests because most of those codes in the tests, they are very straightforward thus there is no need to explain IMHO.

# GH21003 - code sample from 2018-05-10 is equivalent to this test
df = DataFrame(np.random.random((7, 4)))
# combining a color string and a marker letter should be allowed
df.plot(color="green", style="d") # d for diamond
Copy link
Member

@charlesdong1991 charlesdong1991 May 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we also check color and style here?
something like:

ax = df.plot(color="green", style="d")
assert ax.lines[0].get_color() == 'green'
assert ax.lines[0].get_marker() == 'd'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my reply below.

# in a plot where each individual line was assigned a list of colors:
# ax.lines[i].get_color() == ['yellow', 'red', 'green', 'blue']
# which resulted in a ValueError when plt.draw() was called.
assert [line.get_color() for line in ax.lines] == color_list
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, also use ax.lines.get_marker() to verify that marker is set correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! This will introduce lots of duplicate code between the tests test_color_and_style_arguments, test_color_and_marker and test_color_list_and_marker. Namely df.plot and the list comprehensions. I'll see how I can refactor this.

Copy link
Contributor Author

@joooeey joooeey May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it more readable to test these four combinations of color and style arguments (test_color_and_style_arguments has two) in four separate test methods or all in one? I'd go for the former but I've noticed lots of huge test methods in frame.py so I'm not sure.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this will work in one test?

assert [line.get_color() for line in ax.lines] == color_list
assert all(line.get_marker() == 'd' for line in ax.lines)

Copy link
Member

@charlesdong1991 charlesdong1991 May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, i would prefer to combine both of your tests, something like this

@pytest.mark.parametrize("color, output_color_list", 
   [('green', ['green'] * 4), 
    (['yellow', 'red', 'green', 'blue'], ['yellow', 'red', 'green', 'blue'])
])
def test_color_and_marker(self, color, output_color_list):
     df = DataFrame(np.random.random((7, 4)))
     ax = df.plot(color=color, style="d")
     assert [line.get_color() for line in ax.lines] == output_color_list
     assert all(line.get_marker() == 'd' for line in ax.lines)

And regarding test_color_and_style_arguments, it's kinda out of the scope of your PR, and you can just leave it aside.
@joooeey does this sound good to you?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, import pandas results in this error:

import pandas
Traceback (most recent call last):

  File "<ipython-input-1-38d4b0363d82>", line 1, in <module>
    import pandas

  File "C:\Users\Joey\pandas-joooeey\pandas\__init__.py", line 52, in <module>
    from pandas.core.api import (

  File "C:\Users\Joey\pandas-joooeey\pandas\core\api.py", line 15, in <module>
    from pandas.core.arrays import Categorical

  File "C:\Users\Joey\pandas-joooeey\pandas\core\arrays\__init__.py", line 8, in <module>
    from pandas.core.arrays.datetimes import DatetimeArray

  File "C:\Users\Joey\pandas-joooeey\pandas\core\arrays\datetimes.py", line 46, in <module>
    from pandas.core.arrays import datetimelike as dtl

  File "C:\Users\Joey\pandas-joooeey\pandas\core\arrays\datetimelike.py", line 11, in <module>
    from pandas._libs.tslibs.timestamps import (

ImportError: cannot import name 'integer_op_not_supported' from 'pandas._libs.tslibs.timestamps' (C:\Users\Joey\pandas-joooeey\pandas\_libs\tslibs\timestamps.cp37-win_amd64.pyd)

Copy link
Member

@charlesdong1991 charlesdong1991 May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so it's jupyter notebook, have you tried to shut down the notebook and restart? and then import?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is the IPython Console in Spyder. Restarting Spyder didn't help.

Copy link
Contributor Author

@joooeey joooeey May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest pandas fails too with Windows fatal exception: access violation:

Anaconda prompt stack trace

[...]
pandas\tests\arrays\test_numpy.py ...................................................                                                       [ 15%]
pandas\tests\arrays\test_period.py ..................................................Windows fatal exception: access violation

Thread 0x00001f80 (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 576 in _handle_results
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Thread 0x00004e4c (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 528 in _handle_tasks
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Thread 0x00001fe8 (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\connection.py", line 810 in _exhaustive_wait
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\connection.py", line 878 in wait
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 499 in _wait_for_updates
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 519 in _handle_workers
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Thread 0x000028a8 (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 114 in worker
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Thread 0x000047c0 (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 114 in worker
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Thread 0x00004cb8 (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 114 in worker
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Thread 0x0000559c (most recent call first):
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 114 in worker
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 870 in run
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 932 in _bootstrap_inner
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\threading.py", line 890 in _bootstrap

Current thread 0x000014a0 (most recent call first):
  File "C:\Users\Joey\pandas-joooeey\pandas\tests\arrays\test_period.py", line 413 in test_arrow_table_roundtrip
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\python.py", line 184 in pytest_pyfunc_call
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\python.py", line 1479 in runtest
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 135 in pytest_runtest_call
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 217 in <lambda>
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 244 in from_call
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 216 in call_runtest_hook
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 186 in call_and_report
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 100 in runtestprotocol
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\runner.py", line 85 in pytest_runtest_protocol
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 272 in pytest_runtestloop
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 247 in _main
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 191 in wrap_session
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\_pytest\config\__init__.py", line 124 in main
  File "C:\ProgramData\Anaconda3\envs\pandas-dev\Scripts\pytest-script.py", line 10 in <module>

(pandas-dev) c:\Users\Joey\pandas-joooeey>

I've seen this error before, mentioned on April 27 in this thread but I don't remember how I resolved it.

Copy link
Member

@charlesdong1991 charlesdong1991 May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

emm, have no idea, i really think

python setup.py build_ext --inplace -j 4
python -m pip install -e . --no-build-isolation --no-use-pep517

would solve the issue if you already merge the master

@WillAyd could you help with the environment issue?

@WillAyd
Copy link
Member

WillAyd commented May 19, 2020 via email

@joooeey
Copy link
Contributor Author

joooeey commented May 20, 2020

I still can't even import pandas. see #34280 .

@joooeey
Copy link
Contributor Author

joooeey commented Jul 10, 2020

@joooeey it looks like you're running pytest from the base environment. Can you do conda activate pandas-dev first?

Thanks, good catch! Activating pandas-dev helped a little. Now the test suite fails at 22% with a stack overflow: #35218

@joooeey
Copy link
Contributor Author

joooeey commented Jul 10, 2020

For now I don't intend to bother with Docker or WSL as the instructions you linked to are too vague for me to follow. I'm really not keen on more experiments. Or how would I set up the Docker container? Would that make my dev environment more stable?

@MarcoGorelli
Copy link
Member

You probably don't need to run all the tests, just the plotting ones

pytest pandas/tests/plotting

@joooeey
Copy link
Contributor Author

joooeey commented Jul 10, 2020

You probably don't need to run all the tests, just the plotting ones

pytest pandas/tests/plotting

Running pytest pandas/tests/plotting on my local master branch results in two test failures which I don't understand where they come from:

(pandas-dev) C:\Users\Joey\pandas-joooeey>pytest pandas/tests/plotting
======================================= test session starts =======================================
platform win32 -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\Joey\pandas-joooeey, inifile: setup.cfg
plugins: hypothesis-5.15.0, asyncio-0.12.0, cov-2.8.1, forked-1.1.2, xdist-1.32.0
collected 418 items

pandas\tests\plotting\test_backend.py ...F.s.                                                [  1%]
pandas\tests\plotting\test_boxplot_method.py .........................                       [  7%]
pandas\tests\plotting\test_converter.py ............................                         [ 14%]
pandas\tests\plotting\test_datetimelike.py F................................................ [ 26%]
.......................................................x.                                    [ 39%]
pandas\tests\plotting\test_frame.py .........................x.............................. [ 53%]
.................................................................x.......................... [ 75%]
....                                                                                         [ 76%]
pandas\tests\plotting\test_groupby.py .....                                                  [ 77%]
pandas\tests\plotting\test_hist_method.py .....................                              [ 82%]
pandas\tests\plotting\test_misc.py s...............                                          [ 86%]
pandas\tests\plotting\test_series.py ....................................................x.. [ 99%]
...                                                                                          [100%]

============================================ FAILURES =============================================
____________________________________ test_register_entrypoint _____________________________________

restore_backend = None

    @td.skip_if_no_mpl
    def test_register_entrypoint(restore_backend):

>       dist = pkg_resources.get_distribution("pandas")

pandas\tests\plotting\test_backend.py:52:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pkg_resources\__init__.py:482: in get_distribution
    dist = get_provider(dist)
C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pkg_resources\__init__.py:358: in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pkg_resources\__init__.py:901: in require
    needed = self.resolve(parse_requirements(requirements))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pkg_resources.WorkingSet object at 0x000001E7D20E0D90>, requirements = []
env = <pkg_resources.Environment object at 0x000001E7D4D64BB0>, installer = None
replace_conflicting = False, extras = None

    def resolve(self, requirements, env=None, installer=None,
                replace_conflicting=False, extras=None):
        """List all distributions needed to (recursively) meet `requirements`

        `requirements` must be a sequence of ``Requirement`` objects.  `env`,
        if supplied, should be an ``Environment`` instance.  If
        not supplied, it defaults to all distributions available within any
        entry or distribution in the working set.  `installer`, if supplied,
        will be invoked with each requirement that cannot be met by an
        already-installed distribution; it should return a ``Distribution`` or
        ``None``.

        Unless `replace_conflicting=True`, raises a VersionConflict exception
        if
        any requirements are found on the path that have the correct name but
        the wrong version.  Otherwise, if an `installer` is supplied it will be
        invoked to obtain the correct version of the requirement and activate
        it.

        `extras` is a list of the extras to be used with these requirements.
        This is important because extra requirements may look like `my_req;
        extra = "my_extra"`, which would otherwise be interpreted as a purely
        optional requirement.  Instead, we want to be able to assert that these
        requirements are truly required.
        """

        # set up the stack
        requirements = list(requirements)[::-1]
        # set of processed requirements
        processed = {}
        # key -> dist
        best = {}
        to_activate = []

        req_extras = _ReqExtras()

        # Mapping of requirement to set of distributions that required it;
        # useful for reporting info about conflicts.
        required_by = collections.defaultdict(set)

        while requirements:
            # process dependencies breadth-first
            req = requirements.pop(0)
            if req in processed:
                # Ignore cyclic or redundant dependencies
                continue

            if not req_extras.markers_pass(req, extras):
                continue

            dist = best.get(req.key)
            if dist is None:
                # Find the best distribution and add it to the map
                dist = self.by_key.get(req.key)
                if dist is None or (dist not in req and replace_conflicting):
                    ws = self
                    if env is None:
                        if dist is None:
                            env = Environment(self.entries)
                        else:
                            # Use an empty environment and workingset to avoid
                            # any further conflicts with the conflicting
                            # distribution
                            env = Environment([])
                            ws = WorkingSet([])
                    dist = best[req.key] = env.best_match(
                        req, ws, installer,
                        replace_conflicting=replace_conflicting
                    )
                    if dist is None:
                        requirers = required_by.get(req, None)
>                       raise DistributionNotFound(req, requirers)
E                       pkg_resources.DistributionNotFound: The 'pandas' distribution was not found and is required by the application

C:\ProgramData\Anaconda3\envs\pandas-dev\lib\site-packages\pkg_resources\__init__.py:787: DistributionNotFound
_____________________________ TestTSPlot.test_ts_plot_with_tz['UTC'] ______________________________

self = <pandas.tests.plotting.test_datetimelike.TestTSPlot object at 0x000001E7DCE814C0>
tz_aware_fixture = 'UTC'

    @pytest.mark.slow
    def test_ts_plot_with_tz(self, tz_aware_fixture):
        # GH2877, GH17173, GH31205, GH31580
        tz = tz_aware_fixture
        index = date_range("1/1/2011", periods=2, freq="H", tz=tz)
        ts = Series([188.5, 328.25], index=index)
        with tm.assert_produces_warning(None):
            _check_plot_works(ts.plot)
            ax = ts.plot()
            xdata = list(ax.get_lines())[0].get_xdata()
            # Check first and last points' labels are correct
>           assert (xdata[0].hour, xdata[0].minute) == (0, 0)
E           AttributeError: 'numpy.datetime64' object has no attribute 'hour'

pandas\tests\plotting\test_datetimelike.py:57: AttributeError
======================================== warnings summary =========================================
pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_mpl2_color_cycle_str
  C:\Users\Joey\pandas-joooeey\pandas\plotting\_matplotlib\style.py:64: MatplotlibDeprecationWarning: Support for uppercase single-letter colors is deprecated since Matplotlib 3.1 and will be removed in 3.3; please use lowercase instead.
    [conv.to_rgba(c) for c in colors]

pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_errorbar_plot
pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_errorbar_plot
pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_errorbar_plot
pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_errorbar_timeseries
pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_errorbar_timeseries
pandas/tests/plotting/test_frame.py::TestDataFramePlots::test_errorbar_timeseries
  C:\Users\Joey\pandas-joooeey\pandas\plotting\_matplotlib\__init__.py:61: UserWarning: To output multiple subplots, the figure containing the passed axes is being cleared
    plot_obj.generate()

pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_series.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_series.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_series.py::TestSeriesPlots::test_hist_legacy
  C:\Users\Joey\pandas-joooeey\pandas\tests\plotting\common.py:535: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
    kwargs.get("ax", fig.add_subplot(211))

pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_series.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_series.py::TestSeriesPlots::test_hist_legacy
pandas/tests/plotting/test_series.py::TestSeriesPlots::test_hist_legacy
  C:\Users\Joey\pandas-joooeey\pandas\tests\plotting\common.py:543: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
    kwargs["ax"] = fig.add_subplot(212)

pandas/tests/plotting/test_hist_method.py::TestDataFramePlots::test_tight_layout
pandas/tests/plotting/test_hist_method.py::TestDataFramePlots::test_hist_column_order_unchanged[None-expected0]
pandas/tests/plotting/test_hist_method.py::TestDataFramePlots::test_hist_column_order_unchanged[column1-expected1]
  C:\Users\Joey\pandas-joooeey\pandas\tests\plotting\common.py:545: UserWarning: To output multiple subplots, the figure containing the passed axes is being cleared
    ret = f(**kwargs)

pandas/tests/plotting/test_hist_method.py::TestDataFramePlots::test_hist_subplot_xrot
  C:\Users\Joey\pandas-joooeey\pandas\plotting\_matplotlib\hist.py:364: UserWarning: To output multiple subplots, the figure containing the passed axes is being cleared
    axes = _grouped_hist(

pandas/tests/plotting/test_misc.py::TestSeriesPlots::test_autocorrelation_plot
pandas/tests/plotting/test_misc.py::TestSeriesPlots::test_autocorrelation_plot
  C:\Users\Joey\pandas-joooeey\pandas\plotting\_matplotlib\misc.py:410: UserWarning: Requested projection is different from current axis projection, creating new axis with requested projection.
    ax = plt.gca(xlim=(1, n), ylim=(-1.0, 1.0))

pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_andrews_curves
  C:\Users\Joey\pandas-joooeey\pandas\plotting\_matplotlib\misc.py:241: UserWarning: Requested projection is different from current axis projection, creating new axis with requested projection.
    ax = plt.gca(xlim=(-np.pi, np.pi))

pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_radviz
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_radviz
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_radviz
pandas/tests/plotting/test_misc.py::TestDataFramePlots::test_radviz
  C:\Users\Joey\pandas-joooeey\pandas\plotting\_matplotlib\misc.py:131: UserWarning: Requested projection is different from current axis projection, creating new axis with requested projection.
    ax = plt.gca(xlim=[-1, 1], ylim=[-1, 1])

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===================================== short test summary info =====================================
FAILED pandas/tests/plotting/test_backend.py::test_register_entrypoint - pkg_resources.Distributi...
FAILED pandas/tests/plotting/test_datetimelike.py::TestTSPlot::test_ts_plot_with_tz['UTC'] - Attr...
========== 2 failed, 410 passed, 2 skipped, 4 xfailed, 37 warnings in 223.73s (0:03:43) ===========

(pandas-dev) C:\Users\Joey\pandas-joooeey>

If everything is set up right, there should never be test failures on master, no?

@MarcoGorelli
Copy link
Member

MarcoGorelli commented Jul 10, 2020

Unfortunately sometimes it can happen. Pulling the latest commits from upstream master can sometimes fix it if it's been solved upstream

@charlesdong1991
Copy link
Member

hey @joooeey

I think if you rebase and resolve the conflict, and solve the linting error shown in CI, then it's fine, because seems the change of your PR doesn't seem to cause other tests to fail(the error you showed seems unrelated to your change, but related to your development environment probably).

@joooeey
Copy link
Contributor Author

joooeey commented Jul 11, 2020

hey @joooeey

I think if you rebase and resolve the conflict, and solve the linting error shown in CI, then it's fine, because seems the change of your PR doesn't seem to cause other tests to fail(the error you showed seems unrelated to your change, but related to your development environment probably).

What's the command sequence for rebasing? Do I start with git checkout master or git checkout plot-style-error?

@MarcoGorelli
Copy link
Member

Assuming your local branch is up-to-date with origin/plot-style-error, I'd advise:

git checkout plot-style-error
git fetch upstream master
git merge upstream/master  # you'll need to fix some conflicts here
git commit

@joooeey
Copy link
Contributor Author

joooeey commented Jul 11, 2020

Assuming your local branch is up-to-date with origin/plot-style-error, I'd advise:

git checkout plot-style-error
git fetch upstream master
git merge upstream/master  # you'll need to fix some conflicts here
git commit

I did that and it helped quite a bit. In the Travis CI there's still one error which I can't comprehend.

@MarcoGorelli
Copy link
Member

Thanks @joooeey ! The Travis CI error is unrelated

@joooeey
Copy link
Contributor Author

joooeey commented Jul 12, 2020

Thanks @joooeey ! The Travis CI error is unrelated

@jreback can you please have another look at this PR? I incorporated your requested change a few months ago -- there are no more staticmethods in the code of this PR. Can you please mark that change request done? Is this PR finally ready to merge?

@jreback
Copy link
Contributor

jreback commented Jul 12, 2020

@joooeey someone will have a look

we have hundreds of PRs so things do take time

Comment on lines 214 to 217
green_line = ax.lines[2]
assert green_line.get_color() == "green"
assert green_line.get_marker() == "d"
assert green_line.get_linestyle() == "--"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since you might need another rebase, just one minor comment, as mentioned in #33821 (comment)

I still prefer to not test only green line but all lines using parametrization, i had an example there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since you might need another rebase, just one minor comment, as mentioned in #33821 (comment)

Why do I need another rebase? I just merged upstream/master 6 days ago which unfortunately left one CI issue. I don't think another rebase will solve that?

I still prefer to not test only green line but all lines using parametrization, i had an example there

Well the function is parametrized I'm testing the inputs color="green" and color=["yellow", "red", "green", "blue"] . The only thing is, to simplify the code, I'm only checking the third line, the one that is green for both inputs in the parametrize. I'm confident that this is enough. Checking all four lines would be overkill. After all, checking all four lines (as opposed to the third) would only guard against absurd bugs such as if we suddenly had code that ignored the color kwarg and made every line green.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just merged upstream/master 6 days ago which unfortunately left one CI issue. I don't think another rebase will solve that?

Might do if upstream/master has changed

Regarding the test - unfortunately, absurd bugs do sometimes happen. FWIW, I think the suggestion to check all lines' colours is sensible

@simonjayhawkins
Copy link
Member

@joooeey can you move release note to 1.2

@MarcoGorelli
Copy link
Member

Hi @joooeey - is this still active? If you address the latest comments I think this is close to being ready to be merged

@MarcoGorelli MarcoGorelli requested review from charlesdong1991 and removed request for jreback September 5, 2020 10:39
@jreback jreback added this to the 1.2 milestone Sep 5, 2020
@jreback jreback merged commit 6fa1a45 into pandas-dev:master Sep 5, 2020
@jreback
Copy link
Contributor

jreback commented Sep 5, 2020

thanks @joooeey very nice. thanks @MarcoGorelli and @charlesdong1991 for reviewing.

@joooeey
Copy link
Contributor Author

joooeey commented Sep 22, 2020

Thanks guys for pulling this to completion! I couldn't spare any time to work on this this summer unfortunately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pandas plotting raises ValueError on style strings that should be valid according to spec.
7 participants