BUG: pathlib.Path in io #16292

chris-b1 · 2017-05-09T00:43:52Z

closes Regression in 0.20: IO functions can't handle pathlib.Path objects anymore #16291, closes pd.read_pickle, pd.DataFrame.to_pickle do not accept pathlib.Path objects #14932
tests added / passed
passes git diff upstream/master --name-only -- '*.py' | flake8 --diff
whatsnew entry

jreback · 2017-05-09T00:45:30Z

pandas/tests/io/test_pickle.py

@@ -299,6 +300,15 @@ def test_pickle_v0_15_2():
    tm.assert_categorical_equal(cat, pd.read_pickle(pickle_path))


+def test_pickle_path_pathlib():
+    tm._skip_if_no_pathlib()


instead of making a specific test can u add a general routine like we have for round_trip_pickle

and then use it here

ideally add tests for all of the io functions (and xfail id they don't work)

and test for localpath as well

Good point, let me know if what I just pushed is what you had in mind and I'll add to other io functions

yep exactly like this!

maybe put a little docs in the testing file above these 3 tests routines.

jreback · 2017-05-09T01:08:47Z

pandas/tests/io/test_pickle.py

+
+
+def test_pickle_path_localpath():
+    tm._skip_if_no_localpath()


I would move the skip routines to the actual test functions themselves (in util/testing)

ignore this, you can just use a skip importer below.

jreback · 2017-05-09T01:09:04Z

pandas/util/testing.py

+
+
+def round_trip_localpath(writer, reader, path=None):
+    from py.path import local as LocalPath


e.g. here.

and you need an pytimportor.skip here (for both of these).

codecov · 2017-05-09T02:04:50Z

Codecov Report

Merging #16292 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #16292      +/-   ##
==========================================
- Coverage   90.39%   90.37%   -0.02%     
==========================================
  Files         161      161              
  Lines       50863    50880      +17     
==========================================
+ Hits        45977    45985       +8     
- Misses       4886     4895       +9

Flag	Coverage Δ
#multiple	`88.16% <100%> (ø)`	⬆️
#single	`40.31% <11.76%> (-0.12%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/common.py	`70.21% <100%> (+0.12%)`	⬆️
pandas/util/testing.py	`80.71% <100%> (+0.3%)`	⬆️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/core/frame.py	`97.59% <0%> (-0.1%)`	⬇️
pandas/core/indexes/datetimes.py	`95.23% <0%> (-0.1%)`	⬇️
pandas/core/common.py	`91.03% <0%> (+0.34%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80abd97...ae22641. Read the comment docs.

codecov · 2017-05-09T02:05:04Z

Codecov Report

Merging #16292 into master will decrease coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #16292      +/-   ##
==========================================
- Coverage   90.39%   90.36%   -0.04%     
==========================================
  Files         161      161              
  Lines       50863    50885      +22     
==========================================
+ Hits        45977    45981       +4     
- Misses       4886     4904      +18

Flag	Coverage Δ
#multiple	`88.13% <100%> (-0.03%)`	⬇️
#single	`40.21% <85.71%> (-0.22%)`	⬇️

Impacted Files	Coverage Δ
pandas/util/testing.py	`80.75% <100%> (+0.34%)`	⬆️
pandas/io/formats/format.py	`95.66% <100%> (ø)`	⬆️
pandas/io/common.py	`70.21% <100%> (+0.12%)`	⬆️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/plotting/_converter.py	`63.23% <0%> (-1.82%)`	⬇️
pandas/core/util/hashing.py	`90.35% <0%> (-1.69%)`	⬇️
pandas/core/indexes/category.py	`98.18% <0%> (-0.31%)`	⬇️
pandas/core/indexes/multi.py	`96.56% <0%> (-0.17%)`	⬇️
pandas/core/series.py	`94.71% <0%> (-0.1%)`	⬇️
pandas/plotting/_core.py	`81.87% <0%> (ø)`	⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80abd97...f135e32. Read the comment docs.

jreback

fantastic. thanks! (tiny comments). merge when ready.

jreback · 2017-05-09T02:06:49Z

pandas/io/common.py

@@ -314,6 +314,8 @@ def _get_handle(path_or_buf, mode, encoding=None, compression=None,

    handles = list()
    f = path_or_buf
+    # Convert pathlib.Path/py.path.local or string


can you add a blank line

jreback · 2017-05-09T02:07:22Z

doc/source/whatsnew/v0.20.2.txt

@@ -33,6 +33,8 @@ Performance Improvements
 Bug Fixes
 ~~~~~~~~~

+- Bug in using ``pathlib.Path`` object with io functions (:issue:`16291`)


localpath too?

jorisvandenbossche · 2017-05-09T07:47:47Z

pandas/tests/io/parser/common.py

+
+    def test_pickle_path_localpath(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_pathlib(df.to_csv,


round_trip_localpath ?

jreback · 2017-05-09T10:15:19Z

for those ones which xfail this, can you create an issue (put all in issue). Though might be one that just needs updating.

jreback · 2017-05-09T10:16:03Z

pandas/tests/io/sas/test_sas7bdat.py

@@ -65,6 +65,26 @@ def test_from_iterator(self):
                tm.assert_frame_equal(df, df0.iloc[2:5, :])
                rdr.close()

+    def test_path_pathlib(self):


looks like needs xfailing here

jreback · 2017-05-10T11:26:34Z

can you add tests for feather-format as well (it should work I think, not sure)

chris-b1 · 2017-05-12T12:37:17Z

@jreback - this should be good now, looks like most of the xfails will be fixed by #16301

TomAugspurger · 2017-05-12T13:17:22Z

There was a failure in one job: https://travis-ci.org/pandas-dev/pandas/jobs/231516184#L2140 but that looks unrelated.

jreback · 2017-05-12T13:30:04Z

thanks!

* BUG: pathlib.Path in io * CLN: factor out pathlib roundtrip * add localpath tests for other io * fixup * xfail SAS; type in parser * missing import * xfail for pandas-dev#14704 * fix to_csv * lint * lint cleanup * add feather (xfail)

* BUG: pathlib.Path in io * CLN: factor out pathlib roundtrip * add localpath tests for other io * fixup * xfail SAS; type in parser * missing import * xfail for pandas-dev#14704 * fix to_csv * lint * lint cleanup * add feather (xfail) (cherry picked from commit 4cd8458)

* BUG: pathlib.Path in io * CLN: factor out pathlib roundtrip * add localpath tests for other io * fixup * xfail SAS; type in parser * missing import * xfail for #14704 * fix to_csv * lint * lint cleanup * add feather (xfail) (cherry picked from commit 4cd8458)

* BUG: pathlib.Path in io * CLN: factor out pathlib roundtrip * add localpath tests for other io * fixup * xfail SAS; type in parser * missing import * xfail for pandas-dev#14704 * fix to_csv * lint * lint cleanup * add feather (xfail)

BUG: pathlib.Path in io

cff750f

chris-b1 added Bug IO Data IO issues that don't fit into a more specific label Regression Functionality that used to work in a prior pandas version labels May 9, 2017

chris-b1 added this to the 0.20.2 milestone May 9, 2017

jreback requested changes May 9, 2017

View reviewed changes

CLN: factor out pathlib roundtrip

adf77f0

jreback reviewed May 9, 2017

View reviewed changes

add localpath tests for other io

ae22641

jreback approved these changes May 9, 2017

View reviewed changes

fixup

086a11b

jorisvandenbossche reviewed May 9, 2017

View reviewed changes

jreback reviewed May 9, 2017

View reviewed changes

xfail SAS; type in parser

3dda8bc

chris-b1 mentioned this pull request May 9, 2017

ENH: Support pathlib.Path and LocalPath objects in all io functions #16299

Closed

4 tasks

chris-b1 added 2 commits May 9, 2017 07:13

missing import

1ecbe0d

xfail for pandas-dev#14704

8a4e291

jreback mentioned this pull request May 9, 2017

ENH: Support fspath protocol #16301

Merged

chris-b1 added 2 commits May 9, 2017 18:34

fix to_csv

9293b14

lint

83415a7

jorisvandenbossche approved these changes May 10, 2017

View reviewed changes

chris-b1 added 2 commits May 12, 2017 06:13

lint cleanup

577796e

add feather (xfail)

f135e32

jreback merged commit 4cd8458 into pandas-dev:master May 12, 2017

TomAugspurger added Backported and removed Needs Backport labels May 30, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: pathlib.Path in io #16292

BUG: pathlib.Path in io #16292

chris-b1 commented May 9, 2017 •

edited

Loading

jreback May 9, 2017

jreback May 9, 2017

chris-b1 May 9, 2017

jreback May 9, 2017

jreback May 9, 2017

jreback May 9, 2017

jreback May 9, 2017

codecov bot commented May 9, 2017

codecov bot commented May 9, 2017 •

edited

Loading

jreback left a comment

jreback May 9, 2017

jreback May 9, 2017

jorisvandenbossche May 9, 2017

jreback commented May 9, 2017

jreback May 9, 2017

jreback commented May 10, 2017

chris-b1 commented May 12, 2017

TomAugspurger commented May 12, 2017

jreback commented May 12, 2017



		def round_trip_localpath(writer, reader, path=None):
		from py.path import local as LocalPath

BUG: pathlib.Path in io #16292

BUG: pathlib.Path in io #16292

Conversation

chris-b1 commented May 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 9, 2017

Codecov Report

codecov bot commented May 9, 2017 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented May 9, 2017

Choose a reason for hiding this comment

jreback commented May 10, 2017

chris-b1 commented May 12, 2017

TomAugspurger commented May 12, 2017

jreback commented May 12, 2017

chris-b1 commented May 9, 2017 •

edited

Loading

codecov bot commented May 9, 2017 •

edited

Loading