ENH: added new cache_dates parameter for read_csv func #25990

anmyachev · 2019-04-04T18:15:54Z

closes N/A
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This change allows the user to use caching when parsing dates, which is provided by to_datetime function

pep8speaks · 2019-04-04T18:15:59Z

Hello @anmyachev! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-05-07 08:30:55 UTC

pandas/io/parsers.py

codecov · 2019-04-04T18:55:04Z

Codecov Report

Merging #25990 into master will decrease coverage by 49.95%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           master   #25990       +/-   ##
===========================================
- Coverage   91.86%    41.9%   -49.96%     
===========================================
  Files         175      175               
  Lines       52543    52544        +1     
===========================================
- Hits        48267    22021    -26246     
- Misses       4276    30523    +26247

Flag	Coverage Δ
#multiple	`?`
#single	`41.9% <100%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/parsers.py	`48.24% <100%> (-47.55%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
pandas/io/formats/html.py	`0% <0%> (-99.36%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-95.46%)`	⬇️
pandas/io/sas/sas7bdat.py	`0% <0%> (-91.16%)`	⬇️
pandas/io/sas/sas_xport.py	`0% <0%> (-90.1%)`	⬇️
pandas/core/tools/numeric.py	`10.44% <0%> (-89.56%)`	⬇️
... and 131 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70773d9...c9b258f. Read the comment docs.

codecov · 2019-04-04T18:55:06Z

Codecov Report

Merging #25990 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25990      +/-   ##
==========================================
- Coverage   91.99%   91.98%   -0.01%     
==========================================
  Files         175      175              
  Lines       52377    52378       +1     
==========================================
- Hits        48182    48180       -2     
- Misses       4195     4198       +3

Flag	Coverage Δ
#multiple	`90.53% <100%> (ø)`	⬆️
#single	`40.77% <100%> (-0.13%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/parsers.py	`95.84% <100%> (ø)`	⬆️
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`96.9% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.7% <0%> (+0.1%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 64c1127...687d0c4. Read the comment docs.

pandas/io/parsers.py

jreback · 2019-04-05T00:08:57Z

while we are doing this may as well flip the flag in pd.to_datetime as well to True. (it could be a separate PR), but need to run asv's.

anmyachev · 2019-04-05T08:45:00Z

@jreback I made sure that changing to_datetime signature should be done in a separate request, because, unfortunately, it is not enough to change it alone, I will have to change the signatures of other functions. So last commit will be removed from this PR.

anmyachev · 2019-04-05T09:40:28Z

Note for reviewers: it seems that this PR have the same errors as master branch

gfyoung · 2019-04-08T03:21:59Z

We already have so many datetime parameters at this point, but I'm thinking at some point we should condense them into a dict. This would allow for ease of parameterization without bloat.

@jreback : What do you think?

vnlitvinov · 2019-04-08T14:28:01Z

I'm thinking at some point we should condense them into a dict.

Maybe declare a collections.namedtuple for those arguments, so they're fix-named?

gfyoung · 2019-04-08T15:31:49Z

Maybe declare a collections.namedtuple for those arguments, so they're fix-named?

@vnlitvin : I'm actually inclined to leave it as dict for the time being. I would like to keep that portion of the API flexible so that the only limits on how people want to parse dates is the signature underlying our datetime parsing.

anmyachev · 2019-04-08T18:13:05Z

asv continuous -f 1.1 origin/master cache_dates_api_change -a warmup_time=1 -a sample_time=1

master	our_patch	ratio	test_name
1.45±0.01ms	1.87±0.01ms	1.29	io.csv.ReadCSVParseDates.time_baseline
1.74±0.02ms	2.24±0.02ms	1.29	io.csv.ReadCSVParseDates.time_multiple_date
1.41±0.01ms	1.56±0.02ms	1.11	io.csv.ReadCSVDInferDatetimeFormat.time_read_csv(False, 'iso8601')
1.34±0ms	1.47±0.02ms	1.10	io.csv.ReadCSVDInferDatetimeFormat.time_read_csv(False, 'ymd')
47.3±0.3ms	6.45±0.04ms	0.14	io.csv.ReadCSVCachedParseDates.time_read_csv_cached(True)

anmyachev · 2019-04-09T10:45:19Z

@gfyoung condensation of parameters into dict should be in a separate PR, shouldn't it?

gfyoung · 2019-04-09T15:43:01Z

@anmyachev : Absolutely! Separate PR would be most appropriate.

anmyachev · 2019-04-09T19:40:24Z

@gfyoung ok. Anything left to fix in this PR?

gfyoung

Over to you @jreback

doc/source/whatsnew/v0.25.0.rst

asv_bench/benchmarks/io/csv.py

jreback · 2019-04-09T21:30:53Z

small comments, otherwise lgtm. Is that an issue for changing this in pd.to_datetime as well? (the default)

anmyachev · 2019-04-10T06:07:00Z

@jreback when changing the flag in pd.to_datetime to true, errors occur on some platforms. I do not remember exactly what the problem was there. But in the near future I will create the PR right with this change. That's when I can answer your question.

jreback · 2019-04-10T12:02:07Z

@anmyachev this is a troubling comment. #25990 (comment)

can you show me an example?

anmyachev · 2019-04-10T18:14:26Z

@jreback after getting result of CI tests for #26043 I can show you an example.
I think we can continue our discussion there.

vnlitvinov · 2019-04-17T13:28:45Z

@jreback is there anything left to fix before this could be merged?

vnlitvinov · 2019-04-18T15:16:38Z

@jreback

@anmyachev this is a troubling comment. #25990 (comment)

can you show me an example?

Does our example (#26097) satisfy you here? I feel that it shouldn't be a blocker for this PR.

P.S. I've rebased on master to get rid of merge conflicts (and to have a clean history).

vnlitvinov · 2019-04-25T14:36:30Z

@jreback @gfyoung anything needed from us?

doc/source/whatsnew/v0.25.0.rst

jreback · 2019-04-28T18:07:19Z

is there an issue for this?

anmyachev · 2019-04-28T18:21:22Z

is there an issue for this?

@jreback no errors detected for this.

jreback · 2019-04-28T18:24:22Z

no errors detected for this.

what does this mean?

anmyachev · 2019-04-28T18:29:25Z

no errors detected for this.

what does this mean?

I mean: I not found issue for this.

vnlitvinov · 2019-04-29T12:17:25Z

is there an issue for this?

If you mean Github issue then no, we didn't file one and we didn't fine any.
That's why this PR is used as a reference in whatsnew.

jreback · 2019-05-07T01:21:53Z

can you merge master.

…/io.rst

anmyachev · 2019-05-07T09:11:25Z

@jreback Done & green

jreback · 2019-05-07T11:28:48Z

thanks @anmyachev

TomAugspurger · 2019-05-10T03:35:47Z

Are we OK with this slowdown on the parse dates benchmark? http://pandas.pydata.org/speed/pandas/index.html#io.csv.ReadCSVParseDates.time_multiple_date?commits=f65742cd47d6b364a2cbf150869b282c5aa7daf0

jreback requested changes Apr 4, 2019

View reviewed changes

pandas/io/parsers.py Outdated Show resolved Hide resolved

gfyoung added Enhancement IO CSV read_csv, to_csv Performance Memory or execution speed performance Datetime Datetime data dtype labels Apr 4, 2019

jreback requested changes Apr 4, 2019

View reviewed changes

pandas/io/parsers.py Outdated Show resolved Hide resolved

gfyoung reviewed Apr 4, 2019

View reviewed changes

pandas/io/parsers.py Show resolved Hide resolved

anmyachev force-pushed the cache_dates_api_change branch from 9b0a443 to 2fd52d0 Compare April 5, 2019 06:43

anmyachev force-pushed the cache_dates_api_change branch from 0925e56 to 2fd52d0 Compare April 5, 2019 08:46

anmyachev force-pushed the cache_dates_api_change branch from 7ca68a1 to 002c031 Compare April 5, 2019 16:58

gfyoung mentioned this pull request Apr 8, 2019

ENH: Allow compression in NDFrame.to_csv to be a dict with optional arguments (#26023) #26024

Merged

4 tasks

gfyoung approved these changes Apr 9, 2019

View reviewed changes

jreback added this to the 0.25.0 milestone Apr 9, 2019

jreback requested changes Apr 9, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

asv_bench/benchmarks/io/csv.py Outdated Show resolved Hide resolved

anmyachev force-pushed the cache_dates_api_change branch from ed521d6 to 94c0eb5 Compare April 10, 2019 06:59

anmyachev mentioned this pull request Apr 10, 2019

PERF: changed default value of cache parameter to True in to_datetime function #26043

Merged

4 tasks

vnlitvinov force-pushed the cache_dates_api_change branch from 94c0eb5 to 1c85780 Compare April 18, 2019 15:14

jreback requested changes Apr 28, 2019

View reviewed changes

doc/source/whatsnew/v0.25.0.rst Outdated Show resolved Hide resolved

anmyachev force-pushed the cache_dates_api_change branch from ee0b37f to af757f2 Compare April 28, 2019 18:24

anmyachev added 9 commits May 7, 2019 11:28

added new cache_dates parameter for read_csv func

20fae8b

fix PEP 8 issues

028b55d

changed default value of cache_dates var to true

f83c65c

added whatsnew entry

4da942d

move cache_dates var in docstring on the rigth place

995d8d5

fix wrong pandas version in docstring

07575a2

added cache_dates argument for read_csv func in doc/source/user_guide…

f3186a2

…/io.rst

fixed typo; writed 'cache_dates=True' in doc/source/whatsnew/v0.25.0.rst

fba9407

replaced information about cache_dates=True in IO section

687d0c4

anmyachev force-pushed the cache_dates_api_change branch from af757f2 to 687d0c4 Compare May 7, 2019 08:30

jreback approved these changes May 7, 2019

View reviewed changes

jreback merged commit f65742c into pandas-dev:master May 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: added new cache_dates parameter for read_csv func #25990

ENH: added new cache_dates parameter for read_csv func #25990

anmyachev commented Apr 4, 2019 •

edited

Loading

pep8speaks commented Apr 4, 2019 •

edited

Loading

codecov bot commented Apr 4, 2019

codecov bot commented Apr 4, 2019 •

edited

Loading

jreback commented Apr 5, 2019

anmyachev commented Apr 5, 2019

anmyachev commented Apr 5, 2019

gfyoung commented Apr 8, 2019

vnlitvinov commented Apr 8, 2019

gfyoung commented Apr 8, 2019

anmyachev commented Apr 8, 2019

anmyachev commented Apr 9, 2019

gfyoung commented Apr 9, 2019

anmyachev commented Apr 9, 2019

gfyoung left a comment

jreback commented Apr 9, 2019

anmyachev commented Apr 10, 2019

jreback commented Apr 10, 2019

anmyachev commented Apr 10, 2019 •

edited

Loading

vnlitvinov commented Apr 17, 2019

vnlitvinov commented Apr 18, 2019

vnlitvinov commented Apr 25, 2019

jreback commented Apr 28, 2019

anmyachev commented Apr 28, 2019

jreback commented Apr 28, 2019

anmyachev commented Apr 28, 2019

vnlitvinov commented Apr 29, 2019

jreback commented May 7, 2019

anmyachev commented May 7, 2019

jreback commented May 7, 2019

TomAugspurger commented May 10, 2019

ENH: added new cache_dates parameter for read_csv func #25990

ENH: added new cache_dates parameter for read_csv func #25990

Conversation

anmyachev commented Apr 4, 2019 • edited Loading

pep8speaks commented Apr 4, 2019 • edited Loading

Comment last updated at 2019-05-07 08:30:55 UTC

codecov bot commented Apr 4, 2019

Codecov Report

codecov bot commented Apr 4, 2019 • edited Loading

Codecov Report

jreback commented Apr 5, 2019

anmyachev commented Apr 5, 2019

anmyachev commented Apr 5, 2019

gfyoung commented Apr 8, 2019

vnlitvinov commented Apr 8, 2019

gfyoung commented Apr 8, 2019

anmyachev commented Apr 8, 2019

anmyachev commented Apr 9, 2019

gfyoung commented Apr 9, 2019

anmyachev commented Apr 9, 2019

gfyoung left a comment

Choose a reason for hiding this comment

jreback commented Apr 9, 2019

anmyachev commented Apr 10, 2019

jreback commented Apr 10, 2019

anmyachev commented Apr 10, 2019 • edited Loading

vnlitvinov commented Apr 17, 2019

vnlitvinov commented Apr 18, 2019

vnlitvinov commented Apr 25, 2019

jreback commented Apr 28, 2019

anmyachev commented Apr 28, 2019

jreback commented Apr 28, 2019

anmyachev commented Apr 28, 2019

vnlitvinov commented Apr 29, 2019

jreback commented May 7, 2019

anmyachev commented May 7, 2019

jreback commented May 7, 2019

TomAugspurger commented May 10, 2019

anmyachev commented Apr 4, 2019 •

edited

Loading

pep8speaks commented Apr 4, 2019 •

edited

Loading

codecov bot commented Apr 4, 2019 •

edited

Loading

anmyachev commented Apr 10, 2019 •

edited

Loading