Skip to content

25708 performance degradation when dumping large dataframe with datetime to csv #25764

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 0 commits into from
Closed

25708 performance degradation when dumping large dataframe with datetime to csv #25764

wants to merge 0 commits into from

Conversation

hksonngan
Copy link
Contributor

        values = self.values
        i8values = self.values.view('i8')

        if slicer is not None:
            i8values = i8values[..., slicer]

        from pandas.io.formats.format import _get_format_datetime64_from_values
        fmt = _get_format_datetime64_from_values(values, date_format)

@codecov
Copy link

codecov bot commented Mar 18, 2019

Codecov Report

Merging #25764 into master will decrease coverage by 49.5%.
The diff coverage is 0%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #25764       +/-   ##
===========================================
- Coverage   91.25%   41.74%   -49.51%     
===========================================
  Files         172      172               
  Lines       52977    52968        -9     
===========================================
- Hits        48342    22113    -26229     
- Misses       4635    30855    +26220
Flag Coverage Δ
#multiple ?
#single 41.74% <0%> (-0.01%) ⬇️
Impacted Files Coverage Δ
pandas/io/parsers.py 48.27% <ø> (-47.08%) ⬇️
pandas/plotting/_core.py 17.71% <0%> (-65.95%) ⬇️
pandas/core/internals/blocks.py 51.89% <0%> (-42.2%) ⬇️
pandas/core/reshape/merge.py 9.51% <0%> (-84.98%) ⬇️
pandas/io/formats/latex.py 0% <0%> (-100%) ⬇️
pandas/io/sas/sas_constants.py 0% <0%> (-100%) ⬇️
pandas/tseries/plotting.py 0% <0%> (-100%) ⬇️
pandas/tseries/converter.py 0% <0%> (-100%) ⬇️
pandas/io/formats/html.py 0% <0%> (-99.36%) ⬇️
pandas/core/groupby/categorical.py 0% <0%> (-95.46%) ⬇️
... and 133 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 707c720...a15579d. Read the comment docs.

@codecov
Copy link

codecov bot commented Mar 18, 2019

Codecov Report

Merging #25764 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #25764      +/-   ##
==========================================
+ Coverage   91.25%   91.25%   +<.01%     
==========================================
  Files         172      172              
  Lines       52977    52968       -9     
==========================================
- Hits        48342    48334       -8     
+ Misses       4635     4634       -1
Flag Coverage Δ
#multiple 89.82% <100%> (-0.01%) ⬇️
#single 41.74% <0%> (-0.01%) ⬇️
Impacted Files Coverage Δ
pandas/io/parsers.py 95.34% <ø> (ø) ⬆️
pandas/plotting/_core.py 83.65% <100%> (ø) ⬆️
pandas/core/internals/blocks.py 94.08% <100%> (ø) ⬆️
pandas/core/reshape/merge.py 94.4% <100%> (-0.08%) ⬇️
pandas/util/testing.py 89.08% <0%> (+0.09%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 707c720...9eec9b8. Read the comment docs.

@hksonngan hksonngan closed this Mar 18, 2019
@hksonngan
Copy link
Contributor Author

Sorry, my mistake git rebase. I close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Performance degradation when dumping large dataframe with np.datetime to csv
1 participant