ENH #6416: performance improvements on write

[mangecoeur]

• tradoff higher memory use for faster writes. This replaces the earlier PR where the history was a mess!

[jorisvandenbossche]

can be removed?

[jreback]

@mangecoeur instead of using iterrows.....just do the loop directly; will be much faster as you are not creating a Series (which you then decompose) each time

e.g. just do:

        columns = self.columns
        for k, v in zip(self.index, self.values):
          # work with k,v here

this munges dtypes btw...e.g. everything gets put into a single dtype. I don't think this matters though?

you might want to consider doing this by dtype, e.g. use df.as_blocks (which you then select the correct column out of each block). the advantage is that they are already dtype separated

[mangecoeur]

@jreback ok i see what you mean. will look into it.

[jreback]

@mangecoeur this latter approach is used in to_csv actually, see here: https://github.com/pydata/pandas/blob/master/pandas/core/format.py#L1285
(as well as chunking the writes to use a constant amount of memory). (so see CSVFormatter._save). You can basically do this almost exactly I think. This also has substitution routines for example NaT and such (though may need slightly modifications in to_native_types to not stringify (or you might want to do this part inline as you want to return SQLAlchemy types). But you could do something like this in the sql.py:

class Block(object):

      def __init__(self, block):
            self.block = block

       def format(self, slicer):
            # return a 'formated block suitable for direct insertion to sql


class DatetimeBlock(Block):

      def format(self, slicer):
          ....

etc

[mangecoeur]

@jreback I think we can keep it simple, since the column types are defined on the SQLAlchemy side by the DB table, sqlalchemy deals with converting python values to SQL types already, all we need is to supply a list of row dicts. Maybe we can optimize this more later.

[jreback]

sure....just some thoughts....always profile of course!

[jorisvandenbossche]

@mangecoeur Maybe you can add a vbench to see the effect of this PR?

[mangecoeur]

@jorisvandenbossche not got vbench working yet. was just from daily use found it was much faster this way :P will get round to it soonish

[mangecoeur]

@jreback had a look at that method, turns out that if you iterate over values you get problems with datetime conversion which using iterrows appears to solve. Would need to test on pure numeric data for perf comparison to see if its work trying to fix datetime issue.

[jreback]

@mangecoeur yeh....as I said, the self.values call in and of itself bascially munges everything togther. We can merge this and then put up a future investigate perf issue.

so ready to merge then?

[mangecoeur]

@jreback ok, just cleaned up based on the comments, should be good to go once travis has done its thing.

[jreback]

@mangecoeur thanks!