DOC: Update index parameter in pandas to_parquet

[galuhsahid]

• closes #xxxx (reference: Update index parameter in pandas to_parquet python-sprints/pandas-mentoring#156)
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

The documentation (https://dev.pandas.io/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet) says that for DataFrame.to_parquet when the value for the index parameter is None, the behavior will depend on the engine. However if we try this both engines, pyarrow and fastparquet, will keep the index. This PR proposes to set the default value of the index parameter as index=True instead of index=None since both engines keep the index anyway. Would appreciate any discussion/feedback :)

Sample code to demonstrate how both engines treat the index:

import pandas as pd 
  
data = {'name':['a', 'b', 'c', 'd']} 
df = pd.DataFrame(data, index =['rank_1', 'rank_2', 'rank_3', 'rank_4'])

df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, rank_1 to rank_4
Data columns (total 1 columns):
name    4 non-null object
dtypes: object(1)
memory usage: 64.0+ bytes

# pyarrow
df.to_parquet("test_pyarrow.parquet", engine="pyarrow")
df_read_pyarrow = pd.read_parquet("test_pyarrow.parquet", engine="pyarrow")
df_read_pyarrow.info()

Output:

<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, rank_1 to rank_4
Data columns (total 1 columns):
name    4 non-null object
dtypes: object(1)
memory usage: 64.0+ bytes

# fastparquet
df.to_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet = pd.read_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet.info()

Output:

<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, rank_1 to rank_4
Data columns (total 1 columns):
name    4 non-null object
dtypes: object(1)
memory usage: 64.0+ bytes

[datapythonista]

Thanks for working on this @galuhsahid, looks great. There are couple of tests that need to be updated (see the CI). Let me know if you need help with it.

@wesm does this change make sense? I think by default we're saving the index with parquet with both engines. But we've got this index=None that the documentation doesn't clarify what it does, and personally I think it doesn't add value, but make things confusing (may be I'm missing something).

[galuhsahid]

@datapythonista Okay so because of the failing tests I tried to_parquet without defining an index, and I observed a different behavior when I used the fastparquet engine for index=True and index=None. If I use index=True it will return Int64Index, and if I use index=None it will return RangeIndex.

I'm not really sure if this difference has any impact to our proposal, should we take this into consideration? Otherwise I can update the test to expect Int64Index instead of RangeIndex.

import pandas as pd

data = {'name':['a', 'b', 'c', 'd']}
df = pd.DataFrame(data)
df.to_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet = pd.read_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet.index

Output:

RangeIndex(start=0, stop=4, step=1)

import pandas as pd

data = {'name':['a', 'b', 'c', 'd']}
df = pd.DataFrame(data)
df.to_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet = pd.read_parquet("test_fastparquet.parquet", engine="fastparquet", index=True)
df_read_fastparquet.index

Output:

Int64Index([0, 1, 2, 3], dtype='int64', name='index')

[datapythonista]

Thanks @galuhsahid for doing the research. I see that there is a difference between index=None and index=True with fastparquet, but only with the default RangeIndex.

With a default index:

>>> import pandas
>>> pandas.DataFrame({'foo': [1, 2]}).to_parquet('foo.parquet', engine='fastparquet', index=None)
>>> print(pandas.read_parquet('foo.parquet').index)
RangeIndex(start=0, stop=2, step=1)
>>> pandas.DataFrame({'foo': [1, 2]}).to_parquet('foo.parquet', engine='fastparquet', index=True)
>>> print(pandas.read_parquet('foo.parquet').index)
Int64Index([0, 1], dtype='int64', name='index')

With a set index:

>>> import pandas
>>> pandas.DataFrame({'foo': [1, 2]}, index=['a', 'b']).to_parquet('foo.parquet', engine='fastparquet', index=None)
>>> print(pandas.read_parquet('foo.parquet').index)
Index(['a', 'b'], dtype='object', name='index')
>>> pandas.DataFrame({'foo': [1, 2]}, index=['a', 'b']).to_parquet('foo.parquet', engine='fastparquet', index=True)
>>> print(pandas.read_parquet('foo.parquet').index)
Index(['a', 'b'], dtype='object', name='index')

This looks like a bug to me, I don't see why we would like that behavior. @martindurant any thoughts on this. Do you think it makes sense to just have index=True and index=False and avoid this difference when index=None?

[wesm]

Can @jorisvandenbossche have a closer look at this before merging? I'm concerned this may cause a performance regression.

We made changes in pyarrow to use preserve_index=True force serialization of the index column as data in the Parquet file, even for RangeIndex. If preserve_index=None then metadata-only RangeIndex is stored.

[jorisvandenbossche]

Yes, @wesm is right. In the latest pyarrow release, preserve_index=None (the default) or preserve_index=True mean different things (the main difference is for RangeIndex being written as a column or just as metadata). So if pandas would change the default from None to True, that means it would override the default of pyarrow (leading again to writing the RangeIndex as a full column).

[datapythonista]

Thanks for the info @jorisvandenbossche, should we just clarify in the documentation what None means then?

[jorisvandenbossche]

Yes, that is a good idea. From their docs, it seems that fastparquet uses the same logic of None vs True/False.

[datapythonista]

@galuhsahid do you want to update this PR with what discussed above? Restore the default parameter, and explain in the docstring what None means.

[galuhsahid]

@datapythonista Sure thanks for notifying! Will push an update later today.

[pep8speaks]

Hello @galuhsahid! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-09-15 15:33:29 UTC

[jorisvandenbossche]

Thanks for the the update!

[jorisvandenbossche]

The default is still None, so need to change this back I think

[jorisvandenbossche]

Maybe mention that other indexes are included as columns in the output in this case?

[datapythonista]

Thanks @galuhsahid for the update.

Can you change the default back to None in the docstrings too?

And for the None description, looks good, but I'm wondering if it'd be clearer to explain that None is like True and the index is stored, but that a RangeIndex (the default in pandas) is not stored as the values like True, but as a range in the metadata, so it doesn't require so much space, and it's faster.

The same you're saying, but a bit more extended, so people without much knowledge about parquet or pandas indices can still understand whether it's a good choice for them.

Btw, minor thing. Every time you push, people who participated in the conversation of the PR receives a notification. I think it's a good practice to commit as many times as you think it's useful, but when it doesn't make a difference, it may be good to just push when your local changes are ready for another review. Not a big deal if pushing more often is useful to you, and you surely want to push too if you stop working on this for the day and want to back up your changes in your remote branch... But wanted to let you know, in case it's useful

[galuhsahid]

Yes I think that's a good idea - I've expanded the explanation, would appreciate any input :) also thanks for the tip, I'll be more mindful when pushing my commits!

[datapythonista]

Thanks @galuhsahid, lgtm.

@jorisvandenbossche do you want to have a look?

[jorisvandenbossche]

Thanks a lot @galuhsahid !