read_parquet S3 dir support #26388

Seanspt · 2019-05-14T10:36:57Z

When using spark to process data and save to s3, the files are like

s3://bucket/path/_SUCCESS
s3://bucket/path/part-00000-uuid.snappy.parquet
s3://bucket/path/part-00002-uuid.snappy.parquet
s3://bucket/path/part-00001-uuid.snappy.parquet
s3://bucket/path/part-00003-uuid.snappy.parquet

It would be nice to load these files in one line.

df = pd.read_parquet("s3://bucket/path")

jreback · 2019-05-14T10:48:48Z

pls file your issue on the arrow tracker

these are just passed thru

jorisvandenbossche · 2019-05-14T18:27:03Z

Both pyarrow and fastparquet support reading from a directory of files. So it can also be an issue in how we call those libraries.

@Seanspt what is the error that you get?

Seanspt · 2019-05-16T10:39:41Z

Pandas works fine if I download the spark-saved dir and read it by passing a local dir path.
However, when using s3:// dir path, the error got is as followed.

>>> pd.read_parquet("s3://xxx/xxx/df")
Traceback (most recent call last):
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 524, in info
    Key=key, **self.req_kw)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 195, in _call_s3
    return method(**additional_kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/botocore/client.py", line 624, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/pandas/io/s3.py", line 29, in get_filepath_or_buffer
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 350, in open
    s3_additional_kwargs=kw)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 1206, in __init__
    info = self.info()
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 1224, in info
    refresh=refresh, **kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 535, in info
    raise_from(FileNotFoundError(path), e)
  File "<string>", line 3, in raise_from
FileNotFoundError: xxx/xxx/df

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 524, in info
    Key=key, **self.req_kw)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 195, in _call_s3
    return method(**additional_kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/botocore/client.py", line 624, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/pandas/io/parquet.py", line 288, in read_parquet
    return impl.read(path, columns=columns, **kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/pandas/io/parquet.py", line 124, in read
    path, _, _, should_close = get_filepath_or_buffer(path)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/pandas/io/common.py", line 209, in get_filepath_or_buffer
    mode=mode)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/pandas/io/s3.py", line 38, in get_filepath_or_buffer
    filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 350, in open
    s3_additional_kwargs=kw)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 1206, in __init__
    info = self.info()
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 1224, in info
    refresh=refresh, **kwargs)
  File "/Users/sean/miniconda3/envs/py3/lib/python3.6/site-packages/s3fs/core.py", line 535, in info
    raise_from(FileNotFoundError(path), e)
  File "<string>", line 3, in raise_from
FileNotFoundError: xxx/xxx/df

Seanspt · 2019-05-16T10:43:11Z

As a reference here. Pyarrow and fast parquet can do this in several lines of code.

jorisvandenbossche · 2019-05-16T10:50:12Z

Thanks for the follow-up!

If you read the directory on s3 directly with pyarrow as done in the answer of the StackOverflow question, then it also works for you? Can you show that?

Seanspt · 2019-05-17T06:19:36Z

Yes, the following code works for me.

import s3fs
import pyarrow.parquet as pq
fs = s3fs.S3FileSystem(region_name='cn-north-1')

bucket_uri = f's3://xxx/xxx/df'

dataset = pq.ParquetDataset(bucket_uri, filesystem=fs)
table = dataset.read()
df = table.to_pandas()

jorisvandenbossche · 2019-05-17T06:56:03Z

Thanks!

@TomAugspurger I suppose the problem is that our s3 compat code assumes it is a single file and tries to open it? (read_parquet also accepts a directory name to read)
We use S3FileSystem.open(..), which is for single files.

I suppose one option might be to, in case of an s3 path, not to use our get_filepath_or_buffer but let pyarrow handle the s3?

For reference, the PR where this was added: #19135

TomAugspurger · 2019-05-17T14:06:17Z

Letting the engine handle the user-provided path seems reasonable.

…

On Fri, May 17, 2019 at 1:56 AM Joris Van den Bossche < ***@***.***> wrote: Thanks! @TomAugspurger <https://github.com/TomAugspurger> I suppose the problem is that our s3 compat code assumes it is a single file and tries to open it? (read_parquet also accepts a directory name to read) We use S3FileSystem.open(..), which is for single files. I suppose one option might be to, in case of an s3 path, not to use our get_filepath_or_buffer but let pyarrow handle the s3? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26388?email_source=notifications&email_token=AAKAOIX5Y4M6WD7VFL6TR43PVZJIXA5CNFSM4HMYSOK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVT5GFY#issuecomment-493343511>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIV6QNEL4GDVMVQVQP3PVZJIXANCNFSM4HMYSOKQ> .

alimcmaster1 · 2020-04-17T00:10:53Z

Thanks!

@TomAugspurger I suppose the problem is that our s3 compat code assumes it is a single file and tries to open it? (read_parquet also accepts a directory name to read)
We use S3FileSystem.open(..), which is for single files.

I suppose one option might be to, in case of an s3 path, not to use our get_filepath_or_buffer but let pyarrow handle the s3?

For reference, the PR where this was added: #19135

@jorisvandenbossche i'm happy to take a look at this.

You ideas make sense to me. Think we can get rid of our get_filepath_or_buffer and pyarrow.parquet.read_table here and generalise to ParquetDataset any thoughts/concerns?

jreback · 2020-04-26T21:41:36Z

closed by #33632

ghost · 2020-06-30T04:31:50Z

Can we reopen this issue? As mentioned in the 1.0.5 release notes (#34785) this is not yet fixed, and took me a while to find out why I was still getting it in 1.0.5 despite the fact it was fixed in 1.0.4

alimcmaster1 · 2020-06-30T23:06:59Z

Can we reopen this issue? As mentioned in the 1.0.5 release notes (#34785) this is not yet fixed, and took me a while to find out why I was still getting it in 1.0.5 despite the fact it was fixed in 1.0.4

It’s being released in pandas 1.1 -> happy to reopen if you have issues.

Perhaps try build master and check it works as expected?

animesh-wynk · 2020-10-15T08:10:10Z

Yes, the following code works for me.

import s3fs
import pyarrow.parquet as pq
fs = s3fs.S3FileSystem(region_name='cn-north-1')

bucket_uri = f's3://xxx/xxx/df'

dataset = pq.ParquetDataset(bucket_uri, filesystem=fs)
table = dataset.read()
df = table.to_pandas()

But what if the parquet file size is huge? In such cases, can I make python generators over the parquet file? If yes, how?
Thanks in advance: )

ilivans · 2021-06-11T14:28:36Z

I'm getting the same exception here
Although if I specify one partition it works 🤔 :

df = pd.read_parquet("bucket/key/country=X", filesystem=s3, filters=[('day', '=', 10)])

With the source directory it doesn't:

df = pd.read_parquet("bucket/key", filesystem=s3, filters=[('day', '=', 10)])
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-6ebb8c990af1> in <module>
----> 1 df = pd.read_parquet("bucket/key", filesystem=s3, filters=[('day', '=', 10)])

~/.virtualenvs/ltr/lib/python3.8/site-packages/pandas/io/parquet.py in read_parquet(path, engine, columns, use_nullable_dtypes, **kwargs)
    457     """
    458     impl = get_engine(engine)
--> 459     return impl.read(
    460         path, columns=columns, use_nullable_dtypes=use_nullable_dtypes, **kwargs
    461     )

~/.virtualenvs/ltr/lib/python3.8/site-packages/pandas/io/parquet.py in read(self, path, columns, use_nullable_dtypes, storage_options, **kwargs)
    219         )
    220         try:
--> 221             return self.api.parquet.read_table(
    222                 path_or_handle, columns=columns, **kwargs
    223             ).to_pandas(**to_pandas_kwargs)

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/parquet.py in read_table(source, columns, use_threads, metadata, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes)
   1698             )
   1699         try:
-> 1700             dataset = _ParquetDatasetV2(
   1701                 source,
   1702                 filesystem=filesystem,

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/parquet.py in __init__(self, path_or_paths, filesystem, filters, partitioning, read_dictionary, buffer_size, memory_map, ignore_prefixes, **kwargs)
   1556                 infer_dictionary=True)
   1557 
-> 1558         self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
   1559                                    format=parquet_format,
   1560                                    partitioning=partitioning,

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/dataset.py in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
    654 
    655     if _is_path_like(source):
--> 656         return _filesystem_dataset(source, **kwargs)
    657     elif isinstance(source, (tuple, list)):
    658         if all(_is_path_like(elem) for elem in source):

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/dataset.py in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
    409     factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
    410 
--> 411     return factory.finish(schema)
    412 
    413 

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset.DatasetFactory.finish()

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/_fs.pyx in pyarrow._fs._cb_open_input_file()

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/fs.py in open_input_file(self, path)
    312 
    313         if not self.fs.isfile(path):
--> 314             raise FileNotFoundError(path)
    315 
    316         return PythonFile(self.fs.open(path, mode="rb"), mode="r")

FileNotFoundError: bucket/key/

In both cases: import s3fs; s3 = s3fs.core.S3FileSystem()
Environment:

pandas==1.2.4
pyarrow==4.0.1
s3fs==0.4.2

lu-liu-rft · 2021-06-15T12:34:00Z

I'm getting the same exception here
Although if I specify one partition it works 🤔 :

df = pd.read_parquet("bucket/key/country=X", filesystem=s3, filters=[('day', '=', 10)])

With the source directory it doesn't:

df = pd.read_parquet("bucket/key", filesystem=s3, filters=[('day', '=', 10)])
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-6ebb8c990af1> in <module>
----> 1 df = pd.read_parquet("bucket/key", filesystem=s3, filters=[('day', '=', 10)])

~/.virtualenvs/ltr/lib/python3.8/site-packages/pandas/io/parquet.py in read_parquet(path, engine, columns, use_nullable_dtypes, **kwargs)
    457     """
    458     impl = get_engine(engine)
--> 459     return impl.read(
    460         path, columns=columns, use_nullable_dtypes=use_nullable_dtypes, **kwargs
    461     )

~/.virtualenvs/ltr/lib/python3.8/site-packages/pandas/io/parquet.py in read(self, path, columns, use_nullable_dtypes, storage_options, **kwargs)
    219         )
    220         try:
--> 221             return self.api.parquet.read_table(
    222                 path_or_handle, columns=columns, **kwargs
    223             ).to_pandas(**to_pandas_kwargs)

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/parquet.py in read_table(source, columns, use_threads, metadata, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes)
   1698             )
   1699         try:
-> 1700             dataset = _ParquetDatasetV2(
   1701                 source,
   1702                 filesystem=filesystem,

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/parquet.py in __init__(self, path_or_paths, filesystem, filters, partitioning, read_dictionary, buffer_size, memory_map, ignore_prefixes, **kwargs)
   1556                 infer_dictionary=True)
   1557 
-> 1558         self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
   1559                                    format=parquet_format,
   1560                                    partitioning=partitioning,

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/dataset.py in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
    654 
    655     if _is_path_like(source):
--> 656         return _filesystem_dataset(source, **kwargs)
    657     elif isinstance(source, (tuple, list)):
    658         if all(_is_path_like(elem) for elem in source):

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/dataset.py in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
    409     factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
    410 
--> 411     return factory.finish(schema)
    412 
    413 

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset.DatasetFactory.finish()

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/_fs.pyx in pyarrow._fs._cb_open_input_file()

~/.virtualenvs/ltr/lib/python3.8/site-packages/pyarrow/fs.py in open_input_file(self, path)
    312 
    313         if not self.fs.isfile(path):
--> 314             raise FileNotFoundError(path)
    315 
    316         return PythonFile(self.fs.open(path, mode="rb"), mode="r")

FileNotFoundError: bucket/key/

In both cases: import s3fs; s3 = s3fs.core.S3FileSystem()
Environment:

pandas==1.2.4
pyarrow==4.0.1
s3fs==0.4.2

Hi I'm seeing the same issue, just wondering if you have a way to resolve it?

jreback closed this as completed May 14, 2019

jreback added the IO Parquet parquet, feather label May 14, 2019

jreback added this to the No action milestone May 14, 2019

jorisvandenbossche reopened this May 14, 2019

jorisvandenbossche removed this from the No action milestone May 14, 2019

alimcmaster1 mentioned this issue Apr 16, 2020

pd.read_parquet can't handle an S3 directory #28490

Closed

alimcmaster1 self-assigned this Apr 17, 2020

alimcmaster1 mentioned this issue Apr 18, 2020

IO: Fix parquet read from s3 directory #33632

Merged

6 tasks

mroeschke added the Enhancement label Apr 19, 2020

jreback added this to the 1.1 milestone Apr 22, 2020

jreback closed this as completed Apr 26, 2020

simonjayhawkins modified the milestones: 1.1, 1.0.4 May 26, 2020

simonjayhawkins mentioned this issue Jun 15, 2020

DOC: add release note about revert for 1.0.5 #34785

Merged

NahsiN mentioned this issue Sep 30, 2020

BUG: read_parquet Google Cloud Storage (gcs) dir support #36743

Closed

3 tasks

datapythonista mentioned this issue Jul 15, 2022

Support reading lists of Parquet files with read_parquet bodo-ai/Bodo-Pandas-Collaboration#7

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

read_parquet S3 dir support #26388

read_parquet S3 dir support #26388

Seanspt commented May 14, 2019

jreback commented May 14, 2019

jorisvandenbossche commented May 14, 2019

Seanspt commented May 16, 2019

Seanspt commented May 16, 2019

jorisvandenbossche commented May 16, 2019

Seanspt commented May 17, 2019

jorisvandenbossche commented May 17, 2019 •

edited

Loading

TomAugspurger commented May 17, 2019 via email

alimcmaster1 commented Apr 17, 2020

jreback commented Apr 26, 2020

ghost commented Jun 30, 2020

alimcmaster1 commented Jun 30, 2020

animesh-wynk commented Oct 15, 2020

ilivans commented Jun 11, 2021

lu-liu-rft commented Jun 15, 2021

read_parquet S3 dir support #26388

read_parquet S3 dir support #26388

Comments

Seanspt commented May 14, 2019

jreback commented May 14, 2019

jorisvandenbossche commented May 14, 2019

Seanspt commented May 16, 2019

Seanspt commented May 16, 2019

jorisvandenbossche commented May 16, 2019

Seanspt commented May 17, 2019

jorisvandenbossche commented May 17, 2019 • edited Loading

TomAugspurger commented May 17, 2019 via email

alimcmaster1 commented Apr 17, 2020

jreback commented Apr 26, 2020

ghost commented Jun 30, 2020

alimcmaster1 commented Jun 30, 2020

animesh-wynk commented Oct 15, 2020

ilivans commented Jun 11, 2021

lu-liu-rft commented Jun 15, 2021

jorisvandenbossche commented May 17, 2019 •

edited

Loading