From 1a1803187bca25643ed6491763f221ac2b58b15a Mon Sep 17 00:00:00 2001 From: Kevin Date: Thu, 27 Jul 2023 10:23:35 -0700 Subject: [PATCH 1/2] Added AWS_S3_HOST environment variable workaround to user_guide for issue #26195 --- doc/source/user_guide/io.rst | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/doc/source/user_guide/io.rst b/doc/source/user_guide/io.rst index 3dba120c0c64b..db162064899b0 100644 --- a/doc/source/user_guide/io.rst +++ b/doc/source/user_guide/io.rst @@ -1718,9 +1718,26 @@ at `fsimpl1`_ for implementations built into ``fsspec`` and `fsimpl2`_ for those not included in the main ``fsspec`` distribution. -You can also pass parameters directly to the backend driver. For example, -if you do *not* have S3 credentials, you can still access public data by -specifying an anonymous connection, such as +You can also pass parameters directly to the backend driver. Because we can no +longer utilize the ``AWS_S3_HOST`` environment variable, we can directly define a +dictionary containing the endpoint_url and pass the object into the storage +option parameter: + +.. code-block:: python + + def s3so(worker_id): + worker_id = "5" if worker_id == "master" else worker_id.lstrip("gw") + return dict(client_kwargs={"endpoint_url": f"http://127.0.0.1:555{worker_id}/"}) + + roundtripped_df = pd.read_json( + "s3://pandas-test/test-1", compression=compression, storage_options=s3so, + ) + +More sample configurations and documentation can be found at `S3Fs documentation +`__. + +If you do *not* have S3 credentials, you can still access public +data by specifying an anonymous connection, such as .. versionadded:: 1.2.0 @@ -2992,7 +3009,7 @@ Read a URL with no options: Read in the content of the "books.xml" file and pass it to ``read_xml`` as a string: -.. ipython:: python +.. ipython:: python :okexcept: file_path = "books.xml" with open(file_path, "w") as f: From c08b89a138e2f2d1937934b39b275d80d0115be4 Mon Sep 17 00:00:00 2001 From: Kevin Date: Fri, 28 Jul 2023 12:12:11 -0700 Subject: [PATCH 2/2] Updated wording for issue #26195 --- doc/source/user_guide/io.rst | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/doc/source/user_guide/io.rst b/doc/source/user_guide/io.rst index db162064899b0..42e12b9741d99 100644 --- a/doc/source/user_guide/io.rst +++ b/doc/source/user_guide/io.rst @@ -1718,20 +1718,15 @@ at `fsimpl1`_ for implementations built into ``fsspec`` and `fsimpl2`_ for those not included in the main ``fsspec`` distribution. -You can also pass parameters directly to the backend driver. Because we can no -longer utilize the ``AWS_S3_HOST`` environment variable, we can directly define a +You can also pass parameters directly to the backend driver. Since ``fsspec`` does not +utilize the ``AWS_S3_HOST`` environment variable, we can directly define a dictionary containing the endpoint_url and pass the object into the storage option parameter: .. code-block:: python - def s3so(worker_id): - worker_id = "5" if worker_id == "master" else worker_id.lstrip("gw") - return dict(client_kwargs={"endpoint_url": f"http://127.0.0.1:555{worker_id}/"}) - - roundtripped_df = pd.read_json( - "s3://pandas-test/test-1", compression=compression, storage_options=s3so, - ) + storage_options = {"client_kwargs": {"endpoint_url": "http://127.0.0.1:5555"}}} + df = pd.read_json("s3://pandas-test/test-1", storage_options=storage_options) More sample configurations and documentation can be found at `S3Fs documentation `__. @@ -3009,7 +3004,7 @@ Read a URL with no options: Read in the content of the "books.xml" file and pass it to ``read_xml`` as a string: -.. ipython:: python :okexcept: +.. ipython:: python file_path = "books.xml" with open(file_path, "w") as f: