Skip to content

Commit 6cf6963

Browse files
authored
DOC: Add Hugging Face Hub access (#60608)
* Update pyproject.toml * Update install.rst * Update io.rst * remove pip extra * Update ecosystem.md * link to docs * Revert change in io.rst
1 parent 710f7d2 commit 6cf6963

File tree

1 file changed

+25
-0
lines changed

1 file changed

+25
-0
lines changed

web/pandas/community/ecosystem.md

+25
Original file line numberDiff line numberDiff line change
@@ -468,6 +468,31 @@ df.dtypes
468468

469469
ArcticDB also supports appending, updating, and querying data from storage to a pandas DataFrame. Please find more information [here](https://docs.arcticdb.io/latest/api/query_builder/).
470470

471+
### [Hugging Face](https://huggingface.co/datasets)
472+
473+
The Hugging Face Dataset Hub provides a large collection of ready-to-use datasets for machine learning shared by the community. The platform offers a user-friendly interface to explore, discover and visualize datasets, and provides tools to easily load and work with these datasets in Python thanks to the [huggingface_hub](https://github.com/huggingface/huggingface_hub) library.
474+
475+
You can access datasets on Hugging Face using `hf://` paths in pandas, in the form `hf://datasets/username/dataset_name/...`.
476+
477+
For example, here is how to load the [stanfordnlp/imdb dataset](https://huggingface.co/datasets/stanfordnlp/imdb):
478+
479+
```python
480+
import pandas as pd
481+
482+
# Load the IMDB dataset
483+
df = pd.read_parquet("hf://datasets/stanfordnlp/imdb/plain_text/train-00000-of-00001.parquet")
484+
```
485+
486+
Tip: on a dataset page, click on "Use this dataset" to get the code to load it in pandas.
487+
488+
To save a dataset on Hugging Face you need to [create a public or private dataset](https://huggingface.co/new-dataset) and [login](https://huggingface.co/docs/huggingface_hub/quick-start#login-command), and then you can use `df.to_csv/to_json/to_parquet`:
489+
490+
```python
491+
# Save the dataset to my Hugging Face account
492+
df.to_parquet("hf://datasets/username/dataset_name/train.parquet")
493+
```
494+
495+
You can find more information about the Hugging Face Dataset Hub in the [documentation](https://huggingface.co/docs/hub/en/datasets).
471496

472497
## Out-of-core
473498

0 commit comments

Comments
 (0)