DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs. #57843

yukikitayama · 2024-03-14T18:34:00Z

closes DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs. #57831
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

…n in Ecosystem web page

mroeschke · 2024-03-14T20:48:07Z

environment.yml

@@ -62,7 +62,6 @@ dependencies:
  # downstream packages
  - dask-core<=2024.2.1


Suggested change

- dask-core<=2024.2.1

- dask-core

@mroeschke Thanks for your instructions. This reveals that I didn't understand version pinning, but I learned something.

mroeschke · 2024-03-14T20:48:15Z

requirements-dev.txt

@@ -49,7 +49,6 @@ xlsxwriter>=3.0.5
 zstandard>=0.19.0
 dask<=2024.2.1


Suggested change

dask<=2024.2.1

dask

datapythonista

Couple of details, but changes look good, thanks @yukikitayama for taking care of this.

datapythonista · 2024-03-14T21:19:38Z

doc/source/user_guide/scale.rst

@@ -217,190 +217,10 @@ require too sophisticated of operations. Some operations, like :meth:`pandas.Dat
 much harder to do chunkwise. In these cases, you may be better switching to a
 different library that implements these out-of-core algorithms for you.

-.. _scale.other_libraries:


Can you leave this label please, so we can link to this section if needed.

I see we have warning undefined label because I removed it. Makes sense. Thanks for letting me know.

datapythonista · 2024-03-14T21:23:55Z

doc/source/user_guide/scale.rst

-Use Dask
--------
+Use Other Libraries
+-------------------

 pandas is just one library offering a DataFrame API. Because of its popularity,


This paragraph was probably ok when Dask was discussed in detail here, but I think now it does a very poor job at pointing out to the ecosystem.

A bit of context: This is the documentation for scaling pandas (using pandas with data too big to fit in memory, or to process with a single computer). Besides what's explained above of this section, we want users to know that there are a set of libraries such as PySpask, Dask and Modin that implement an API almost identical to the pandas one, but run in clusters. And that they can find more information in the ecosystem page.

Do you mind trying to rephrase this section in a way that is helpful for users to understand this @yukikitayama ?

Thank you very much for the work here.

…cosystem page

datapythonista

Thanks for the update and the work on this @yukikitayama.

I added couple of minor suggestions, but looks good to me.

datapythonista · 2024-03-15T19:01:56Z

doc/source/user_guide/scale.rst

-.. _`MPI through unidist`: https://github.com/modin-project/unidist
-.. _HDK: https://github.com/intel-ai/hdk
-.. _dask.dataframe: https://docs.dask.org/en/latest/dataframe.html
+There are many other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,


Suggested change

There are many other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,

There are other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,

Thank you for reviewing and giving me suggestions @datapythonista . I saw the unit test ASAN/UBSAN failed, but is it okay?

Yes, our CI seems to be failing now, you can ignore that failure.

datapythonista · 2024-03-15T19:02:19Z

doc/source/user_guide/scale.rst

-.. _HDK: https://github.com/intel-ai/hdk
-.. _dask.dataframe: https://docs.dask.org/en/latest/dataframe.html
+There are many other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,
+but can give you the ability to scale your large dataset processing and analytics


Suggested change

but can give you the ability to scale your large dataset processing and analytics

and can give you the ability to scale your large dataset processing and analytics

lumberbot-app · 2024-03-15T21:43:39Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 2.2.x
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 d4ddc805a03586f9ce0cc1cc541709419ae47c4a

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #57843: DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs.'

Push to a named branch:

git push YOURFORK 2.2.x:auto-backport-of-pr-57843-on-2.2.x

Create a PR against branch 2.2.x, I would have named this PR:

"Backport PR #57843 on branch 2.2.x (DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs.)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

mroeschke · 2024-03-15T21:44:17Z

Thanks @yukikitayama

(Backporting since the dask dependency changes were backported too)

…scale.rst in favor of linking to ecosystem docs.

yukikitayama · 2024-03-15T22:03:30Z

Thank you for reviewing @mroeschke !

…in favor of linking to ecosystem docs. (#57861) Co-authored-by: Yuki Kitayama <[email protected]>

…to ecosystem docs. (pandas-dev#57843) * remove Use Dask adn Use Modin sections * add a new section: Use Other Libraries and link to Out-of-core section in Ecosystem web page * remove dask-expr * remove version pinning from dask and dask-core * put other libraries label back in * update use other libraries description to have a better transfer to ecosystem page * change minor sentences for suggestions * remove unnecessary characters

yukikitayama added 3 commits March 13, 2024 16:39

remove Use Dask adn Use Modin sections

ec5ae9c

add a new section: Use Other Libraries and link to Out-of-core sectio…

1ff37d1

…n in Ecosystem web page

remove dask-expr

08b6ec0

mroeschke reviewed Mar 14, 2024

View reviewed changes

mroeschke added the Docs label Mar 14, 2024

yukikitayama added 2 commits March 14, 2024 13:53

remove version pinning from dask and dask-core

504f02f

Merge remote-tracking branch 'upstream/main' into update-docs

1cb265b

datapythonista reviewed Mar 14, 2024

View reviewed changes

yukikitayama added 2 commits March 15, 2024 09:29

put other libraries label back in

eb77c42

update use other libraries description to have a better transfer to e…

1e5bc9f

…cosystem page

datapythonista approved these changes Mar 15, 2024

View reviewed changes

yukikitayama added 3 commits March 15, 2024 12:33

Merge remote-tracking branch 'upstream/main' into update-docs

bfb5195

change minor sentences for suggestions

1bd146b

remove unnecessary characters

cba7b81

mroeschke approved these changes Mar 15, 2024

View reviewed changes

mroeschke added this to the 2.2.2 milestone Mar 15, 2024

mroeschke merged commit d4ddc80 into pandas-dev:main Mar 15, 2024
46 of 47 checks passed

lumberbot-app bot added the Still Needs Manual Backport label Mar 15, 2024

mroeschke pushed a commit to mroeschke/pandas that referenced this pull request Mar 15, 2024

Backport PR pandas-dev#57843: DOC: Remove Dask and Modin sections in …

0b702f9

…scale.rst in favor of linking to ecosystem docs.

mroeschke removed the Still Needs Manual Backport label Mar 15, 2024

mroeschke added a commit that referenced this pull request Mar 15, 2024

Backport PR #57843: DOC: Remove Dask and Modin sections in scale.rst …

962e233

…in favor of linking to ecosystem docs. (#57861) Co-authored-by: Yuki Kitayama <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs. #57843

DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs. #57843

yukikitayama commented Mar 14, 2024

mroeschke Mar 14, 2024

yukikitayama Mar 14, 2024

mroeschke Mar 14, 2024

datapythonista left a comment

datapythonista Mar 14, 2024

yukikitayama Mar 14, 2024

datapythonista Mar 14, 2024

datapythonista left a comment

datapythonista Mar 15, 2024

yukikitayama Mar 15, 2024

datapythonista Mar 15, 2024

datapythonista Mar 15, 2024

lumberbot-app bot commented Mar 15, 2024

mroeschke commented Mar 15, 2024

yukikitayama commented Mar 15, 2024

		@@ -62,7 +62,6 @@ dependencies:
		# downstream packages
		- dask-core<=2024.2.1

		@@ -49,7 +49,6 @@ xlsxwriter>=3.0.5
		zstandard>=0.19.0
		dask<=2024.2.1

	There are many other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,
	There are other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,

	but can give you the ability to scale your large dataset processing and analytics
	and can give you the ability to scale your large dataset processing and analytics

DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs. #57843

DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs. #57843

Conversation

yukikitayama commented Mar 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lumberbot-app bot commented Mar 15, 2024

mroeschke commented Mar 15, 2024

yukikitayama commented Mar 15, 2024