Skip to content

Switch mongoItemReader to use a cursor instead of paging #3824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Switch mongoItemReader to use a cursor instead of paging #3824

wants to merge 1 commit into from

Conversation

jbotuck
Copy link
Contributor

@jbotuck jbotuck commented Jan 1, 2021

Paging in mongo runs in On^2 time since each skip needs to count all the records it is skipping

@jbotuck
Switch mongoItemReader to use a cursor instead of paging
602afb7 
Paging in mongo runs in On^2 time since each skip needs to count all the records it is skipping
@jbotuck jbotuck force-pushed the feature/mongoCursor branch from b36633c to 602afb7 Compare January 10, 2021 19:00
@fmbenhassine
Copy link
Contributor

Thank you for this PR! We can't change the MongoItemReader in such a way: you have not only changed the behaviour of the reader, but also the signature of several public APIs (some of them have even been removed in this PR). This would be a big breaking change and we can't introduce it as such.

I suggest to create a new cursor-based mongo item reader named MongoCursorItemReader next to the current one, similar to what we already have for JDBC (JdbcCursorItemReader / JdbcPagingItemReader), JPA (JpaCursorItemReader, JpaPagingItemReader), etc.

The current MongoItemReader could be deprecated and extended with a class named MongoPagingItemReader to have a consistent naming.

Please update the PR accordingly and we can continue from there.

@fmbenhassine fmbenhassine added pr-for: feature status: waiting-for-reporter Issues for which we are waiting for feedback from the reporter labels Jan 14, 2021
@MinJunKweon MinJunKweon mentioned this pull request Mar 13, 2023
Closed
@fmbenhassine
Copy link
Contributor

We have a PR for the cursor-based implementation here: #4323 , which seems similar to what we have here. Please do not hesitate to review it if you want.

As mentioned previously, the current MongoItemReader should be renamed to MongoPagingItemReader to have a consistent naming with other readers. I will take care of that in #4341, and I will close this PR in favor of #4323.

Thank you for your contribution anyway.

@fmbenhassine fmbenhassine added status: superseded Issues that are superseded by other issues and removed status: waiting-for-reporter Issues for which we are waiting for feedback from the reporter labels Mar 29, 2023
@jbotuck jbotuck deleted the feature/mongoCursor branch March 29, 2023 15:26
@jbotuck
Copy link
Contributor Author

jbotuck commented Mar 29, 2023

Makes sense to me. Reworking this pr has been on the bottom of my to do list for a while.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
pr-for: feature status: superseded Issues that are superseded by other issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jbotuck @fmbenhassine