Skip to content

GitHub OAuth: use bigger pages to make fewer requests #9020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 17, 2022

Conversation

humitos
Copy link
Member

@humitos humitos commented Mar 16, 2022

Currently, we are using the default pagination that GitHub offers, which is 30.
This commit increases the page size to 100 when fetching organization for a user
and repositories for each organizations.

Note we are already using 100 page size when fetching repositories for a user.

Related to #8979

Currently, we are using the default pagination that GitHub offers, which is 30.
This commit increases the page size to 100 when fetching organization for a user
and repositories for each organizations.

Note we are already using 100 page size when fetching repositories for a user.

Related to #8979
@humitos humitos force-pushed the humitos/oauth-big-page branch from 25b742f to 12f70cf Compare March 16, 2022 16:55
@humitos humitos requested a review from stsewd March 16, 2022 17:04
Copy link
Member

@stsewd stsewd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should test if

resp = self.get_session().get(url, data=kwargs)
works as expected, because requests says we should be using params https://docs.python-requests.org/en/latest/user/quickstart/#passing-parameters-in-urls 🤔

A quick test shows that data is passed as the body, which get requests shouldn't have... we are only using the kwargs for gitlab, maybe it's just a happy coincidence that they handled the params inside the body as query args.

>>> r = requests.get('https://example.com', params={'foo': 'bar'})
>>> r.url
'https://example.com/?foo=bar'
>>> r.request.body
>>> r = requests.get('https://example.com', data={'foo': 'bar'})
>>> r.url
'https://example.com/'
>>> r.request.body
'foo=bar'

Maybe it's just safer to change data to params.

@humitos
Copy link
Member Author

humitos commented Mar 16, 2022

Good catch! This is definitely not working:

In [9]: s = GitHubService.for_user(User.objects.get(username='github'))[0]

In [10]: response = s.get_session().get('https://api.github.com/user/repos', data={'per_page': 1})

In [11]: len(response.json())
Out[11]: 30

In [12]: response = s.get_session().get('https://api.github.com/user/repos', params={'per_page': 1})

In [13]: len(response.json())
Out[13]: 1

We need to use params= to pass them as GET attributes.

@humitos
Copy link
Member Author

humitos commented Mar 17, 2022

@stsewd I did a final QA locally with all the oauth services supported and all of them worked fine with the params=. I think we are ready to merge this PR.

@humitos humitos enabled auto-merge March 17, 2022 15:58
@humitos humitos merged commit a7e3083 into master Mar 17, 2022
@humitos humitos deleted the humitos/oauth-big-page branch March 17, 2022 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants