GitHub OAuth: skip conda-forge organization when syncing

[humitos]

conda-forge on GitHub has a lot of repositories and that produces different issues on our side. The most recent one was that our Celery task was finishing due to timeout and the sync wasn't performed.

Since nobody (?) imports projects from conda-forge into Read the Docs, we thought it could be a good idea to skip it from the synchronization. This will reduce the time required considerably and also the database's size.

Related #8974

[astrojuanlu]

Not even GitHub can render them:

However, instead of making a special case for conda-forge, perhaps it would make more sense to disable orgs with, say, > 1000 repositories?

[humitos]

That makes sense to me 👍. I'll see if we have that data on the first request and make the decision there before start consuming the resulting pages

[humitos]

I took a quick look at this today and I don't think it's possible to implement it without a considerable refactor.

1. we are using this GitHub endpoint now https://docs.github.com/en/rest/reference/repos#list-repositories-for-the-authenticated-user which lists all the repositories for the logged-in user. I tried asking only for ?affiliation=owner to avoid fetching all the repositories for all the organizations the user belongs to, but it fetches repositories the user owns under the organizations as well --so I'm not sure that will solve the issue
2. we would need to check the links HTTP response header after fetching the first page to calculate how many pages are available to calculate how many repositories the organization has
3. we would need to raise an exception to stop the pagination at

   readthedocs.org/readthedocs/oauth/services/github.py

   Line 197 in 902590d

   return response.links.get('next', {}).get('url')

   and skip the organization

Even making those changes, I'm not 100% sure that this will work 😄 . Ideally, we would like to improve step 1) as much as possible but I didn't find a way to do that yet.

[humitos]

I opened #9020 to increase the page size to 100 when dealing with organizations and their repositories. This is not the best fix, but at least it will reduce the number of requests required.