Skip to content

CI/WEB: Use Github token to authenticate API calls #50388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 28, 2022

Conversation

mroeschke
Copy link
Member

Hopefully to help the doc and web build sometimes failing because of hitting the rate limit of the GIthub API

Traceback (most recent call last):
  File "web/pandas_web.py", line 403, in <module>
    main(args.source_path, args.target_path, args.base_url, args.ignore_io_errors)
  File "web/pandas_web.py", line 349, in main
    context = get_context(config_fname, ignore_io_errors, base_url=base_url)
  File "web/pandas_web.py", line 305, in get_context
    context = preprocessor(context)
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 172, in maintainers_add_info
    resp.raise_for_status()
  File "/home/runner/micromamba/envs/test/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: rate limit exceeded for url: https://api.github.com/users/shoyer

@mroeschke mroeschke added CI Continuous Integration Web pandas website labels Dec 21, 2022
Copy link
Member

@phofl phofl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this work locally? My impression was, that multiple doc builds were running on the same machine which caused us to hit the limit.

@lithomas1
Copy link
Member

Haven't looked into this closely, but maybe the problem is that we have too many maintainers 🤣 , causing us to hit the API rate limit(I think it's set really low by default by Github).

We should probably figure out a way to use the GITHUB_TOKEN to call the API on CI, and fall back to unauthenticated API use when its not present(e.g. developing locally).
https://docs.github.com/en/actions/security-guides/automatic-token-authentication#example-2-calling-the-rest-api

@mroeschke mroeschke changed the title WEB: Add backoff + retry for Github API calls CI/WEB: Use Github token to authenticate API calls Dec 28, 2022
@mroeschke mroeschke added this to the 2.0 milestone Dec 28, 2022
@mroeschke
Copy link
Member Author

Changed to use the GITHUB_TOKEN if available as an environment variable. Looks to be building the site correctly in the CI

@phofl phofl merged commit ddde1dd into pandas-dev:main Dec 28, 2022
@phofl
Copy link
Member

phofl commented Dec 28, 2022

Thx, lets give it a shot

@mroeschke mroeschke deleted the web/github_retry branch December 28, 2022 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration Web pandas website
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants