Self-service API rate-limit increases

[mdtro]

Problem Statement

There are several issues and discussions around the rate-limits crates.io enforces being problematic when publishing multiple crates within a short window. In addition, to the user impact there is also impact on the crates.io team fielding requests to increase rate-limits for particular users.

Proposed Solution

I'd like to propose a self-service option for users who hit the API rate-limits to temporarily increase their limits to finish their crate publishing.

1. When a user reaches a 429 (Too Many Requests) error while publishing their crates, crates.io will send an email to the user's verified email address with an expiring unique magic link.
2. The user should receive an error message in the body of the 429 response indicating they should check their email.
3. When visiting the magic link, the user is taken to a special page where we 1) verify the logged in user matches and 2) the client is being operated by a human (maybe with a privacy-maintaining captcha or forcing re-authentication with GitHub?)
4. After verification, the user's rate-limit is temporarily increased.

Abuse Mitigations

While this doesn't prevent abuse completely, it does not degrade our current state at all. Users will not have to wait for a crates.io team member to manually increase limits and team members will service less requests.

Some potential mitigations to abusing the self-service system include:

• Monitoring for frequent self-services increases in API rate-limits. Determine if they are legitimate or abusive and act accordingly.
• Implement a sort of cooldown window where a limited number of rate increases is allowed during a defined time window.
• Implement a gradual backoff on the extensions. For example, the first extension bumps the limit to 50/hr, the second extension to 25/hr, the third to 10/hr, etc.
• Implement 12 or 24 hour user-scoped maximum in addition to the hourly rate-limit.
• Define additional limitations and restrictions for new GitHub accounts.
• Once the Trusted Publishing implementation is completed we can also opt-in publish operations using trusted publishing to higher rate-limits automatically.

[LawnGnome]

Just to capture discussions from our team meetings and in person at the all hands, I'm definitely in favour of us experimenting with this.

I think the initial numbers here make sense as a starting point, although I might prefer to frame the extension limits as percentages of what the user already has, just to cover cases where a user who already has a higher limit unexpectedly finds themselves needing to burst through their normal limit, which is something we occasionally see when a user with a large workspace has an issue that requires multiple publishes in rapid succession.