Deprecate old /github, /bitbucket, /build endpoints

[gjtorikian]

Sorry @ericholscher, I should've reached out sooner, but it suddenly occurred to me: is the https://readthedocs.org/ integration into GitHub still using GitHub Services? If so, services are being deprecated, and we suggest migrating away as soon as possible.

Please let me know if I can provide any more info or help!

[agjohnson]

Thanks for the heads up! We used GitHub services in the past, and there are likely a non-trivial number of projects still using the old service. However, if you sign up your project today, your project will get a webhook, not the service.

An easy path forward for us is to collect projects hitting the mostly deprecated /github endpoint and send a warning. We should:

• Collect projects as the /github, /bitbucket, or /build endpoint is hit
• Use our notification backend to send site/email notifications for reconnecting project with a webhook integration
• Announce hard deprecation of old endpoints and GitHub services

I also think we should support signed GitHub webhooks with v3, as we'll be pushing more users to it. See #4886 for this work.

[humitos]

@agjohnson how are you thinking to collect these projects? our logs are not saved for too many days now.

Maybe creating a temporal model in the database and calling get_or_create on each /github, /bitbucket, /build and send an email notification every two weeks. After depreciation, we can just remove this model safely. What do you think?

This will cover most of your points. We will need to announce the deprecation (maybe a blog post?) and "signing webhook", but that's a completely different issue for me (I wrote some thoughts on corporate issue tracker, https://github.com/rtfd/readthedocs-corporate/issues/364, that maybe should be moved here)

[gjtorikian]

I pulled numbers for you all at one point (I work at GitHub) but forgot to share them. I think there was something like 27,000 users on the old services, with 24,000 of them being enabled (not active -- simply "turned on." I can't pull active numbers.)

I can grab the users and projects associated with the service, but I'm not sure if that's useful.

[agjohnson]

@humitos perhaps we don't even need a database to track this. our notification system should be able to deduplicate messages and only send one every 2 weeks. So on requests to the deprecated webhook endpoints, we create a notification for the project owners every two weeks.

@gjtorikian 24000 is way more than i would have guessed! it might help to have this information if it's not a pain to pull it. Feel free to move this conversation to a private channel -- [email protected] and if required, my public pgp is at https://github.com/agjohnson.gpg

If GitHub has announced this, I think announcing the deprecation on our side and alerting users with active deprecated webhooks might be enough.

[humitos]

@agjohnson I will write some code today for your idea around notifications.

[humitos]

@humitos perhaps we don't even need a database to track this. our notification system should be able to deduplicate messages and only send one every 2 weeks. So on requests to the deprecated webhook endpoints, we create a notification for the project owners every two weeks.

I'm not seeing this clearly because we have 2 different types of notifications: site and email. Site allows deduplication since it uses the message storage, but email send the email immediately when .send() is called and the notification is not saved anywhere to be recovered from a Celery task later.

I opened #4898 to discuss over the code proposed.

[humitos]

Considering Bitbucket and the deprecation of APIv1 we have,

In [6]: SocialAccount.objects.filter(provider='bitbucket').count()
Out[6]: 1128

These accounts use /1.0/users/{username} as resource_uri and it will stop working on December 31st.

I suppose we want to send a site/email notification to these users communicating them to disconnect from current (old) Bitbucket connection and re-connect their account under https://readthedocs.org/accounts/social/connections/ (the old Bitbucket button is not more available so they will press the correct one ;) )