RTD is down

[agjohnson]

Here's an update for anyone who experienced problems with Read the Docs:

Early this morning, at 3:07am PT, I was alerted that RTD went down. As it turns out, Rackspace encountered a global outage with their load balancers as well as with their dashboard. Our load balancers were completely offline, and our DNS couldn't be updated as it was hosted with Rackspace as well. Fortunately, I eventually found that the API was still up -- it was just the dashboard that was inoperable. I repointed DNS from the load balancers to single web servers and attempted to try to increase throughput of the web services to handle the load. At this point, Read the Docs was up, but started flapping and buckling under the load.

Shortly after this, Rackspace load balancers seem to return to normal operations. I have since reverted the changes I made and we are pushing traffic through the load balancers once again.

I'm still working to resolve some issues, it doesn't appear that we're back to 100% yet. I'll continue updating here. Thanks for the patience!

[agjohnson]

Things have now quieted down. I've squashed the remaining issues I was being alerted on. I've noticed residual problems from browsers still pointing to the cached DNS entries. I expect traffic will level off to this box shortly. If you are noticing a slow connection, you might still have this dns query cached.

[jvanasco]

Sorry that happened. I just wanted to suggest something -- which you might be doing already...

If RTD is registered via namecheap (which it might be, since eNom is on your whois...) you can create a secondary namecheap account that only has API access to update DNS hosts on a given domain(s), then run a simple python script to swap DNS. I use that setup a lot to deal with outages, load balancing issues, and DNS challenges for letsencrypt certificates. It's much faster than logging in through 2-Factor-Auth and using their web interface.

[willingc]

@agjohnson Update: Seems back to normal for me. Thanks for the heads up re: sluggish connection.

[monicacecilia]

@agjohnson Thanks for the update. Ours is one of those resources 'not back 100% yet', but I'll be patiently waiting until all things go back to normal. Sincere thanks for keeping us in the loop. RTD rocks!

[agjohnson]

@jvanasco DNS propagation time is still an issue there, but yes, having a secondary place for DNS, or at least decoupling from Rackspace makes sense. Where we stop with this process is another question though.

@monicacecilia This downtime would not have affected build processes, your issue is unrelated.

I'm closing this as there wasn't any fallout from the downtime. Thanks for the reports everyone!