Retry on API failure when connecting from builders #4902
Conversation
humitos
force-pushed
the
humitos/api/retry-on-fail
branch
from
f80fb1e to
d9a9af1
Compare
| connect=3, | ||
| status=3, | ||
| backoff_factor=0.5, # 0.5, 1, 2 seconds | ||
| method_whitelist=('GET', 'PUT', 'PATCH', 'POST'), |
There was a problem hiding this comment.
POST could be a little dangerous here, but considering that we are using it to create a new BuildCommand I think it's necessary.
There was a problem hiding this comment.
From the docs
By default, we only retry on methods which are considered to be idempotent (multiple requests with the same parameters end with the same state). See Retry.DEFAULT_METHOD_WHITELIST.
There was a problem hiding this comment.
I'd be a little concerned about the POST retry as well, but these are the calls that we're currently failing on, so I'm not sure if there is anything we want to do differently here.
humitos
force-pushed
the
humitos/api/retry-on-fail
branch
from
d9a9af1 to
eee2c36
Compare
|
Lint is failing with |
|
In the worst case, retrying on POST will give us duplicated BuildCommand; which is not good at all but at least it won't break anything. We could merge and investigate after deploy or maybe build something to de-duplicate these entries :/ |
Codecov Report
@@ Coverage Diff @@
## master #4902 +/- ##
==========================================
+ Coverage 76.65% 76.65% +<.01%
==========================================
Files 158 158
Lines 10057 10059 +2
Branches 1269 1269
==========================================
+ Hits 7709 7711 +2
Misses 2007 2007
Partials 341 341
|
I used 3 retries for now at 0.5, 1 and 2 seconds.
We can test with these values and we can increase later if we consider necessary.
Closes #2258