[$100] Failure when closing challenge #244

jmgasper · 2019-11-22T23:40:21Z

@afrisalyp - I think we get a lot of failures when closing challenges with errors like this:

Payment failed: [400]: Failed to update challenge. Detail: javax.ejb.EJBTransactionRolledbackException

We've experienced that for awhile, and from what I can figure out, I think it happens when we end up trying to process the same ticket twice, at the same time. Can you double check if that could happen please? Maybe we can add a lock on the record in DynamoDB to indicate a record is being paid so we don't attempt to duplicate the payment?

The text was updated successfully, but these errors were encountered:

jmgasper · 2019-11-22T23:42:37Z

Contest https://www.topcoder-dev.com/challenges/30107810 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-22T23:43:01Z

Contest https://www.topcoder-dev.com/challenges/30107811 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

afrisalyp · 2019-11-23T01:38:51Z

@jmgasper
This might be another issue.
I see the challenge creation comment is

Contest https://www.topcoder-dev.com/challenges/30107811 has been created for this ticket.

But the URL is for the dev environment.
So if we open the URL https://www.topcoder-dev.com/challenges/30107811, we'll see "challenge doesn't exist".
But if we change the URL to https://www.topcoder.com/challenges/30107811. We'll get the challenge page.

So it might be we need to check the configuration for the production env. Maybe we have something missed.

afrisalyp · 2019-11-23T04:16:00Z

The issue is confirmed. A double process can lead to the error message.
I can reproduce the issue by

reopening a paid ticket,
removing the tcx_paid label,
then close the ticket again.

Fixed by the pull request.

But somehow I found that the issue can also occur related to error from API calls.

jmgasper · 2019-11-23T05:27:53Z

@afrisalyp - Yeah, the config had the wrong URL. It hasn't been affecting anything other than the messages back to Github, and it should be fixed now.

Is there any way to avoid a double-process of a payment?

afrisalyp · 2019-11-23T05:35:55Z

@jmgasper
Yeah, we have implemented the lock status in the DB record.
A pull request submitted.
It's ready for review.

jmgasper · 2019-11-23T06:23:31Z

@afrisalyp - Here's some logs:

debug: Getting the billing account ID for project ID: 17159
debug: Getting project billing detail 17159
debug: Assigning the billing account id 80000863 to challenge
debug: Updating challenge 30107635 with {"billingAccountId":80000863,"prizes":[100]}
debug: fetch resource from challenge 30107635
debug: Getting the topcoder member ID for copilot name : ghostar
debug: adding resource to challenge 30107635
debug: resource is added to challenge 30107635 successfully.
debug: Getting the topcoder member ID for member name: afrisalyp
debug: fetch resource from challenge 30107635
debug: Assignee is already set, so skipping
debug: Activating challenge 30107635
debug: Activating challenge 30107635
debug: Challenge 30107635 is activated successfully.
debug: Closing challenge with winner afrisalyp(40697084)
debug: Closing challenge 30107635
debug: ENTER handleEventGracefully
debug: input arguments
debug: { event:
   { event: 'issue.closed',
     provider: 'github',
     data: { issue: [Object], repository: [Object], assignee: [Object] },
     copilot:
      { accessToken: '<removed>',
        userProviderId: 21790,
        topcoderUsername: 'ghostar' },
     dbIssue:
      { projectId: '4eeb7236-6721-4bc0-99b1-fabda68f2dee-1574306795229',
        challengeId: 30107635,
        repositoryId: 135514613,
        status: 'challenge_creation_successful',
        assignee: 'afrisalyp',
        assignedAt: '2019-11-22T01:16:09.796Z',
        number: 237,
        labels: [Array],
        updatedAt: '2019-11-22T07:11:29.727Z',
        provider: 'github',
        id: '59dbd27d-e129-43d6-8412-e6869c3a05fd-1574385243608',
        prizes: [Array],
        title: 'Investigate blank screen when loading dev',
        body: '<p>As discussed, dev loads to a blank screen.</p>\n' },
     assigneeMember:
      { githubUserId: 8887344,
        githubUsername: 'afrisalyp',
        topcoderUsername: 'afrisalyp',
        gitlabUsername: 'afrisalyp',
        id: 'c6472584-11dc-451d-bb85-893d0d162b67-1574306163398',
        gitlabUserId: 1932242 },
     paymentSuccessful: false },
  data:
   { number: 237,
     title: 'Investigate blank screen when loading dev',
     body: '<p>As discussed, dev loads to a blank screen.</p>\n',
     provider: 'github',
     repositoryId: 135514613,
     labels: [ 'tcx_FixAccepted' ],
     projectId: '4eeb7236-6721-4bc0-99b1-fabda68f2dee-1574306795229',
     prizes: [ 100 ],
     assignee: 'afrisalyp' },
  err: { name: 'ProcessorError', statusCode: 500, errorAt: 'topcoder' } }
debug: Scheduling event for next retry
error: Error happened in handleEventGracefully
ProcessorError: Failed to close challenge.
    at Object.convertTopcoderApiError (/topcoder-x-processor/utils/errors.js:81:20)
    at Object.closeChallenge (/topcoder-x-processor/utils/topcoder-api-helper.js:247:18)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
error:  message=Failed to close challenge., stack=ProcessorError: Failed to close challenge.
    at Object.convertTopcoderApiError (/topcoder-x-processor/utils/errors.js:81:20)
    at Object.closeChallenge (/topcoder-x-processor/utils/topcoder-api-helper.js:247:18)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7), name=ProcessorError, statusCode=500, errorAt=topcoder, logged=true

This is with the latest code - can we add some logging around that 500 error?

jmgasper · 2019-11-23T06:28:34Z

Another one. I think this is eventually going to close though - but it's reporting a 500 that we need to capture and decide if we can ignore it or not:

debug:  repoUrl=https://github.com/topcoder-platform/topcoder-x-ui, tcDirectId=17159, archived=false, copilot=ghostar, secretWebhookKey=783ac1dd-45e0-735a-e095-d3de524014f6, registeredWebhookId=159650955, owner=ghostar, id=4eeb7236-6721-4bc0-99b1-fabda68f2dee-1574306795229, title=Topcoder-X
debug: Getting the billing account ID for project ID: 17159
debug: Getting project billing detail 17159
debug: Assigning the billing account id 80000863 to challenge
debug: Updating challenge 30107660 with {"billingAccountId":80000863,"prizes":[75]}
debug: fetch resource from challenge 30107660
debug: Getting the topcoder member ID for copilot name : ghostar
debug: adding resource to challenge 30107660
debug: resource is added to challenge 30107660 successfully.
debug: Getting the topcoder member ID for member name: afrisalyp
debug: fetch resource from challenge 30107660
debug: Assignee is already set, so skipping
debug: Activating challenge 30107660
debug: Challenge 30107660 is activated successfully.
debug: Closing challenge with winner afrisalyp(40697084)
debug: Closing challenge 30107660
debug: ENTER handleEventGracefully
debug: input arguments
debug: { event:
   { event: 'issue.closed',
     provider: 'github',
     data: { issue: [Object], repository: [Object], assignee: [Object] },
     copilot:
      { accessToken: '<removed>',
        userProviderId: 21790,
        topcoderUsername: 'ghostar' },
     dbIssue:
      { projectId: '4eeb7236-6721-4bc0-99b1-fabda68f2dee-1574306795229',
        challengeId: 30107660,
        number: 238,
        repositoryId: 135514613,
        labels: [Array],
        updatedAt: '2019-11-22T07:11:18.727Z',
        status: 'challenge_creation_successful',
        provider: 'github',
        id: 'e32c112d-7b97-4bca-9513-e3353865d398-1574405435903',
        prizes: [Array],
        title: 'Gitlab access link not showing required information',
        body: '<p>Sample link emailed directly</p>\n<p>Resultant page doesn\'t look right, although the access was granted.  We need to ensure that the page is filled in appropriately.</p>\n<p>&lt;img width=&quot;1380&quot; alt=&quot;Screen Shot 2019-11-22 at 3 40 24 pm&quot; src=&quot;https://user-images.githubusercontent.com/21790/69398454-67dea780-0d3e-11ea-8cd9-ce632fa38ffc.png&quot;&gt;</p>\n' },
     assigneeMember:
      { githubUserId: 8887344,
        githubUsername: 'afrisalyp',
        topcoderUsername: 'afrisalyp',
        gitlabUsername: 'afrisalyp',
        id: 'c6472584-11dc-451d-bb85-893d0d162b67-1574306163398',
        gitlabUserId: 1932242 },
     paymentSuccessful: false },
  data:
   { number: 238,
     title: 'Gitlab access link not showing required information',
     body: '<p>Sample link emailed directly</p>\n<p>Resultant page doesn\'t look right, although the access was granted.  We need to ensure that the page is filled in appropriately.</p>\n<p>&lt;img width=&quot;1380&quot; alt=&quot;Screen Shot 2019-11-22 at 3 40 24 pm&quot; src=&quot;https://user-images.githubusercontent.com/21790/69398454-67dea780-0d3e-11ea-8cd9-ce632fa38ffc.png&quot;&gt;</p>\n',
     provider: 'github',
     repositoryId: 135514613,
     labels: [ 'tcx_FixAccepted' ],
     projectId: '4eeb7236-6721-4bc0-99b1-fabda68f2dee-1574306795229',
     prizes: [ 75 ],
     assignee: 'afrisalyp' },
  err: { name: 'ProcessorError', statusCode: 500, errorAt: 'topcoder' } }
debug: Scheduling event for next retry
error: Error happened in handleEventGracefully
ProcessorError: Failed to close challenge.
    at Object.convertTopcoderApiError (/topcoder-x-processor/utils/errors.js:81:20)
    at Object.closeChallenge (/topcoder-x-processor/utils/topcoder-api-helper.js:247:18)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
error:  message=Failed to close challenge., stack=ProcessorError: Failed to close challenge.
    at Object.convertTopcoderApiError (/topcoder-x-processor/utils/errors.js:81:20)
    at Object.closeChallenge (/topcoder-x-processor/utils/topcoder-api-helper.js:247:18)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7), name=ProcessorError, statusCode=500, errorAt=topcoder, logged=true

afrisalyp · 2019-11-23T06:42:27Z

@jmgasper
Actually, the error is exactly the same as the error I used to get when we still use the v3 challenge API version. For the topcoder-dev environment.
But when we migrated it to v4. The error fixed.

But I think it doesn't apply in the prod.
Do you think its a good idea to try to use the v3 version of the challenge API in the prod instead of v4?

jmgasper · 2019-11-23T06:51:05Z

@afrisalyp - Aren't we using the v4 API in prod? We should be.

afrisalyp · 2019-11-23T06:56:01Z

@jmgasper
Yeah. I mean, for the production, v3 is the stable version.
Now, the prod is using v4. I was wondering if we keep using v3 in the prod.

I'm very familiar with the error so I'm pretty sure that the 500 error is from the API problem.

jmgasper · 2019-11-23T07:01:06Z

@afrisalyp - Let's add some logging first so we know what the 500 error is and then we'll decide if we want to downgrade or not.

afrisalyp · 2019-11-23T07:06:21Z

@jmgasper Okay.

afrisalyp · 2019-11-23T07:59:34Z

@jmgasper it's done.

jmgasper · 2019-11-26T23:29:21Z

Contest https://www.topcoder.com/challenges/30108186 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T23:29:21Z

This ticket was not processed for payment. If you would like to process it for payment, please reopen it, add the tcx_FixAccepted label, and then close it again

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-27T00:16:11Z

This ticket was not processed for payment. If you would like to process it for payment, please reopen it, add the tcx_FixAccepted label, and then close it again

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-27T00:25:00Z

This ticket was not processed for payment. If you would like to process it for payment, please reopen it, add the tcx_FixAccepted label, and then close it again

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-27T00:27:02Z

Contest https://www.topcoder.com/challenges/30108186 has been updated - it has been assigned to afrisalyp.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-27T00:30:25Z

Payment task has been updated: https://software.topcoder-dev.com/review/actions/ViewProjectDetails?pid=30108186

This is an automated message for ghostar via Topcoder X

jmgasper added the tcx_OpenForPickup label Nov 22, 2019

jmgasper assigned afrisalyp Nov 22, 2019

afrisalyp mentioned this issue Nov 22, 2019

[$350] Duplicate tickets #234

Closed

afrisalyp added the tcx_Assigned label Nov 23, 2019

afrisalyp mentioned this issue Nov 23, 2019

Prevent double payment processing. topcoder-archive/topcoder-platform-topcoder-x-processor#27

Merged

afrisalyp added tcx_ReadyForReview and removed tcx_Assigned tcx_OpenForPickup labels Nov 23, 2019

afrisalyp mentioned this issue Nov 23, 2019

Add logs for api call error topcoder-archive/topcoder-platform-topcoder-x-processor#28

Merged

jmgasper closed this as completed Nov 26, 2019

jmgasper reopened this Nov 26, 2019

jmgasper unassigned afrisalyp Nov 26, 2019

jmgasper added tcx_OpenForPickup and removed tcx_ReadyForReview labels Nov 27, 2019

jmgasper closed this as completed Nov 27, 2019

jmgasper reopened this Nov 27, 2019

jmgasper closed this as completed Nov 27, 2019

jmgasper reopened this Nov 27, 2019

jmgasper assigned afrisalyp Nov 27, 2019

jmgasper added tcx_Assigned and removed tcx_OpenForPickup labels Nov 27, 2019

jmgasper added the tcx_FixAccepted label Nov 27, 2019

jmgasper closed this as completed Nov 27, 2019

jmgasper added the tcx_Paid label Nov 27, 2019

[$100] Failure when closing challenge #244

[$100] Failure when closing challenge #244

Comments

jmgasper commented Nov 22, 2019

jmgasper commented Nov 22, 2019

Uh oh!

jmgasper commented Nov 22, 2019

Uh oh!

afrisalyp commented Nov 23, 2019

Uh oh!

afrisalyp commented Nov 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmgasper commented Nov 23, 2019

Uh oh!

afrisalyp commented Nov 23, 2019

Uh oh!

jmgasper commented Nov 23, 2019

Uh oh!

jmgasper commented Nov 23, 2019

Uh oh!

afrisalyp commented Nov 23, 2019

Uh oh!

jmgasper commented Nov 23, 2019

Uh oh!

afrisalyp commented Nov 23, 2019

Uh oh!

jmgasper commented Nov 23, 2019

Uh oh!

afrisalyp commented Nov 23, 2019

Uh oh!

afrisalyp commented Nov 23, 2019

Uh oh!

jmgasper commented Nov 26, 2019

Uh oh!

jmgasper commented Nov 26, 2019

Uh oh!

jmgasper commented Nov 27, 2019

Uh oh!

jmgasper commented Nov 27, 2019

Uh oh!

jmgasper commented Nov 27, 2019

Uh oh!

jmgasper commented Nov 27, 2019

Uh oh!

afrisalyp commented Nov 23, 2019 •

edited

Loading