[$100] Update `scanOne` to query #256

jmgasper · 2019-11-26T02:44:19Z

@afrisalyp - We narrowed down the problem with duplicates to us not implementing scanOne properly.

Scanning only scans a subset of larger tables at once, meaning we have to scan multiple times to scan the whole table, which we don't do. This explains why sometimes it randomly works (found in the first "page"), but other times doesn't (on a page that wasn't scanned).

We've set up indices on dev and prod that we can query directly, instead of using scanOne. Can you make the proper code updates for this please? This should fix the last big issue (with the duplicate issues being created).

The text was updated successfully, but these errors were encountered:

jmgasper · 2019-11-26T02:44:30Z

Contest https://www.topcoder.com/challenges/30108050 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:46:31Z

Contest https://www.topcoder.com/challenges/30108051 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:48:46Z

Contest https://www.topcoder.com/challenges/30108052 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:50:46Z

Contest https://www.topcoder.com/challenges/30108053 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:52:49Z

[500]: Challenge for the updated issue 256 is creating, rescheduling this event

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:52:49Z

[500]: Challenge for the updated issue 256 is creating, rescheduling this event

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:53:04Z

Contest https://www.topcoder.com/challenges/30108054 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-26T02:53:06Z

Contest https://www.topcoder.com/challenges/30108054 has been updated - it has been assigned to afrisalyp.

This is an automated message for ghostar via Topcoder X

afrisalyp · 2019-11-26T03:22:19Z

Scanning only scans a subset of larger tables at once, meaning we have to scan multiple times to scan the whole table, which we don't do. This explains why sometimes it randomly works (found in the first "page"), but other times doesn't (on a page that wasn't scanned).

@jmgasper - Owh, that's weird behavior of dynamodb that I've never realized it.
But it sounds good that we already know the real problem.

We've set up indices on dev and prod that we can query directly, instead of using scanOne. Can you make the proper code updates for this please? This should fix the last big issue (with the duplicate issues being created).

Yeah, I'm going to work on it.
Thanks.

jmgasper · 2019-11-26T03:25:08Z

@afrisalyp - You're not the only one. Multiple reviewers and myself missed that as well, and because we don't have as much test data in dev as we do in prod, we never really saw the issue. At least we have an idea of what the issue is now!

afrisalyp · 2019-11-26T04:03:11Z

@jmgasper - Yeah, that's good.

Topcoder X uses scan for all data fetching, not only the Issue table.
Do we need to change all of scan usage to query?
Or we just focus on the Issue table?

jmgasper · 2019-11-26T04:04:09Z

Eventually we'll have to, but let's focus on Issue table for now, just to get past this blocking issue.

afrisalyp · 2019-11-26T04:12:17Z

Yeah. agreed.
Project and UserMapping tables aren't ready for query yet.
They don't have required indexes.

afrisalyp · 2019-11-26T04:57:11Z

Two PR for topcoder-x-processor and topcoder-x-ui submitted.

jmgasper · 2019-11-26T05:25:47Z

@afrisalyp - I'm not sure this is working properly for new tickets, where the query won't return anything:

debug: Enter queryOne.
debug: {"number":{"eq":258},"provider":{"eq":"github"},"repositoryId":{"eq":135514613}}
debug: queryOne. Error. TypeError: Cannot read property 'map' of undefined
error: Error happened in process
TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
error:  message=Cannot read property 'map' of undefined, stack=TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7), logged=true
debug: queryOne. Error. TypeError: Cannot read property 'map' of undefined
error: Error happened in process
TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
error:  message=Cannot read property 'map' of undefined, stack=TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36)
    at Attribute.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:630:25)
    at Schema.parseDynamo (/topcoder-x-processor/node_modules/dynamoose/dist/Schema.js:196:38)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7), logged=true

afrisalyp · 2019-11-26T05:51:13Z

That's weird again. It's the same error as seen in #245. TypeError: Cannot read property 'map' of undefined

I'm sure that I've successfully tested the code in my local dev.
Btw, what's the version of node and npm using by the prod?

jmgasper · 2019-11-26T05:54:22Z

@afrisalyp - Looks like prod uses node 8.12.0

Not sure what NPM version is used.

afrisalyp · 2019-11-26T06:26:14Z

I'm not really sure if this has an effect, but I found that the model in the migration script has different type in labels. The value in migration script is Array and it is required, but in topcoder-x-ui and topcoder-x-processor we have [String] and not required.

https://github.com/topcoder-platform/topcoder-x-ui/blob/d2cf9f93b6f295dd82e6beb922238fa42f94e929/scripts/mongodb-migration/dynamodb/models/Issue.js#L50

https://github.com/topcoder-platform/topcoder-x-ui/blob/d2cf9f93b6f295dd82e6beb922238fa42f94e929/src/models/Issue.js#L50

https://github.com/topcoder-platform/topcoder-x-processor/blob/71d442d4889e35353cc0c4292165571a80ac51f9/models/Issue.js#L51

jmgasper · 2019-11-26T06:27:54Z

@afrisalyp - Not sure that would affect us here because this is for a new ticket, so not something that was migrated over.

jmgasper · 2019-11-26T06:32:07Z

We can try to change it though - do you want to test in dev first?

afrisalyp · 2019-11-26T06:39:29Z

I don't see this error #256 (comment) in my local dev.
Is that appear in x.topcoder-dev.com?

Also, didn't the migration script use for the prod only?

jmgasper · 2019-11-26T06:44:22Z

Yes, the migration script was for prod only. We can attempt to adjust the code to account for that change in prod, if you want.

afrisalyp · 2019-11-26T06:57:20Z

@jmgasper
I'm not really sure that it's the problem.
Still investigating in my local db

afrisalyp · 2019-11-26T07:36:41Z

@jmgasper - Now I'm pretty sure that it's the problem. :D
I dumped my local DB.
Here is it looks like

        {
            "assignedAt": {
                "N": "1574461009424"
            },
            "title": {
                "S": "Simultan issue 3"
            },
            "body": {
                "S": "<p>3</p>\n"
            },
            "labels": {
                "SS": [
                    "tcx_Assigned"
                ]
            },
            "number": {
                "N": "11"
            },
            "challengeId": {
                "N": "30055718"
            },
            "provider": {
                "S": "github"
            },
            "prizes": {
                "NS": [
                    "13"
                ]
            },
            "repositoryId": {
                "N": "223172322"
            },
            "id": {
                "S": "e67b2d56-112e-4451-b75a-684126413c21-1574460996831"
            },
            "assignee": {
                "S": "telaga"
            },
            "projectId": {
                "S": "93aa952e-da9e-45b9-b920-88b2c0560d04-1574340401139"
            },
            "updatedAt": {
                "N": "1574461013506"
            },
            "status": {
                "S": "challenge_creation_successful"
            }
        }

And if we compare with the prod DB record that you just sent me

        {
            "projectId": {
                "S": "951d1d30-75c8-4b6f-8195-f0f4bd1c90d4-1574306750843"
            },
            "challengeId": {
                "N": "30085497"
            },
            "repositoryId": {
                "N": "10469433"
            },
            "status": {
                "S": "challenge_creation_successful"
            },
            "assignedAt": {
                "N": "1551860245570"
            },
            "number": {
                "N": "133"
            },
            "labels": {
                "S": "[\"tcx_Assigned\",\"tcx_FixAccepted\",\"tcx_Assigned\"]"
            },
            "updatedAt": {
                "N": "1551921403829"
            },
            "provider": {
                "S": "gitlab"
            },
            "id": {
                "S": "aada0c1c-c170-4074-9ba1-de6db1096266-1574309109010"
            },
            "title": {
                "S": "PRIORITY - Handle NDA"
            },
            "body": {
                "S": "<p>This ticket will update the NDA page to be wired into the service:</p>\n<p>http://174.129.220.35:8000/registration/developer/nda</p>\n<p>Once the user has agreed to the NDA"
            },
            "prizes": {
                "NS": [
                    "350"
                ]
            }
        },

We have different labels type. "S" and "SS".
And if we take a look into the stacktrace file /topcoder-x-processor/node_modules/dynamoose/dist/Attribute.js:555:36

         const set = json[`${type}S`];
         return Promise.all(set.map((v) => {

It's related to the "S".

jmgasper · 2019-11-26T07:43:31Z

@afrisalyp - I made the change in prod, but now the query isn't working properly: Note that below the repositoryId isn't matching, nor is the provider.

debug: Enter queryOne.
debug: {"number":{"eq":260},"provider":{"eq":"github"},"repositoryId":{"eq":135514613}}
debug: EXIT process
debug: output arguments
debug: undefined
debug: queryOne. Result.
debug:  projectId=951d1d30-75c8-4b6f-8195-f0f4bd1c90d4-1574306750843, challengeId=30086732, repositoryId=10469433, status=challenge_creation_successful, assignedAt=Wed Mar 20 2019 11:39:42 GMT+0000 (UTC), number=260, labels=[tcx_OpenForPickup], updatedAt=Tue Nov 26 2019 07:41:40 GMT+0000 (UTC), provider=gitlab, id=27dbd483-841b-405f-8d50-9bfd3e542554-1574309381278, title=Update left menu component, body=<p>@suppermancool - We updated the left menu component to be able to be driven by a secondary nav:</p>
<p>https://gitlab.com/herc-dev-prtl/project-resources-devportal/wikis/Setup-Secondary-Navigation</p>
<p>One thing that's been requested is the ability to render and show up to <em>5</em> separate levels.  Is that something you can add support for in the component please?</p>
<p>We have no UI designs for this, and I've asked for them, but I wouldn't expect to get any.</p>
, prizes=[200]
error: Error happened in process
Error: Issue 260 is already in challenge_creation_successful

afrisalyp · 2019-11-26T10:10:13Z

@jmgasper - Fixed.

according to https://dynamoosejs.com/api/query/#querylimitlimit.
Query also has the limitation. So I added all() to make sure we get proper data.

jmgasper · 2019-11-27T00:05:33Z

Contest https://www.topcoder.com/challenges/30108189 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-27T00:06:18Z

Contest https://www.topcoder.com/challenges/30108189 has been updated - it has been assigned to afrisalyp.

This is an automated message for ghostar via Topcoder X

jmgasper · 2019-11-27T00:07:32Z

Payment task has been updated: https://software.topcoder-dev.com/review/actions/ViewProjectDetails?pid=30108189

This is an automated message for ghostar via Topcoder X

jmgasper added the tcx_OpenForPickup label Nov 26, 2019

jmgasper assigned afrisalyp Nov 26, 2019

jmgasper added tcx_Assigned and removed tcx_OpenForPickup labels Nov 26, 2019

This was referenced Nov 26, 2019

Fetch Issue with query. topcoder-archive/topcoder-platform-topcoder-x-processor#35

Merged

Fetch issue with queryone. #257

Merged

afrisalyp added tcx_ReadyForReview and removed tcx_Assigned labels Nov 26, 2019

jmgasper added tcx_Feedback and removed tcx_ReadyForReview labels Nov 26, 2019

This was referenced Nov 26, 2019

Fix query issue. topcoder-archive/topcoder-platform-topcoder-x-processor#36

Merged

Fix query issue. #262

Merged

jmgasper added tcx_OpenForPickup and removed tcx_Feedback labels Nov 27, 2019

jmgasper unassigned afrisalyp Nov 27, 2019

jmgasper closed this as completed Nov 27, 2019

jmgasper reopened this Nov 27, 2019

jmgasper assigned afrisalyp Nov 27, 2019

jmgasper added tcx_Assigned tcx_FixAccepted and removed tcx_OpenForPickup labels Nov 27, 2019

jmgasper closed this as completed Nov 27, 2019

jmgasper added the tcx_Paid label Nov 27, 2019

This was referenced May 28, 2021

[$250] DynamoDB Timeout #396

Closed

Optimise DynamoDB usage #397

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[$100] Update `scanOne` to query #256

[$100] Update `scanOne` to query #256

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 27, 2019

jmgasper commented Nov 27, 2019

jmgasper commented Nov 27, 2019

[$100] Update scanOne to query #256

[$100] Update scanOne to query #256

Comments

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 26, 2019

afrisalyp commented Nov 26, 2019

jmgasper commented Nov 27, 2019

jmgasper commented Nov 27, 2019

jmgasper commented Nov 27, 2019

[$100] Update `scanOne` to query #256

[$100] Update `scanOne` to query #256