[$250] DynamoDB Timeout

[jmgasper]

@afrisalyp - We're seeing timeouts now in other operations (not just git access links). See below for a log when a user tried to archive a project.

What we're thinking is that:

1. Maybe we should use query instead of scanOne?
2. Are the indices correct for how we are querying / scanning? I see we have an index on tcDirectId, but do we need any other indices to speed things up?

From @rootelement :

i don’t believe this is correct:

https://github.com/topcoder-platform/topcoder-x-ui/blob/develop/src/common/db-helper.js#L85
all() ?

you’re trying to pull the whole table and filter it?

i know query is better than scan, but scan might need a better hash and range key based on the model you’re trying to pull from

2021-05-25T08:08:07.246-04:00	error: DynamoDB scanOne error Error: Wait for Active timed out after 180000 ms.
2021-05-25T08:08:07.246-04:00	error: Error happened in undefined
2021-05-25T08:08:07.246-04:00	Error: Wait for Active timed out after 180000 ms.
2021-05-25T08:08:07.246-04:00	at Timeout.waitForActive [as _onTimeout] (/topcoder-x-ui/node_modules/dynamoose/dist/Table.js:198:36)
2021-05-25T08:08:07.246-04:00	at listOnTimeout (internal/timers.js:554:17)
2021-05-25T08:08:07.246-04:00	at processTimers (internal/timers.js:497:7)
2021-05-25T08:08:07.247-04:00	error: Error happened in system
2021-05-25T08:08:07.247-04:00	TypeError: Cannot read property 'count' of undefined
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/src/common/db-helper.js:91:29
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/node_modules/dynamoose/dist/Scan.js:267:24
2021-05-25T08:08:07.247-04:00	at _rejected (/topcoder-x-ui/node_modules/q/q.js:864:24)
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/node_modules/q/q.js:890:30
2021-05-25T08:08:07.247-04:00	at Promise.when (/topcoder-x-ui/node_modules/q/q.js:1142:31)
2021-05-25T08:08:07.247-04:00	at Promise.promise.promiseDispatch (/topcoder-x-ui/node_modules/q/q.js:808:41)
2021-05-25T08:08:07.247-04:00	at /topcoder-x-ui/node_modules/q/q.js:624:44
2021-05-25T08:08:07.247-04:00	at runSingle (/topcoder-x-ui/node_modules/q/q.js:137:13)
2021-05-25T08:08:07.247-04:00	at flush (/topcoder-x-ui/node_modules/q/q.js:125:13)
2021-05-25T08:08:07.247-04:00	at processTicksAndRejections (internal/process/task_queues.js:75:11)

Standards

https://platform.topcoder-dev.com/model/help/copilots-link

[jmgasper]

Challenge https://www.topcoder.com/challenges/632ec9db-e0e6-46d6-8db2-5141f72c2e08 has been created for this ticket.

This is an automated message for ghostar2020 via Topcoder X

[jmgasper]

Challenge https://www.topcoder.com/challenges/632ec9db-e0e6-46d6-8db2-5141f72c2e08 has been assigned to afrisalyp.

This is an automated message for ghostar2020 via Topcoder X

[afrisalyp]

@jmgasper

I'm not sure that changing to query will solve this issue. Both of the methods calls waitForActive function in dynamoose lib

https://github.com/dynamoose/dynamoose/blob/v1.11.1/lib/Scan.js#L312

https://github.com/dynamoose/dynamoose/blob/v1.11.1/lib/Query.js#L302

But I agree that it's better to migrate to use query. I remember that we've tried to address it.

#256

We've planed it actually :)

#256 (comment)

This is the explanation why we call the all() function.

#256 (comment)

[afrisalyp]

Abiut the error Error: Wait for Active timed out after 180000 ms.

I think it's possible that the error came either from the dynamodb server or from the dynamoose library since we're using old version of dynamoose v 1.

Any idea?

[jmgasper]

@afrisalyp - How much effort would it be to update to a newer version of Dynamoose?

[afrisalyp]

Seems pretty complex https://dynamoosejs.com/other/FAQ/#how-do-i-migrate-from-v1-to-v2

Btw do we have other Topcoder services that run DynamoDB and dynamoose?

[jmgasper]

I'll ask.

[rootelement]

Everything in v5 uses Dynamoose 1, but most other services that require any sort of query have Elasticsearch in front of Dynamoose and the searching is done against that ES index.

…

On Mon, May 31, 2021 at 5:41 PM Justin Gasper ***@***.***> wrote: I'll ask. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#396 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJGMXH6OPKK7WU765JOJULTQP7ARANCNFSM45QHRLKQ> .

[rootelement]

I haven't looked closely at the code, but what I do believe is that the problem stems from us not using properly defined hash and range keys, and using a generic "get model" function in the code. I think if we switched to Query and use a range key defined on the project id, some of this would get a lot faster, but again, not an expert and not positive i'm using the correct vocabulary.

Looking in Dynamo, it doesn't look like there's an index on the projectId on the issues table.

On the challenge resource table, there's a defined range and hash key to partition the table to make the queries faster:

https://github.com/topcoder-platform/resources-api/blob/develop/src/models/Resource.js#L10

It seems like there should be a hash key or range key on projectId

[jmgasper]

@afrisalyp - Please see above. Note that we use dynamoose 2.3 in this API (which is being deprecated): https://github.com/topcoder-platform/evernym-api

[afrisalyp]

@jmgasper @rootelement

Hi,

One important thing that is missed in my challenge submission is cleaning up the unused indexes.
I've raised PRs to fix this.

To summarize, here is the list of indexes that we use.

Project

Index Name	Key	Type	Range Key
RepoUrlIndex	repoUrl	String	archived

Issue

Index Name	Key	Type	Range Key
RepositoryIdIndex	repositoryId	Number	number

User

Index Name	Key	Type	Range Key
UsernameIndex	username	String	type

UserMapping

Index Name	Key	Type	Range Key
TopcoderUsernameIndex	topcoderUsername	String
GithubUsernameIndex	githubUsername	String
GitlabUsernameIndex	gitlabUsername	String
GithubUserIdIndex	githubUserId	Number
GitlabUserIdIndex	gitlabUserId	Number

CopilotPayment

Index Name	Key	Type	Range Key
ProjectIndex	project	String	username

OwnerUserGroup

Index Name	Key	Type	Range Key
IdentifierIndex	identifier	String
GroupIdIndex	groupId	String

OwnerUserTeam

Index Name	Key	Type	Range Key
IdentifierIndex	identifier	String
TeamIdIndex	teamId	String

UserGroupMapping

Index Name	Key	Type	Range Key
GroupIdIndex	groupId	String	gitlabUserId

UserTeamMapping

Index Name	Key	Type	Range Key
TopcoderTeamIdIndex	teamId	String	githubUserName

For the Issue table, we don't have projectId field, but we use repositoryId to query.

Also, we can consider removing indexes that are not listed there in the AWS console.

Thanks.

[jmgasper]

@afrisalyp - Thanks for that. Will this address the timeout / issue noted above?

[afrisalyp]

@jmgasper

Yes. Too many indexes seem likely to cause this issue.

The error message indicates that the table or one of the indexes is not in "Active" status. I have no idea why they become not "Active" suddenly. Maybe dynamodb did some optimization process.

But reducing the number of indexes will decrease the chance of this issue.

[jmgasper]

Payment task has been updated: https://www.topcoder.com/challenges/632ec9db-e0e6-46d6-8db2-5141f72c2e08
Payments Complete
Winner: afrisalyp
Copilot: ghostar2020
Challenge 632ec9db-e0e6-46d6-8db2-5141f72c2e08 has been paid and closed.

This is an automated message for ghostar2020 via Topcoder X