[$150] Getting 504 error when creating discussions and comments

[sdgun]

Steps
Log in and create a new discussion/comment

Getting Timeouts

[jmgasper]

@atelomycterus - anything we can do to debug this?

[jmgasper]

Challenge https://www.topcoder.com/challenges/b9a974d7-3594-4893-ba79-2e79e6df2a37 has been created for this ticket.

This is an automated message for ghostar via Topcoder X

[atelomycterus]

@jmgasper Duplicate #381 504 (Gateway Timeout) status code indicates that the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request.

anything we can do to debug this?

I ran it with the profiler locally with old Vanilla DB (~500+categories) . It generates 900+MB files per a request for discussion and 400+MB for comment. It works slowly with enabled profiler. With enabled profiler adding a discussion/comment 1.3-2.7 mins, without - 300 ms-7 secs. The greater the depth of the category, the more recalculations and changes in the cache and the execution time increases. I'll ran with 2000+/5000+ categories locally and test it.

A large number of categories become a concern. Are there any applications (using UI or Vanilla API) that insert a large amount of data? Maybe there was testing on the QA server? Adding data by sql script is not correct, because category are implemented using nested set model. If not, then no problem, I'll write PHP scripts that will generate the data.

Now in DEV 1700+ categories if we ran it with the profiler it'll be not less 1,2-1,5GB per request for creating a discussion, maybe 2-3 GB or more.

I am going to work on it today. Keep you updated.

[jmgasper]

Challenge https://www.topcoder.com/challenges/b9a974d7-3594-4893-ba79-2e79e6df2a37 has been assigned to obog.

This is an automated message for ghostar via Topcoder X

[jmgasper]

@atelomycterus - I've asked for a database backup to be provided. If we had that many challenges, that would be that many categories though, correct, since each challenge will have multiple categories?

[atelomycterus]

@jmgasper Thanks!

3 categories are added for a development challenge:

1 category is added for a design challenge:

So I am going to test first with a 2k/5k/10k thousand categories locally. In any case, I will add metrics to the code so that we can look at execution time in log files/debug bar (without a profiler). The profiler cannot be run on a live system (PROD).

Type=Heading - This type as an option will reduce the number of nested categories. We will keep this as a fallback. We should try to do something with the current version.

Category Types

Type=Discussions: A basic category where users can post discussions. Only categories set to the Discussions type can contain discussions. All other types of categories will not be selectable when choosing where to add a new discussion.

Type=Nested: No discussions can be added directly inside a Nested category. Creates a clickable heading that only contains more categories.

Type=Heading. We don't use it.
Unclickable headings that help with forum organization. On the homepage, they will display differently according to your theme. Generally they are bold, and will split your categories into sections:

Type=Flat. We don't use it.

This is a paginated directory structure that can handle thousands of categories. They are sorted alphabetically by default, and cannot be manually arranged.This is typically only recommended for very large communities with a great deal of categories.

[jmgasper]

@atelomycterus - Ok, thanks. The number of challenges will grow as we use it more in more in prod, so if we can address the issue now, that'll be good, thanks.

[atelomycterus]

@jmgasper Do you know if there was a restart/re-deploy of any services (ngnix, memcached, DB or so on) in DEV today? When this issue was created, adding a discussion/comment was very slow and there was 504 error. 1-2 hours ago I posted 9 comments + 7 discussions, no 504/502 errors.

[jmgasper]

@atelomycterus - I don't know, but I've asked Topcoder if they know. @rootelement ☝️

[jmgasper]

@atelomycterus -From Topcoder:

Nothing was done to forums in dev to my knowledge

[sdgun]

@jmgasper Duplicate #381 504 (Gateway Timeout) status code indicates that the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request.

anything we can do to debug this?

I ran it with the profiler locally with old Vanilla DB (~500+categories) . It generates 900+MB files per a request for discussion and 400+MB for comment. It works slowly with enabled profiler. With enabled profiler adding a discussion/comment 1.3-2.7 mins, without - 300 ms-7 secs. The greater the depth of the category, the more recalculations and changes in the cache and the execution time increases. I'll ran with 2000+/5000+ categories locally and test it.

A large number of categories become a concern. Are there any applications (using UI or Vanilla API) that insert a large amount of data? Maybe there was testing on the QA server? Adding data by sql script is not correct, because category are implemented using nested set model. If not, then no problem, I'll write PHP scripts that will generate the data.

Now in DEV 1700+ categories if we ran it with the profiler it'll be not less 1,2-1,5GB per request for creating a discussion, maybe 2-3 GB or more.

I am going to work on it today. Keep you updated.

@atelomycterus For QA testing I created comments from the UI using a selenium script, but I didn't create any categories. I just simulated the user behavior through UI, didn't execute any sql scripts. Here are the comments I created https://vanilla.topcoder-dev.com/discussion/3130/perf-test. Also I see even though it gives time outs, the discussions are being created.

[atelomycterus]

@sdgun Thanks for info! A few questions:

• What was the delay between requests?
• How many comments were added before 504 error occurred?

Also I see even though it gives time outs, the discussions are being created.

I think that if you create discussions with a selenium script, then the same problem will be.

Could you share this script? Thanks!

[sdgun]

@atelomycterus I have added 5.7K comments and the last comment was added Feb 12 2021 at 04:33 PM. The code is attached. The gap between comments were 6 seconds. But please note all these 5.7k were not added the same day. I started running the test on 6th Feb and ran it time to time till 12th Feb. You can see this in https://vanilla.topcoder-dev.com/discussion/3130/perf-test

ChallengeForums.zip

[jmgasper]

@sdgun - Is it alright if I put those test sources into Git? I'd like to keep those captured somewhere for reference, thanks.

[sdgun]

@jmgasper No problem

[atelomycterus]

@jmgasper Should be fixed. See #381.

[jmgasper]

Payment task has been updated: https://www.topcoder.com/challenges/b9a974d7-3594-4893-ba79-2e79e6df2a37
Payments Complete
Winner: obog
Copilot: ghostar
Challenge b9a974d7-3594-4893-ba79-2e79e6df2a37 has been paid and closed.

This is an automated message for ghostar via Topcoder X

[sdgun]

Verified by creating multiple discussions and comments, didn't observe timeouts again.

[sdgun]

Verified in Production