[$350] Performance for hokienick

[jmgasper]

@atelomycterus - In prod, Nick (hokienick) is saying that forums are taking 30+ seconds to load as he browses from one page to another. I don't see that with my account, so I'm wondering if there's something strange set up with his account that could cause that? Can you investigate when you have some time please? I'll let you know if I hear of anyone else that has performance issues.

[jmgasper]

Challenge https://www.topcoder.com/challenges/82eee8b6-b34e-4178-af61-2d61f25a605c has been created for this ticket.

This is an automated message for ghostar via Topcoder X

[jmgasper]

Challenge https://www.topcoder.com/challenges/82eee8b6-b34e-4178-af61-2d61f25a605c has been assigned to obog.

This is an automated message for ghostar via Topcoder X

[atelomycterus]

@jmgasper 1. User with the hokienick username is not found in Vanilla (PROD).

2. Could you specify which pages exactly?

[atelomycterus]

@jmgasper Maybe he's using the old forum?

[jmgasper]

@atelomycterus - Sorry about that - his TC handle is nick.castillo. Not sure what pages, but sounds like most of them.

[atelomycterus]

@jmgasper

I don't see that with my account, so I'm wondering if there's something strange set up with his account that could cause that?

All Topocder accounts have the same setup. I've checked the user ban list just in case and so on. The nick.castillo account looks fine.

I don't see any performance issues with my account as well. Let me know If there is any additional details from other users.

Without any statistics and logs, it is difficult to understand where the problem might be.
We need a tool like Tideways to monitor Vanilla’s performance on PROD.
So we could see the response time and downstream layer performance from SQL, HTTP, Memcached and DB.

Tideways uses sampling to keep the overhead of production monitoring and profiling small.
We can influence the amount of tracing data collected with the sample rate configuration, which is a percentage value used
for randomly deciding which PHP requests are traced by Tideways and which are not.
By default the sample rate is configured at 25% of all PHP requests.
Use the 'basic' mode for monitoring. We don't need to collect profile data/build callgraph traces and so on.

Now Tideways is not deployed on PROD. Let me know if we can use Tideywas.
I'll update a dockerfile.

[jmgasper]

@atelomycterus - We're discussing this. A couple questions:

1. Does this raise costs with Tideways to monitor another application?
2. Are there any potential security issues?

[atelomycterus]

@jmgasper We have a trial account. I see the error message:

We can simply register a new trial account and monitor Vanilla for a month. Just need to update the API KEY and deploy on the environment.

Does this raise costs with Tideways to monitor another application?

The basic plan support 2 envs:

Are there any potential security issues?

The product has been on the market for about 5 years. I have not heard of any security issues.

The company are documenting Technical and Organizational Measures to ensure security in Tideways:

Security

What Does Tideways Technically Do to Prevent Accidental Transmission of Personal or Sensitive Data as Part of Regular Error / Performance Data?

[jmgasper]

@atelomycterus - I think we pay for this, for the dev environment, correct? So we should be good to go since we get 2 services in 2 envs (one dev, one prod)?

[atelomycterus]

@jmgasper I also think the account has been paid but I am confused by this message. I think this is about my account.

I was able to create a new env ( dev) for Topcoder account. So it looks working. Now Tideways configuration has 2 envs:

I've updated dockerfiles and .circleci/config.yml to support both envs. After deploying on DEV/PROD, data should be displayed in 'dev'/'production' tab:

Please apply PR-#537 and ping me after deploying on DEV, I'll verify it.

Thanks!

[jmgasper]

@atelomycterus - Deploying now, thanks. We'll be deploying to prod in a few days, so if there's anything else we need to do there, other than merge from develop --> master, please let me know.

[atelomycterus]

@jmgasper I still don't see any data for dev. I'll check logs.

Before deploying on PROD, we need to fix #523. Otherwise, watching count will be not correct.

Keep you updated.

[atelomycterus]

@jmgasper Updated Tideways configuration.
Before I configured one dockerized Tideways daemon for all envs but I don't have permissions to see logs in dependecy containers and understand why it didn't work. So I made it simpler and Tideways daemon is installed in Vanilla container. Now we can see logs and check it.

Please apply PR-#543. Thanks!

Vanilla logs:

Tidaways Daemon status from UI:

Log file in Vanilla container - /var/log/tideways/daemon.log

[atelomycterus]

@jmgasper Need help to create a service. I don't have admin permission. The service should be named 'web' (https://app.tideways.io/o/Topcoder/Discussions/settings/services).

The service is created automatically for a trial account only.

[jmgasper]

@atelomycterus - The service should be created now:

[jmgasper]

@atelomycterus - Reported by @sdgun:

https://vanilla.topcoder-dev.com/discussion/3130/perf-test

Is taking a long time to load. Used to be max 6 seconds, now about a minute - not specific to any particular user. I see a similar issue in an incognito window, without logging in. Is that something we see in Tideways?

[atelomycterus]

@jmgasper
I've checked env params, Tideways API Key has been configured on DEV. The checklist of all the potential problems that are the cause for no data being transmitted to the Tideways backend:
-Check the daemon if it is running.
-To find out if data gets transmitted, take a look at the tideways-daemon logfile. Check connection from docker to the Tideways server.
-Add TIDEWAYS_DAEMON_EXTRA="–debug". Use the daemon in debug mode to get more information about what is happening.

https://vanilla.topcoder-dev.com/discussion/3130/perf-test
Is taking a long time to load. Used to be max 6 seconds, now about a minute - not specific to any particular user. I see a similar issue in an incognito window, without logging in. Is that something we see in Tideways?

The discussion has 6300+ comments. Before the flat mode was used by default. 30 comments were only displayed.
Now all comments are displayed because the tree mode is used by default. I wrote about it if a discussion has a lot of comments it'll cause a performance issue. As a solution if there are 50/100/200 comments then the flat mode should used by default.

Is that something we see in Tideways?

Yes, we can see the response times and other stats when data is transmitted to the Tideways backend.

[atelomycterus]

@jmgasper Need redeploy it. The last job was failed. This commit includes Tideways changes

The deployed container doesn’t include the last changes with Tideways:

Thanks!

[atelomycterus]

@jmgasper Please apply PR:

• Issues-519: added debug flag #545
• Issues-519: read tideways logs forums-plugins#84

Thanks!
The Tideways log path is hardcoded for security reason.

if the daemon is working then you should see the next log records. Collected XXX means data collected +transmitted to the Tideways backend:

[atelomycterus]

@jmgasper Vanilla logs are available. In Vanilla logs:

2021-04-07 10:49:59,002 INFO spawnerr: command at '/opt/docker/bin/service.d/tideways.sh' is not executable
2021-04-07 10:49:59,016 INFO gave up: tideways entered FATAL state, too many start retries too quickly

I'll updated dockerfiles to ensure files are executable in a container. Please apply PR-#546. Thanks!

[jmgasper]

Closing this out since Tideways integration is working now. Will deploy to prod tomorrow.

[jmgasper]

Payment task has been updated: https://www.topcoder.com/challenges/82eee8b6-b34e-4178-af61-2d61f25a605c
Payments Complete
Winner: obog
Copilot: ghostar
Challenge 82eee8b6-b34e-4178-af61-2d61f25a605c has been paid and closed.

This is an automated message for ghostar via Topcoder X

[jmgasper]

@sdgun - I manually validated this in Tideways and am marking this as Prod QA Pass, just so it's tied to the milestone.