Rebuild is way slower and stop working after several rebuilds

[montella1507]

Affected Package

Propably compiler 11.2.5

We have migrated from:

  "@angular/animations": "^11.0.9",
    "@angular/cdk": "^11.0.4",
    "@angular/common": "^11.0.9",
    "@angular/compiler": "^11.0.9",
    "@angular/core": "^11.0.9",
  "@angular/forms": "^11.0.9",
"@angular/platform-browser": "^11.0.9",
    "@angular/platform-browser-dynamic": "^11.0.9",
    "@angular/router": "^11.0.9",
    "@angular/service-worker": "^11.0.9"
  "@angular-devkit/build-angular": "^0.1100.7",
  "@angular/cli": "^11.0.7",
    "@angular/compiler-cli": "^11.0.9",
    "@angular/language-service": "^11.0.9"

TO

 "@angular/animations": "^11.2.5",
    "@angular/cdk": "^11.2.3",
    "@angular/common": "^11.2.5",
    "@angular/compiler": "^11.2.5",
    "@angular/core": "^11.2.5",
  "@angular/forms": "^11.2.5",
 "@angular/platform-browser": "^11.2.5",
    "@angular/platform-browser-dynamic": "^11.2.5",
    "@angular/router": "^11.2.5",
    "@angular/service-worker": "^11.2.5",
  "@angular-devkit/build-angular": "^0.1102.3",
"@angular/cli": "^11.2.3",
    "@angular/compiler-cli": "^11.2.5",
    "@angular/language-service": "^11.2.5"

Is this a regression?

YES

Description

We have migrated from angular 11.0.9 to 11.2.5. Our application is pretty big 1300components, 170 NgModules, 70 lazy loaded chunks.

After migrations almost all rebuild times took 6-10x more time and what is worse..

1. Rebuild starts afer XYZms (sometimes several seconds) after save of the file
2. Rebuild stopped working after 2-3 saves (rebuilds) , even when change is a breaking change (return 0 to return 1 in component method).

I am willling to help with debug (profile node process / generate and send any log, however it may be imposible to prepare minimal repro :-/ )

[montella1507]

Some screenshots from profilling + profiling profile:

vscode-profile-2021-03-10-21-07-08.zip

[JoostK]

Hi @montella1507,

thanks for the report. We have landed a large change in 11.2.5 with respect to incremental rebuilds, however the outcome should be that it's now faster than before, not slower. It could be that you're hitting a case where the compiler is unable to prove that work can be reused, in which case it may request more code to be regenerated than before.

I'll have a look at the profile dump, thanks for sharing 👍

[montella1507]

Thank you very much. I am here to help if possible.

[JoostK]

Ah, gotcha. You're using JIT compilation and the new Ivy Webpack integration that shipped in 11.1 is incorrectly using the AOT compiler's judgement on which files need to be emitted. But since AOT output is never requested, the AOT compiler will continue to indicate that everything needs to be emitted.

If you can enable AOT, then I suggest you do! Otherwise, I'd suggest to disable the new Ivy Webpack plugin in the CLI for now by setting environment variable NG_BUILD_IVY_LEGACY=1.

I'll transfer this to the CLI repo for now.

[montella1507]

Cool :-) I will try with AOT and will come back with report :-)

[montella1507]

vscode-profile-2021-03-10-21-36-15-deep-component-change.zip

Rebuild propably will not stop working, however rebuild times still super slow :-(

[JoostK]

Ok, thanks for the additional profile. I do indeed see a lot more emit activity than I'd expect, but I can't tell why that would be the case.

Another observation is that the cyclic import check takes 17s, so you may want to disable that in devbuilds (there's a setting in angular.json, not sure what it's called from the top of my head—edit: it's showCircularDependencies).

[JoostK]

@montella1507 Could you also try with NG_BUILD_IVY_LEGACY=1? I doubt that it will help us diagnose the cause of the issue, but hopefully it does allow you to get back reasonable incremental rebuilds.

[montella1507]

@JoostK i can confirm JIT mode stop working after few rebuilds. AOT keep working.

"showCircularDependencies": false, changes nothing i think (not sure if it is not just disabling warnings to console) , however.

When i monkeypatch isCyklic method to return directly FALSE. Rebuild times are little bit better (IVY all, AOT all, cyclic off):
vscode-profile-2021-03-10-22-26-25.zip

I will try to explain our portal to provide you more information what may be the cause.

It is big CRM where you need cross references between 2 modules, for example:

OrderDetailComponent from OrderModule (lazy loaded)
CustomeDetailComponent from CustomerModule (lazy loaded)

We need to show OrderDetail in CustomerDetail and vice-versa CustomerDetail in OrderDetail component.
So we need:

// Customer-detail.html
<OrderDetail></OrderDetail>

and vice-versa
// OrderDetail
<CustomerDetail></CustomerDetail>

This is NOT possible when components are not declared in same module and you cannot Import module B in module A and vice versa. And -> we want to have both modules lazy loaded.

So we usually use our custom DynamicComponent (lazy load module, compile, use component factory and attack to viewcontainerref).

So there are dependencies between modules:
A -> dynamic component -> B
B -> dynamic component -> A

I can imagine this may be big problem for modules buildign dependency tree because it is "lazy" cyklic dependency.
However i dont think there is another solution (i simplified this example)

[JoostK]

Thanks for the additional information. I suspect that it's just Webpack's dependency graph that makes it believe that everything needs to be reemitted, which is introducing a huge bottleneck as emit is the slowest of all phases. The improvements to 11.2.5 are only concerning Angular's involvement in the process, which used to be suboptimal, but given that your JIT recompiles were also slow means that this is not the relevant here (my initial comment about JIT mode being affected by the AOT compiler turned out to be inaccurate). So that leaves a new problem to be investigated with respect to the dependency graph that Webpack is using to drive recompiles.

[montella1507]

set NG_BUILD_IVY_LEGACY=1
ng serve...

mode: JIT
cyclic: DISABLED

rebuild stiill like 5-6 times slower than in previous NG version (but it works atleast even after 10th change/rebuild).
Interesting thing here is, that first few (1-2) rebuilds it changes 4+ chunks, but 3rd+ rebuild correctly rebuilds only one. (so rebuild time drops from 30sec to 7sec).

However i am not able to profile (profiller / debugging and entire build will crash)

[montella1507]

vscode-profile-2021-03-10-23-09-20.zip

Ah finaly:
NG_BUILD_IVY_LEGACY=1
MODE: JIT
cyclic: disabled

So most time takes source map generation i think.

[JoostK]

Thanks. Yeah profiling a long session will take down the Chrome Inspector at some point, it's not uncommon for me to see Chrome take 10GB of memory to show a profiling session.

The last profile you shared does appear to still use the new Ivy pipeline, not the legacy one. Not sure why that would be the case, your set NG_BUILD_IVY_LEGACY=1 should be adequate.

In the meantime we have an idea of what is happening; it's indeed on the CLI side where we can do some experiments to see if things can be improved.

I'm calling it a day now 👋

[montella1507]

@JoostK hi, btw i have made additional benchmarks on 11.0.9 to measure circular-dependency-plugin + sourcmap plugin effects on build time.. here are the results:

When circular-dependency detection is enabled, build and rebuild takes 5-8x more time. I think it has serious issues with async circular dependencies or something...

And yes, "showCircularDependencies": false is enough. Will do the same benchmarks on current version of angular / compiler later (when this issue is closed or requested by anyone explicitely)

[montella1507]

[JoostK]

@montella1507 We think to have found the culprit, it was fixed on master in #20236 and will be ported to patch so should be part of next week's release. I'll share a snapshot build for patch once that's available.

[montella1507]

Thank you very much. :-)

I have realized rebuild is affected by assets too - created this issue - maybe it is duplicit to this one - feel free to close please if neccessary.

#20244

[JoostK]

Snapshot build is here: https://github.com/angular/ngtools-webpack-builds/tree/11.2.x (see top of readme)

[clydin]

That snapshot build includes fixes for the repeat JIT build crash issue (fix via #20237) and the rebuild performance (fix via #20247). If you have the opportunity, please give the snapshot build a try and let us know if you are still encountering issues.

Also, the asset issue is due to a third-party dependency used by the Angular CLI so a separate issue is appropriate. Some more details regarding the asset performance in serve can be found in a comment in that issue.

[montella1507]

Fortunately, i have jus got enabled 4H of Zack Snyder's Justice League CUT so i have enough time to profile on second display :-) TS. is 4.1.5 Profiles will provide soon.

[montella1507]

nosm-nocirc-aot-multiple-components.zip
here we go.

EDIT: those are JIT profiles.

[JoostK]

Thanks, those are all JIT though. A single variant would do, they should be similar enough.

[montella1507]

aot.zip

[montella1507]

Thanks, those are all JIT though. A single variant would do, they should be similar enough.

I have uploaded AOT.zip with aot profiles, let me know if i can help you with anything else. If you think it is already solved - and you okay with results, i can close the issue (from my POV)

[JoostK]

@montella1507 Thanks for the profile. It's mostly what I was expecting from the earlier profiles and the changes that have been made since.

If you feel like it, can you run another experiment for me? Based on your profiling output I have been experimenting with cycle analysis (not the expensive step that you can disable, but an internal Angular compiler analysis step needed for proper output) and I'd like to know what the impact of that would be.

The build artifacts are here. In particular, you'll need @angular/compiler and @angular/compiler-cli. These are based on 12.0 preview builds and only TS 4.2 is supported so you could either update TS to 4.2.x or set disableTypeScriptVersionCheck: true in angularCompilerOptions in a tsconfig.json file. If you encounter too much trouble with this I can probably cherry-pick it to 11.2.x, please let me know if that's desired.

Using those build artifacts you should get slightly better AOT times (I estimate ~400ms less). Next, I'm interested in a more extensive optimization of which I have no idea how effective it is. This is hidden behind the NG_EXPERIMENTAL_CYCLES environment variable; if you set that to 1 it will take a different approach for cycle detection. I'd also like how that compares. If you can, please share AOT / NO CIRC / NO SM profiles using those build artifacts with and without NG_EXPERIMENTAL_CYCLES=1.

Instructions for installing the build artifacts can be found here.

---

Unrelated to the above, the CLI is ramping up for the adoption of Webpack 5 in v12. Your project is a gem in terms of the insights it provides, so if you could have a go at Webpack 5 with the CLI's latest next releases that would also be super valuable. I think that using Webpack 5 does require a Yarn resolutions rule, though, so I'm not sure if testing that is feasible for you.

[montella1507]

Will do at weekend.

[JoostK]

Oh and one more thing that we noticed in the profile that deserves further research: in AOT mode the Angular compiler does template type-checking by feeding the Angular templates into TypeScript. This is done by injecting additional source files into a temporary TypeScript program. Creating that temporary program should be efficient as it's mostly identical to the original. In your profile however, we're seeing a codepath that indicates that TypeScript has determined that it can't reuse everything from the original program, which incurs an extra cost (circa 500ms as taken from your profiles.). So to verify: are the edits you're making entirely redundant, such as whitespace only changes? They probably are, in which case we'd need to do some experimenting to figure out if we can avoid that extra cost (we have tests that tell us this should work, but perhaps it's missing a case).

[DavidDudson]

Hi everyone! Thank you all for sorting these issues out, our team is currently switching from 9 => 11 and are holding off for these performance improvements at present. Do we know roughly what compile times are, comparable to Angular 9/10, rather than just earlier Angular 11 versions? I'll be doing some experimentations next week profiling our codebase with the lastest angular version.

[montella1507]

Oh and one more thing that we noticed in the profile that deserves further research: in AOT mode the Angular compiler does template type-checking by feeding the Angular templates into TypeScript. This is done by injecting additional source files into a temporary TypeScript program. Creating that temporary program should be efficient as it's mostly identical to the original. In your profile however, we're seeing a codepath that indicates that TypeScript has determined that it can't reuse everything from the original program, which incurs an extra cost (circa 500ms as taken from your profiles.). So to verify: are the edits you're making entirely redundant, such as whitespace only changes? They probably are, in which case we'd need to do some experimenting to figure out if we can avoid that extra cost (we have tests that tell us this should work, but perhaps it's missing a case).

i do everytime the same - editing string / number value. eg. return "george"; >> return "georgedcjwekwejkfk"; etc. :-) I can do bigger changes, if you wish. But NO, not at all, my changes are NOT redundant.

[montella1507]

Hi everyone! Thank you all for sorting these issues out, our team is currently switching from 9 => 11 and are holding off for these performance improvements at present. Do we know roughly what compile times are, comparable to Angular 9/10, rather than just earlier Angular 11 versions? I'll be doing some experimentations next week profiling our codebase with the lastest angular version.

as far as i know... Angular rebuild times went worse and worse from 8 -> 11 in all projects i participate to... HOWEVER... latest version is deal breaker + we have realised how BAD is circular detection in our bigger projects (rebuild times dropped by 80% by disabling that check)

[montella1507]

@JoostK BTW i am not changing HTML templates (split from TS file) so maybe you are right, my changes are redundant from POV of template compiling... Do you wish CPU profiles, when i change template (html) files? Let me know.

[JoostK]

Nope, the current profiles are sufficient. My question was to ensure that you're indeed only making trivial changes. TypeScript has some logic in tryReuseStructureFromOldProgram to determine to what extend the structure of a previous program can be reused. That should be Completely for the template type-check program, but your profiles show it's not. I can't tell whether it's Not or SafeModules (the other variants of structure reuse). It would be super helpful to know which code-paths are hit there that prevent complete structure reuse, but getting that information out requires some debugging and that's not trivial for me to guide you through. The simplest way would be to hack in a lot of console.log's into all places where structureIsReused is set or the function early-returns, or to attach a debugger and set breakpoints at these places to find out where it's hit. The console.log approach is probably much simpler to share the results with, though...

[montella1507]

Look, we have 2 options...

1. I can put console.log anywhere you want
2. I can provide Remote desktop for you and you can do it yourself (public IP / powerfull PC / powerfull connection). You are in the same timezone i guess (Netherland / UTC+1) so it should not be a problem.. We can do that this weekend or other day if you willing to.

I have no problem to spend several hours to it :-))

[JoostK]

Let's keep it offline :-) I am on Angular's Discord as Joost#9883 if you want to chat.

[montella1507]

Discussed on discord. Will let this issue open, @JoostK close when you think all is solved :D

[gabrielguerrero]

Hi I have a similar issue our build went from in angular 11.0.3 ng serve started in around 1 min is a big project after migrating to 11.2.9 the ng serves takes 3mins to start, I started downgrading libs to understand what libs was causing the issue, in the end the problem seems to be related to @angular-devkit/build-angular, I manage to run the project with everything updated to 11.2.9 except to that library which I set to version 0.1100.7 and runs again in 1 min, if I change to any version after that one the time is 3mins again. I college tried with 12 and said that was faster but still slower than before.
I also generated chrome-profiler-events.json took around 30 min to generate is 500mb and chrome://tracing fails to load it, I could post it here if interested.
Also Im thinking as a workaround keep angular 11.2.9 with angular-devkit/[email protected] till 12 is release, could there be a problem with this?, it seems to run fine

[montella1507]

TBH i think i should have closed this issue, because it was resolved.

[gabrielguerrero]

@montella1507 sorry I think is fine to close it, nx just released a new release , and it seems to be working fine with it , build is back again in 1 min using latest angular-build, I think they have a custom builder that wraps yours and not sure exactly what went wrong there, in their change log looks like they fixed a bug were ngcc was been called on incremental builds, I though it was related to this because like I said the issue seem to be related to angular-build in my test. Anyway sorry to bother you, I spend so many hours investigating this , thanks anyway.

[JoostK]

In the meantime we have also addressed another slow path that became clear from @montella1507's profile, albeit only for v12. Closing this issue as not further actionable.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}