bug(schematics): execution in a CWD w/ many directories and files causes a major delay or crash

[alexeagle]

From @Splaktar on October 31, 2017 3:24

Bug Report or Feature Request

- [x] bug report
- [ ] feature request

Area

- [ ] devkit
- [x] schematics

Versions

npm: 5.5.1
node: v8.8.1
@angular-devkit/[email protected]
@angular-devkit/[email protected]

Repro steps

1. cd into a directory that contains a lot of Angular CLI projects (i.e. your git/ directory where all of your projects are and where you want to create a new application)
2. run schematics application --name tmp --directory tmp

The log given by the failure

Most of the time there is no error, but the generation of a new app may take 1-3 minutes!

In extreme cases where you have a large number of projects, the heap will fill up resulting in this:

==== JS stack trace =========================================

Security context: 0x2ef8d4cfb51 <JS Object>
    1: new constructor [fs.js:150] [pc=0x344fa7d1aef4] (this=0x287332d90369 <JS Object>,dev=16777220,mode=33152,nlink=1,uid=1759047591,gid=1293641148,rdev=0,blksize=4096,ino=2858228,size=253,blocks=8,atim_msec=0x286887bfffe9 <Number: 1.50854e+12>,mtim_msec=0x286887bfffb9 <Number: 1.47277e+12>,ctim_msec=0x286887bffef9 <Number: 1.5082e+12>,birthtim_msec=0x286887bffe21 <Number: 1.47277e+12>)
    6:...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::factory::NewByteArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 5: v8::internal::factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [/usr/local/bin/node]
 6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::MacroAssembler*, v8::internal::CompilationInfo*) [/usr/local/bin/node]
 7: v8::internal::LChunk::Codegen() [/usr/local/bin/node]
 8: v8::internal::OptimizedCompileJob::GenerateCode() [/usr/local/bin/node]
 9: v8::internal::Compiler::FinalizeOptimizedCompileJob(v8::internal::OptimizedCompileJob*) [/usr/local/bin/node]
10: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [/usr/local/bin/node]
11: v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node]
12: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
13: 0x344fa77092a7

Desired functionality

It is understandable that the execution of most schematics would scan all of the files and directories in the CWD (current working directory) as it may assume that it is in an app and needs to know the structure.

But when creating a new app, there is no need to do this scanning in the CWD. Perhaps in the directory specified by --directory, but it is likely empty in most cases unless --force is being used.

Mention any other details that might be useful

Running the same command in an empty directory usually completes in just a few seconds (compared to 1m+ in a dir w/ Angular projects in it).

Copied from original issue: angular/devkit#248

[alexeagle]

From @Splaktar on November 2, 2017 21:35

This is caused by https://github.com/angular/devkit/blob/master/packages/angular_devkit/schematics/bin/schematics.ts#L171-L172. There doesn't seem to be any way for schematics to tell at this point what kind of schematic will be executed.

Should a --root arg be added here that can override process.cwd() if supplied?

[alexeagle]

From @Splaktar on November 2, 2017 21:37

@hansl @Brocco I can make a PR for this, but I would like some feedback on the design and whether this would be an acceptable solution or not. I'd also be happy to look into an alternative solution. Thank you.

[alexeagle]

From @hansl on November 3, 2017 11:26

Please note that we don’t scan (or shouldn’t) the directories now unless the schematics requires it (e.g. list files). I think this might be coming from the sink where we apply those changes?

Basically the issue is that schematics should not care about creating an empty host for some, but the code outside the schematics itself should not force a scan. And the schematics itself should be able to listcertains dirswithout scanning the whole structure.

I need to take a quick look. But if you want I can tell you where the issue is and you can fix it.

[alexeagle]

From @Splaktar on November 3, 2017 16:19

I implemented a change locally to remove process.cwd() from the following lines https://github.com/angular/devkit/blob/master/packages/angular_devkit/schematics/bin/schematics.ts#L171-L172. I replaced it with the --directory argument used for the application schematic and the issue was resolved. The execution time for my local test case dropped from 1.2 minutes to 5 seconds. Of course that doesn't work for the rest of the schematics which depend on being executed from within a project.

Since not all schematics have a --directory option, I proposed something like --root, but I'm open to options and would love an option which didn't require an additional argument.

[alexeagle]

From @Splaktar on November 6, 2017 18:43

There is another related issue with the --directory argument of the application schematic. If you do something like schematics application --name tmp --directory ../tmp it will fail with Path "/../tmp" is invalid.. This is because you are asking it to generate files above the CWD in the file tree, which is not hooked up to the sink.

[alexeagle]

From @clydin on November 27, 2017 16:22

@Splaktar are you still experiencing the performance issues with the latest versions?

[Splaktar]

@clydin this is no longer possible to reproduce with the change to require creating a workspace first and the removal of the --directory argument from the application schematic. This was non-trivial to try to reproduce with a recent version of the Schematics CLI due to a lack of documentation described here.

This can be closed.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}