Support mutation testing with angular cli

[nicojs]

We're investigating the possibility of supporting angular in Stryker, a mutation testing framework. I was wondering if it was possible to run test with ng test --watch=false --single-run=false and keep open the karma server so we can fire karma run commands.

The idea would be that we make small code changes and rerun the tests with karma run. Ideally we would be able to distinguish between a failed test and a compile error.

[filipesilva]

I read about what you guys are doing and think it's pretty cool!

I'm not familiar with what karma run does. Can you fill me in?

I think it should be possible to just leave the server open. I mean, you can do that right now by simply running ng test (default watch mode). It will stay open, but reload on code changes.

Ultimately it's karma itself that closes the server so this is mostly a matter of configuring karma itself to stay open.

[nicojs]

@filipesilva thanks for your interest and kind words

I had to look it up myself as well. With karma run you can send a command to the karma server to signal a new test run to start.

This is only part of the solution though, as we also need exact test results (inc code coverage results per test if we can). We manage this with karma because we use karma's pubic api (with a custom coverage reporter) to start the karma server and report the results back to us.

To summarize: we're looking for this kind of api:

import * as ng from 'angular-cli';

// Start a test runner. This in turn should load webpack and other plugins karma needs to run the tests
const testRunner = ng.test({ 
  baseDir: 'sandbox/directory/for/this/testRunner',  
  karmaConfig: { /*override karma config*/ plugins: [{  
      'reporter:rawCoverage': ['type', RawCoverageReporter] 
    }, 
    reporters: [ 'rawCoverage' ]
  }
});

// Forward karma's events so we can collect the data we need
testRunner.on('spec_complete', (browser, spec) => {} ); 
testRunner.on('run_complete', (browser, runResult) => {} ); 
testRunner.on('browser_error', (browser, error) => {} ); 
testRunner.on('raw_coverage_complete', (browser, error) => {} ); // => a custom event that we report from the RawCoverageReporter

 // A way to signal we want a new test run to start
testRunner.run();

That way we can write mutated typescript code to the sandbox directory (directory where we keep a copy of your source code), and signal a new test run (we run multiple sandboxes in parallel for performance reasons).

I think it should be possible to just leave the server open. I mean, you can do that right now by simply running ng test (default watch mode). It will stay open, but reload on code changes.

True, however it doesn't seem to be possible to set singleRun: false in combination with watch: false. Also, this would be only part of the solution as stated above.

Another way of working would be to do all of the work the angular cli is doing ourselves. @Archcry did an awesome proof of concept project here: https://github.com/Archcry/stryker-angular-extra. Here, he basically mimics the angular cli build functionality using the stryker-webpack-transpiler in combination with the stryker-karma-runner. The problem with this approach is that it duplicates the angular-cli test functionality in Stryker, thus i feel it is not maintainable in the long run. However, it might be a sollution if there is a way to pull the entire webpack configuration from angular-cli at runtime (I don't know if that is even possible, with the setup angular-cli is using with regards to webpack and karma).

I hope this clears for you. Don't hesitate to ask more questions. We also have a technical reference page on our website.

Thanks for the help!

[nicojs]

I had some time to think about it in the hope of finding something less intrusive for angular-cli and yet maintainable for us (the Stryker team). I might have a solution.

Would it be possible to use @angular/cli to create a webpack compiler object for us? It would basically look like an ng test with some minor tweaks:

• It should also build the tests (in a separate bundle or not doesn't really matter)
• It should give us the possibility to override the input and output file systems, @Archry's stryker-webpack-transpiler plugin does here: https://github.com/Archcry/stryker-webpack/blob/master/src/compiler/WebpackCompiler.ts#L38
• It should result in a webpack compiler object, so we can could call compiler.run on it when we're mutating the in-memory files.

@filipesilva do you think that's possible? If you give us some pointers, maybe we can help with a PR on your side?

[nicojs]

Also see:
#9089

@filipesilva you've stated about https://angular.io/guide/webpack:

I'm sorry to be the bearer of bad news, but that guide is old and I think is deprecated. I doesn't appear on the sidebar anymore I think.

What does that mean exactly? Is it deprecated because it might break in the future based on current development? Or is it deprecated because you're not maintaining the docs anymore?

We would like to use this approach as we know it works, but we wouldn't want to build a solution based on something that we know we cannot support.

[filipesilva]

I mean that specific guide (the webpack one) is deprecated and not being maintained anymore. I suppose it still works but it's been a while since it was updated. The page is still live but not linked to from anywhere IIRC. We do not maintain the plugins used in that example either.

[nicojs]

@filipesilva thanks for your feedback.

Just one more question. You also stated this yesterday:

Well, the Angular Compiler proper cares about TS files and nothing else. It asks the file system, or a given resource loader, for html/css/sass/etc.

I believe this is the case for the awesome-typescript-loader as well. Does that mean that the awesome-typescript-loader can be a drop-in replacement for the AngularCompilerPlugin (without aot and maybe other performance improvements)?

EDIT:

It sure looks like it is. If I add this line in my webpack.config everything seems to work:

webpackConfig.module.rules.forEach(rule => {
  if (rule.loader === '@ngtools/webpack') {
    delete rule.loader;
    rule.loaders = [
      {
        loader: 'awesome-typescript-loader',
        options: {
          configFileName: path.join(process.cwd(), 'src', appConfig.testTsconfig)
        }
      }, 'angular2-template-loader'
    ]
  }
});

[filipesilva]

awesome-typescript-loader is a TS only compiler, and can be used for JIT only compilation. Only AOT compilation needs @ngtools/webpack. Currently unit tests can only be run in JIT mode anyway, but there will be support for AOT unit testing in the future.

[nicojs]

Cool! This approach seems to be working. I will create an example which uses private angular cli apis to build a webpack config. Once I have that, I will propose a public api here.

@filipesilva thanks for all your afford up until now. 💐

[filipesilva]

No problem at all. I think what you all are doing is super cool 😄

Sorry it wasn't more straightforward to integrate.

[nicojs]

@filipesilva
We are in the middle of merging our webpack transpiler plugin into master and have a working integration test with an angular cli plugin 😌.

See line 3 to 17 in the webpack config. We use private angular-cli apis to reconstruct the webpack config:
https://github.com/stryker-mutator/stryker/blob/b83e304f2de2bb5a60c22cb22ccafdd08cfcc9ce/integrationTest/test/angular-project/webpack-stryker.config.js#L3-L17

Would it be OK to move all that into a single public api call? That would be great as it would allow users to simplify their webpack config used for Stryker, as well as make sure the API doesn't break between patch and minor releases.

EDIT: I think the api could look something like this:

import { buildTestWebpackConfig, getAppFromConfig } from '@angular/cli';

const appConfig = getAppFromConfig(undefined /* app name */);
const webpackConfig = buildTestWebpackConfig(appConfig );

[nicojs]

@filipesilva stryker-webpack-transpiler has been released! See https://github.com/nicojs/angular-stryker-example where I explain how to use it for Anguler.

As a next step, I would like to work on the changes in the angular cli public api. Could you please share your thoughts my comment above? Thanks!

[nicojs]

@clydin @filipesilva

Since the last time we spoke I see a lot of changes in your code. My setup is not working anymore for angular 6 (see nicojs/angular-stryker-example#3). It was always supposed to be a temp solution anyway.

I've got a 2 questions:

1. Is it now possible for the angular-compiler-plugin to compile using an in-memory filesystem? If so, how does this work?
2. Is there an easy way to build the webpack configuration used by karma? If so, how?

[filipesilva]

Heya @nicojs, we've been making a lot of changes in V6, yes.

Notably, in angular/devkit#525 we added support for completely using a virtual file system there. I think this is ultimately what you needed.

We pass it in here

https://github.com/angular/devkit/blob/d715cc9149bd811278a5ac53c8ab1f597bb0c9ed/packages/angular_devkit/build_angular/src/angular-cli-files/models/webpack-configs/typescript.ts#L88

There isn't a super easy way to build the configuration we pass on to Karma, no. The best I can say is that we have separated how we build it here

https://github.com/angular/devkit/blob/d715cc9149bd811278a5ac53c8ab1f597bb0c9ed/packages/angular_devkit/build_angular/src/karma/index.ts#L73-L82

Since we've moved to the new Architect based builders (what's in the new angular.json under the architect and builder keys), it should be possible to make a Stryker builder as well for tests that looks similar to the Karma builder.

But I have to warn you that the Architect APIs are experimental and will change in the next few months, so please proceed at your own risk.

[nicojs]

@filipesilva old friend, hope you're having a good day! Thanks for the quick response.

There is still a chicken-and-egg problem, as the angular-compiler-plugin seems to always override any existing fs with his own here: https://github.com/angular/devkit/blob/d715cc9149bd811278a5ac53c8ab1f597bb0c9ed/packages/ngtools/webpack/src/angular_compiler_plugin.ts#L581-L585. This will override our own virtual fs here: https://github.com/stryker-mutator/stryker/blob/a6029269e1b37f64febabb6d843031ee14eacc56/packages/stryker-webpack-transpiler/src/compiler/WebpackCompiler.ts#L19-L22. This requires some more thinking.

About the building of the webpack configuration using a Stryker builder, how would that work with the architect api? Is there some documentation on the topic?

[nicojs]

Honestly, I'm thinking of using this to capture karma config and webpack config:

function captureKarmaConfig() {
  return new Promise(res => {
    require.cache[require.resolve('karma')] = {
      exports: {
        Server: function (options, cb) {
          process.nextTick(cb);
          return {
            start() {
              res(options);
            }
          }
        }
      }
    };
    process.argv = ['node', require.resolve('@angular/cli'), 'test', '--watch=false'];
    require('@angular/cli/bin/ng');
  });
}

It is ugly... I agree, but at least it uses the public api (ng test --watch=false and karma.Server).

For the chicken-and-egg problem, I now see that you use the passed input filesystem to decorate with the VirtualFileSystem. I'll give it a try later today, it seems promising.

[nicojs]

@filipesilva I've tried to use the VirtualFileSystem implementation, but something doesn't seem to working.

I've traced down the creation of the typescript compiler host. Which is done here: https://github.com/angular/compiler-cli-builds/blob/c2775bba125e9ed40ba60eed4bc6467ffaacf01c/src/ngtools_api2.js#L41. How is the typescript compiler instance supposed react on changes in the virtual file system?

I've tried to use the angular_compiler_plugin with angular 6, but with the same results as 6 months ago, namely a mutation score of 0%. So it seems that changes in the in-memory fs still do not propagate to the typescript compiler host. Reverting back to my PoC implementation (awesome-ts-loader) works fine.

Can you confirm that this is the case?

[nicojs]

Hi again 👋

I've got some good news. I've decided on another approach, implemented it and released it. It works 👍

However, it relies on a private api: '@angular/cli/lib/cli'. Before I close this issue, I would like to make a PR to make the cli public. It would allow programmatic execution of commands:

import { cli } from '@angular/cli';
cli({
    cliArgs: ['test', '--progress=false', /* ...etc */],
    inputStream: process.stdin,
    outputStream: process.stdout
});

@filipesilva would that be OK?

To use angular with mutation testing, you can use this guide: https://github.com/stryker-mutator/stryker-handbook/blob/master/stryker/configuration/angular.md

Basically it works like this:

• It starts the ng test command and keeps the server open.
• It mutates files one by one and reports the result

The implementation is based on the intellij karma plugin, which also executes ng test and keeps the server open. Nice find!

Thanks for making this possible by merging #11181

[filipesilva]

Heya @nicojs, I think that would be a nice addition, yes. We've recently added another binary as an API too in #12022. I'd ask you to use roughly the same approach as https://github.com/angular/angular-cli/blob/master/packages/angular_devkit/benchmark/src/main.ts uses.

Also, you might be interested in karma-runner/karma#3153. We'd like that functionality in so that we can test Karma rebuilds, and I think it might be relevant for you too since there's a lot of tooling involved in Stryker.

[intellix]

This looks interesting and karma-runner/karma#3153 was merged. Is there anything to do here to get it working with Jest?

[nicojs]

Is there anything to do here to get it working with Jest?

What do you mean? You mean mutation testing with Jest? It is already support via the @stryker-mutator/jest-runner.

For anyone else, Stryker does support @angular/cli projects with the [karma.projectType = 'angular-cli'] ](https://github.com/stryker-mutator/stryker/tree/master/packages/karma-runner#karmaprojecttype-custom--angular-cli. Closing this issue.

[intellix]

@nicojs Because we're using Jest within Angular CLI via nrwl/nx: https://github.com/nrwl/nx so not sure if there's anything else to do there

[nicojs]

Aha, that is completely new for me.

It might work out-of-the-box if you create a jest.config.js that mimicks the configuration that angular's jest plugin uses:
https://github.com/nrwl/nx/blob/e88babaf780cf9891bd0c4fe75a70c8d4ba5adf2/packages/jest/src/builders/jest/jest.impl.ts#L63-L66

If you create the jest.config.js file, you should be able to run jest from the command line directly, instead of relying on ng test. If that works, you can configure that in stryker.conf.js and everything should work.

Note: if it does work, feel free to add the use case to https://github.com/stryker-mutator/stryker-handbook/tree/master/stryker/guides

[nicojs]

Aha, that is completely new for me. Is this related to stryker-mutator/stryker-js#1620 ? If so, we will probably support it in the near future.

[alfaproject]

The builder that @intellix mentioned is actually from @nrwl/nx but it's fundamentally the same. You can see the source code here: https://github.com/nrwl/nx/blob/master/packages/jest/src/builders/jest/jest.impl.ts

As you can see, it's quite simple. Basically proxies into jest.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}