Code Coverage report doesn't reflect reality coverage of hits and misses - highlighting random lines

[Dj-Viking]

Logs and screenshots

• screenshot of line coverage report, hit and miss lines don't make sense on the .vue files. Not sure what exactly is being missed or hit by the test running against the app. Just seems kinda randomly placed everywhere.

I'm not sure if I understand this debug step here below. I don't know where the logs are output or if this variable is set before starting the client app or before running the cypress tests. my npm run dev command may be slightly different than what is expected in the README.

Please provide debug logs by running Cypress from the terminal with DEBUG=code-coverage environment variable set, see the Debugging section of the README file.

Versions

• What is this plugin's version? - latest - v3.10.0
• If the plugin worked before in version X, but stopped after upgrading to version Y, please try the released versions between X and Y to see where the breaking change was. - n/a
• What is Cypress version? - latest - v10.6.0
• What is your operating system? MacOS Catalina 10.15.7
• What is the shell? - GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
• What is the Node version? - v16.14.2
• What is the NPM version? - v8.5.0
• How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.

• I instrument the vue app with my babel.config.js file displayed below

const plugins = [];
if (process.env.NODE_ENV === "development") {
  plugins.push([
    "babel-plugin-istanbul",
    {
      extension: [".ts", ".js", ".vue"]
    }
  ]);
}
module.exports = {
  presets: ["@vue/cli-plugin-babel/preset"],
  plugins
};

• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot? - yes

• screenshot of coverage object in application window in dev mode

• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths? - folder and file is there, not empty

{
  "<REDACTED>/Nomad-Lang/client/src/main.ts": {
    "path": "<REDACTED>/Nomad-Lang/client/src/main.ts",
    "statementMap": {
      "0": {
        "start": {
          "line": 2,
          "column": 22
        },
        "end": {
          "line": 4,
          "column": 1
        }
      },
      "1": {
        "start": {
          "line": 3,
          "column": 4
        },
        "end": {
          "line": 3,
          "column": 62
        }
      },
      "2": {
        "start": {
          "line": 5,
          "column": 0
        },
        "end": {
          "line": 5,
          "column": 62
        }
      },

• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files - yes

"nyc": {
    "all": true,
    "include": [
      "src/components/*.vue",
      "src/router/index.ts",
      "src/store/*.ts",
      "src/utils/*.ts",
      "src/views/*.vue",
      "src/App.vue",
      "src/main.ts"
    ]
  },

• Do you run Cypress tests in a Docker container? - no

Describe the bug
A clear and concise description of what the bug is.

• It seems for some reason the hit and miss highlighting is not accurate at all in the .vue files, I cant really tell what I need to cover what's being missed in the .vue files. It just seems like it highlights random places and shows IF ELSE branching in random places where there is no IF ELSE branch in the line coverage report files.

Link to the repo
Bugs with a reproducible example, like an open source repo showing the bug, are the most likely to be resolved.

• repo

• https://github.com/Dj-Viking/Nomad-Lang

• steps to reproduce

git clone [email protected]:Dj-Viking/Nomad-Lang.git && \
cd Nomad-Lang && \
git checkout task/88/update-cypress && \
npm install && \
cd client && \
npm run test:cy:ci

• when tests are finished feel free to killall node as the concurrent shell command running the client dev server and the headless cypress tests, will keep a background shell running of the client dev server even with a SIGTERM in the current foreground shell.
• open coverage report in browser from client/coverage/lcov-report/index.html (right-click and run in live server or just open the index.html file in your browser of choice)
• look at any .vue file that was covered by the cypress tests

Thanks for your time,

Anders a.k.a. DJViking

[Scott-Fischer]

I'm having a very similar issue to this where like half of a function name is highlighted as a coverage miss.

My gut reaction was that it had something to do with a miss match between source maps/transpiled code and the position of code in the report but honestly not 100% sure where to start looking. @Dj-Viking Did you ever figure out the problem and resolve it?

[Dj-Viking]

Hey @Scott-Fischer

I haven't been able to figure this out for a very long time, I'm just getting around to opening an issue on it because I am honestly completely lost why this happens. But you might be on to something with transpilation and sourcemapping - I just can't find any way to know how to pinpoint why it happens unless I dedicate a lot of spare time that I cant dedicate to reading tons of source code on this repo.

Hoping any Subject Matter Experts who works on this repo to chime in their ideas or input if I'm using this in a non-intended way or if theres missing configuration on my part.

[jogelin]

@Dj-Viking @Scott-Fischer Same here.

Spent a lot of time trying to sync the source map with the coverage and I had the same conclusion: sourcemap isn't in sync with the coverage.

And when you try to merge code generate by webpack (with ts-loader) + babel (ts preset) + jest, you see again that it is not aligned.

I was just able to reduce a bit the differences by building the app without any optimization, in dev mode and then it was ok

[Dj-Viking]

I had a little time today to read through some of the source code. There is a function called fixSourcePaths which sounds like something it might do to "fix" where transpiled code is source-mapped to the actual code not sure exactly how it works or how I would debug this code yet.

code-coverage/support-utils.js

Line 137 in 16ef891

function fixSourcePaths(coverage) {

[Dj-Viking]

and the function is called here which doesn't return anything my guess is the coverage object is passed by reference and modified in place...but I just dont quite understand it...Just a shot in the dark.

code-coverage/task.js

Line 149 in 16ef891

fixSourcePaths(coverage)

[jogelin]

Interesting, I had to fix this path because of this function.

I am running tests on multiple machines/VM (parallel) and the absolute path of each file is an issue when you want to merge or generate reports on another machine/VM.
In fact, the path should be valid so I reverted the relative path that is valid in all machines/VM before merging reports.

Could it be the reason for having the wrong highlights? I don't think because IMO this is related to the generated code that is used by the instrumentation that is not inline with the source map. But the source map is present

[Scott-Fischer]

@jogelin When you say that you are providing absolute file paths, do you mean when identifying which files to include/exclude from tests/coverage? For example, via .nycrc or cypress config? Or are you talking about another process that requires paths?

For what it's worth, I've only only been experimented with cypress coverage locally so I don't think it's a specific issue of running things in parallel.

I was thinking though, perhaps there's some commonalities between our projects that we could use to identify where things are going wrong.

My project is using both typescript and JSX
Instrumenting code with babel-plugin-istanbul via webpack (v5.74.0)
Running tests with cypress (v10.9.0)

[jogelin]

@Scott-Fischer I am using typescript (Angular templates) instrumenting with babel-plugin-istanbul + webpack too

But I was explaining that, because of the fixSourcePaths function that is setting the absolute path, I had an issue when trying to merge reports.

For example, if you are generating 2 reports containing the same file, then the absolute path will be different and then they will be considered as 2 different files. Si mainly by removing the absolute path, the path to the files are equals and the merge works correctly.

But I also have this issue locally concerning the highlight of the coverage using webpack and babel-plugin-istanbul.

[Scott-Fischer]

Another interesting development I just found.

My initial discovery of this problem was occurring in a completely customized tool chain that was set up manually. So there's a lot of moving pieces/configs and it was really tough to track down the exact source of this problem. e.g. typescript, babel, webpack etc etc.

However, we have a separate react frontend client that was created with CRA. It was initially just a JS app and we had code coverage through @cypress/instrument-cra + cypress-io/code-coverage which was working fine.

We then converted the app to typescript and started slowly migrating .js files to typescript. However, the coverage report would only list .js files and that's likely because @cypress/instrument-cra isn't instrumenting the typescript files. In order to add support for typescript, I removed the @cypress/instrument-cra module and modified the babel config via CRACO in order to manually add babel-plugin-istanbul.

This ultimately worked in generating the coverage for our TS files. But to my surprise, the exact same issue started to occur where the statement.start and statement.end values in .nyc_output/out.json no longer match the actual source code positions of the corresponding statements.

I tried various ways of generating source maps (through TSC, through babel, through webpack, inline, external files, etc) and did not have any success in resolving the issue.

So I'm wondering, is the problem in the cypress coverage plugin? Or is the issue actually in the underlying babel-plugin-istanbul module? Because when using @cypress/instrument-cra to instrument the JS code, cypress-io/code-coverage worked perfectly.

[Dj-Viking]

@Scott-Fischer Hey man I think you have a point there! There could be some configuration settings that are incorrect/missing with babel-plugin-istanbul that we're not aware of I suppose. I will play with the settings a bit more, we may need to override some default values that are causing us some headache 😓

[Dj-Viking]

So I did some more digging this weekend and I looked at the sourceMapped file and it looks to me with the .vue files the actual .ts file that is being mapped to doesn't include the template part. the top of the file is right were the script tag starts, which makes sense I suppose since the whole .vue file is mostly governed by everything in the <script> tag....but there is some logic within the template as well.

So I just have no clue how anything is being transformed and then source mapped. I am not sure if these plugins have been accounted for the most recent changes to vue3 apps or typescript apps in general because of the transpilation process where the code that executes in the browser are .js files but the code written is now mostly .ts files. Also the readme for babel-plugin-istanbul most of the important parts the git blame shows it hasn't been updated in about 6 years or so. :(

I have had decent success of getting accurate source mapping of the vue files when using jest with vue. but I found it not as intuitive for what I am really trying to test on the app (as if a user was using the application and not a fake browser environment) and have coverage resulting from this "automated test user".

I am just very lost how to set up the app properly for code coverage. I can't find any up to date instructions anywhere. I found one official post by Gleb himself on instrumenting a vue app but it is using vue2 and I am using vue3 so I am not sure if the instructions apply anymore.

[Dj-Viking]

Anyone have an idea how to fix this? Is it even related to the code-coverage package at all?

[vishnu-exerp]

We have the similar issue too, Did you manage to fix this @Dj-Viking ?

[Dj-Viking]

@vishnu-exerp nope :(

[zodman]

2024, I have the same problem with a angular15 application

[MKhasib]

The issue is summarized with:

1. Unit tests (Jest) explicitly report uncovered lines.
2. E2E tests (Cypress) only report covered lines, omitting uncovered ones.
3. When merging coverage reports, this leads to underreporting of overall coverage coming from unit tests

When I inspected the source instrumented code I found e2e test ignoring some lines from instrumenting causing the problem we are facing

The proper fix for this issue is for the e2e tests code-coverage plugin to report all lines the same way jest does