CI: Add print skipped tests into azure step #26698

xcz011 · 2019-06-07T01:44:49Z

closes CI - Azure: print skipped tests #26695
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

codecov · 2019-06-07T02:25:09Z

Codecov Report

Merging #26698 into master will decrease coverage by 50.58%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #26698       +/-   ##
===========================================
- Coverage   91.78%    41.2%   -50.59%     
===========================================
  Files         174      179        +5     
  Lines       50703    50767       +64     
===========================================
- Hits        46538    20918    -25620     
- Misses       4165    29849    +25684

Flag	Coverage Δ
#multiple	`?`
#single	`41.2% <ø> (-0.7%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
pandas/io/formats/html.py	`0% <0%> (-99.37%)`	⬇️
pandas/io/sas/sas7bdat.py	`0% <0%> (-91.16%)`	⬇️
pandas/io/sas/sas_xport.py	`0% <0%> (-90.1%)`	⬇️
pandas/core/sparse/scipy_sparse.py	`10.14% <0%> (-89.86%)`	⬇️
pandas/core/tools/numeric.py	`10.14% <0%> (-89.86%)`	⬇️
... and 139 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf25c5c...68c39a1. Read the comment docs.

codecov · 2019-06-07T02:25:09Z

Codecov Report

Merging #26698 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26698      +/-   ##
==========================================
- Coverage   91.88%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50701    50701              
==========================================
- Hits        46588    46583       -5     
- Misses       4113     4118       +5

Flag	Coverage Δ
#multiple	`90.41% <ø> (ø)`	⬆️
#single	`41.91% <ø> (-0.11%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.6% <0%> (-0.11%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 649ad5c...e06ebb8. Read the comment docs.

datapythonista · 2019-06-07T08:03:51Z

This is the checks job, only the validate_docstrings.py tests are run here, and no tests are skipped there.

This is trickier than what it looks I'd say, ideally, I think we should unify how the tests are called, and add the printing of the skipped tests there. So the output is consistent among builds.

Closing this PR, but feel free to open a new one with those changes.

datapythonista · 2019-06-07T08:04:06Z

CC: @jorisvandenbossche

jorisvandenbossche · 2019-06-07T08:13:27Z

@datapythonista It's not that this "simply" needs to be moved to a step in the posix.yml template? The tests are there always called with ci/run_tests.sh, so not that what should be unified.

datapythonista · 2019-06-07T08:19:52Z

I think the best approach would be to have this in ci/run_tests.sh. The print skipped is already in the travis docs, and in azure we'd probably like to add it to posix but also windows.

I'm ok with moving the code in this PR to posix.yml for now, but ideally I think we should avoid repeating it in every CI system.

jorisvandenbossche · 2019-06-07T08:21:29Z

Isn't it actually valuable that it is a separate step? That makes it simpler to look at it in the Azure interface (one of its advantages that you can split the log)

jorisvandenbossche · 2019-06-07T08:25:16Z

It would maybe be nice to move the if checks (for single multiple tests) to a common script, but I don't see the problem with repeating calling out to that script in each CI. We also repeat the call to ci/run_tests.sh.

Anyway, I think this provides enough short term value (eg for checking if the clipboard tests ran or not) to add like this for now.

datapythonista · 2019-06-07T08:30:06Z

Good point, then I'd simply have the if statement for the single/multi inside the script, and not in the CI config.

@xcz011 do you mind doing that?

The changes would be:

Check for which of the files exist (the if in your code) inside ci/print_skipped.py, so we can simply call the script in the CI config
Update .travis.yml to remove the if
Add your changes (without the if) to ci/azure/posix.yml and ci/azure/windows.yml

Reopening this, so you can keep working on the same branch if you want.

Thanks!

xcz011 · 2019-06-08T13:58:42Z

Good point, then I'd simply have the if statement for the single/multi inside the script, and not in the CI config.

@xcz011 do you mind doing that?

The changes would be:

Check for which of the files exist (the if in your code) inside ci/print_skipped.py, so we can simply call the script in the CI config

Update .travis.yml to remove the if

Add your changes (without the if) to ci/azure/posix.yml and ci/azure/windows.yml

Reopening this, so you can keep working on the same branch if you want.

Thanks!

Sure, I will try to do it and may need some help

This reverts commit e06ebb8.

…test

Update travis and azure related

datapythonista

Good job, @xcz011, the skipped tests are correctly displayed in the CI. Just added a couple of minor comments.

datapythonista · 2019-06-09T09:19:45Z

ci/azure/posix.yml

@@ -90,3 +90,8 @@ jobs:
          Write-Error "$($matches[1]) tests failed"
        }
      displayName: Check for test failures


Can you have this in quotes as the rest of displayName properties in the fie? Looks like they are not needed, but better to have it consistent.

datapythonista · 2019-06-09T09:20:40Z

ci/azure/posix.yml

+        export PATH=$HOME/miniconda3/bin:$PATH
+        source activate pandas-dev
+        python ci/print_skipped.py
+      displayName: 'print skipped test'


Can you capitalize the p and use tests instead of test? so the display name is Print skipped tests

datapythonista · 2019-06-09T09:21:13Z

ci/azure/windows.yml

+        export PATH=$HOME/miniconda3/bin:$PATH
+        source activate pandas-dev
+        python ci/print_skipped.py
+      displayName: 'print skipped test'


Same changes as in the other file

datapythonista

Thanks @xcz011, great job here. @jorisvandenbossche merge when you're happy.

jorisvandenbossche · 2019-06-10T15:48:17Z

@xcz011 thanks a lot!

xcz011 · 2019-06-10T23:09:33Z

@xcz011 thanks a lot!

Thanks all the helps !

Add print skiped into azure step

e06ebb8

datapythonista added CI Continuous Integration Testing pandas testing functions or related to the test suite labels Jun 7, 2019

datapythonista closed this Jun 7, 2019

datapythonista reopened this Jun 7, 2019

xcz011 closed this Jun 8, 2019

xcz011 reopened this Jun 8, 2019

xcz011 added 4 commits June 8, 2019 10:13

Revert "Add print skiped into azure step"

69a9cbc

This reverts commit e06ebb8.

Merge remote-tracking branch 'upstream/master' into azure_print_skip_…

fa338d2

…test

update print_skipped file to find test file

181c09a

Update travis and azure related

update the how to execute print_skipped

82a2910

datapythonista reviewed Jun 9, 2019

View reviewed changes

format displayName value

557d3cc

datapythonista approved these changes Jun 9, 2019

View reviewed changes

fix linting

68c39a1

jorisvandenbossche changed the title ~~Add print skiped into azure step~~ CI: Add print skipped tests into azure step Jun 10, 2019

jorisvandenbossche added this to the 0.25.0 milestone Jun 10, 2019

jorisvandenbossche approved these changes Jun 10, 2019

View reviewed changes

jorisvandenbossche merged commit 959e799 into pandas-dev:master Jun 10, 2019

xcz011 deleted the azure_print_skip_test branch June 10, 2019 23:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Add print skipped tests into azure step #26698

CI: Add print skipped tests into azure step #26698

xcz011 commented Jun 7, 2019 •

edited

Loading

codecov bot commented Jun 7, 2019 •

edited

Loading

codecov bot commented Jun 7, 2019

datapythonista commented Jun 7, 2019

datapythonista commented Jun 7, 2019

jorisvandenbossche commented Jun 7, 2019

datapythonista commented Jun 7, 2019

jorisvandenbossche commented Jun 7, 2019

jorisvandenbossche commented Jun 7, 2019

datapythonista commented Jun 7, 2019

xcz011 commented Jun 8, 2019

datapythonista left a comment

datapythonista Jun 9, 2019

datapythonista Jun 9, 2019

datapythonista Jun 9, 2019

datapythonista left a comment

jorisvandenbossche commented Jun 10, 2019

xcz011 commented Jun 10, 2019

CI: Add print skipped tests into azure step #26698

CI: Add print skipped tests into azure step #26698

Conversation

xcz011 commented Jun 7, 2019 • edited Loading

codecov bot commented Jun 7, 2019 • edited Loading

Codecov Report

codecov bot commented Jun 7, 2019

Codecov Report

datapythonista commented Jun 7, 2019

datapythonista commented Jun 7, 2019

jorisvandenbossche commented Jun 7, 2019

datapythonista commented Jun 7, 2019

jorisvandenbossche commented Jun 7, 2019

jorisvandenbossche commented Jun 7, 2019

datapythonista commented Jun 7, 2019

xcz011 commented Jun 8, 2019

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista Jun 9, 2019

Choose a reason for hiding this comment

datapythonista Jun 9, 2019

Choose a reason for hiding this comment

datapythonista Jun 9, 2019

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

jorisvandenbossche commented Jun 10, 2019

xcz011 commented Jun 10, 2019

xcz011 commented Jun 7, 2019 •

edited

Loading

codecov bot commented Jun 7, 2019 •

edited

Loading