Skip to content

Commit 383df16

Browse files
[llvm][llvm-lit] Add total time for each testsuite in JUnit XML output (llvm#112230)
Currently we write out a time taken to run all test suites: ``` <testsuites time="8.28"> ``` And one for each test: ``` <testcase classname="lldb-shell.Breakpoint" name="breakpoint-command.test" time="2.38"/> ``` However, the schema says there should be one for each suite and test, but none for testsuites: https://github.com/windyroad/JUnit-Schema/blob/cfa434d4b8e102a8f55b8727b552a0063ee9044e/JUnit.xsd#L216 I'm leaving the `testsuites` time in though because no one has complained so far, and someone out there probably has a script relying on it by now. Most XML tools handle unknown attributes quite well anyway. I'm adding a per testsuite time to comply with the schema and maybe be more compatible with other JUnit tools. ``` <testsuite name="lldb-shell" ... time="12.34"> ``` The test suite time is the sum of the time taken for all tests in the suite. This will ignore some overhead in setting up the suite, and means that the sum of the times for all individual suites may not equal the `testsuites` time. As we're usually focusing on the execution time of particular tests, not lit's book keeping, I think this is a reasonable choice.
1 parent 9381c6f commit 383df16

File tree

3 files changed

+13
-5
lines changed

3 files changed

+13
-5
lines changed

llvm/utils/lit/lit/reports.py

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -105,12 +105,20 @@ def write_results(self, tests, elapsed):
105105
file.write("</testsuites>\n")
106106

107107
def _write_testsuite(self, file, suite, tests):
108-
skipped = sum(1 for t in tests if t.result.code in self.skipped_codes)
109-
failures = sum(1 for t in tests if t.isFailure())
108+
skipped = 0
109+
failures = 0
110+
time = 0.0
111+
112+
for t in tests:
113+
if t.result.code in self.skipped_codes:
114+
skipped += 1
115+
if t.isFailure():
116+
failures += 1
117+
time += t.result.elapsed
110118

111119
name = suite.config.name.replace(".", "-")
112120
file.write(
113-
f'<testsuite name={quo(name)} tests="{len(tests)}" failures="{failures}" skipped="{skipped}">\n'
121+
f'<testsuite name={quo(name)} tests="{len(tests)}" failures="{failures}" skipped="{skipped}" time="{time:.2f}">\n'
114122
)
115123
for test in tests:
116124
self._write_test(file, test, name)

llvm/utils/lit/tests/shtest-format.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@
107107

108108
# XUNIT: <?xml version="1.0" encoding="UTF-8"?>
109109
# XUNIT-NEXT: <testsuites time="{{[0-9.]+}}">
110-
# XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="8" skipped="3">
110+
# XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="8" skipped="3" time="{{[0-9.]+}}">
111111

112112
# XUNIT: <testcase classname="shtest-format.external_shell" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
113113
# XUNIT-NEXT: <failure{{[ ]*}}>

llvm/utils/lit/tests/xunit-output.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99

1010
# CHECK: <?xml version="1.0" encoding="UTF-8"?>
1111
# CHECK-NEXT: <testsuites time="{{[0-9.]+}}">
12-
# CHECK-NEXT: <testsuite name="test-data" tests="5" failures="1" skipped="3">
12+
# CHECK-NEXT: <testsuite name="test-data" tests="5" failures="1" skipped="3" time="{{[0-9.]+}}">
1313
# CHECK-NEXT: <testcase classname="test-data.test-data" name="bad&amp;name.ini" time="{{[0-1]\.[0-9]+}}">
1414
# CHECK-NEXT: <failure><![CDATA[& < > ]]]]><![CDATA[> &"]]></failure>
1515
# CHECK-NEXT: </testcase>

0 commit comments

Comments
 (0)