You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.developers.md
+51-44Lines changed: 51 additions & 44 deletions
Original file line number
Diff line number
Diff line change
@@ -183,52 +183,57 @@ Python files are automatically checked using `pylint` to ensure they follow esta
183
183
184
184
VTR has a variety of tests which are used to check for correctness, performance and Quality of Result (QoR).
185
185
186
+
## Tests
186
187
There are 4 main regression testing suites:
187
188
188
-
*`vtr_reg_basic`: ~1 minute serial
189
+
### vtr_reg_basic
189
190
190
-
**Goal:** Fast functionality check
191
+
~1 minute serial
191
192
192
-
**Feature Coverage:**Low
193
+
**Goal:**Fast functionality check
193
194
194
-
**Benchmarks:**A few small and simple circuits
195
+
**Feature Coverage:**Low
195
196
196
-
**Architectures:** A few simple architectures
197
+
**Benchmarks:** A few small and simple circuits
197
198
198
-
This regression test is *not* suitable for evaluating QoR or performance.
199
-
It's primary purpose is to make sure the various tools do not crash/fail in the basic VTR flow.
199
+
**Architectures:** A few simple architectures
200
200
201
-
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
202
-
Occasionally, code changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
201
+
This regression test is *not* suitable for evaluating QoR or performance.
202
+
It's primary purpose is to make sure the various tools do not crash/fail in the basic VTR flow.
203
203
204
-
*`vtr_reg_strong`: ~20 minutes serial, ~15 minutes with `-j4`
204
+
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
205
+
Occasionally, code changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
205
206
206
-
**Goal:** Broad functionality check
207
+
### vtr_reg_strong
207
208
208
-
**Feature Coverage:** High
209
+
~20 minutes serial, ~15 minutes with `-j4`
209
210
210
-
**Benchmarks:**A few small circuits, with some special benchmarks to exercise specific features
211
+
**Goal:**Broad functionality check
211
212
212
-
**Architectures:**A variety of architectures, including special architectures to exercise specific features
213
+
**Feature Coverage:**High
213
214
214
-
This regression test is *not* suitable for evaluating QoR or performance.
215
-
It's primary purpose is try and achieve high functionality coverage.
215
+
**Benchmarks:** A few small circuits, with some special benchmarks to exercise specific features
216
216
217
-
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
218
-
Occasionally, changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
219
-
220
-
*`vtr_reg_nightly_test#, #:1-3`:
217
+
**Architectures:** A variety of architectures, including special architectures to exercise specific features
221
218
222
-
**Goal:** Basic QoR and Performance evaluation
223
-
224
-
**Feature Coverage:** Medium
225
-
226
-
**Architectures:** A wider variety of architectures
227
-
228
-
**Benchmarks:** Small-medium size, diverse. All include:
219
+
This regression test is *not* suitable for evaluating QoR or performance.
220
+
It's primary purpose is try and achieve high functionality coverage.
221
+
222
+
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
223
+
Occasionally, changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
229
224
230
-
* VTR benchmarks
231
-
* Additional benchmarks for each suite.
225
+
### vtr_reg_nightly_test1-3
226
+
227
+
**Goal:** Basic QoR and Performance evaluation
228
+
229
+
**Feature Coverage:** Medium
230
+
231
+
**Architectures:** A wider variety of architectures
232
+
233
+
**Benchmarks:** Small-medium size, diverse. All include:
234
+
235
+
* VTR benchmarks
236
+
* Additional benchmarks for each suite.
232
237
233
238
QoR checks in these regression suites are aimed at evaluating quality and run-time of the VTR flow.
234
239
As a result any QoR failures are a concern and should be investigated and understood.
@@ -237,24 +242,26 @@ There are 4 main regression testing suites:
237
242
238
243
These suites comproise a single large suite, `vtr_reg_nightly` and should be run together to test nightly level regression. They are mostly similar in benchmark coverage interms of size and diversity however each suite tests some unique benchmarks in addition to the VTR benchmarks.
239
244
240
-
| suite | wall-clock time| Additional benchmarks|
241
-
|-------|----------------|----------------------|
242
-
|vtr_reg_nightly_test1|~4.5 hours with `-j8`|ISPD and MCNC20 |
243
-
|vtr_reg_nightly_test2|~6 hours with `-j8`|Titan23 and Titan `other`|
244
-
|vtr_reg_nightly_test3|~5.5 hours with `-j8`|none|
245
-
246
-
*`vtr_reg_weekly`: ~42 hours with `-j4`
245
+
| suite | wall-clock time| Additional benchmarks|
246
+
|-------|----------------|----------------------|
247
+
|vtr_reg_nightly_test1|~4.5 hours with `-j8`|ISPD and MCNC20 |
248
+
|vtr_reg_nightly_test2|~6 hours with `-j8`|Titan23 and Titan `other`|
249
+
|vtr_reg_nightly_test3|~5.5 hours with `-j8`|none|
**Architectures:** A wide variety of architectures
264
+
**Architectures:** A wide variety of architectures
258
265
259
266
QoR checks in this regression are aimed at evaluating quality and run-time of the VTR flow.
260
267
As a result any QoR failures are a concern and should be investigated and understood.
@@ -671,8 +678,8 @@ k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml lstm.v common 38901.96
671
678
### Example: CI Tests QoR Measurement
672
679
Once the changes are pushed into the remote repository, or a PR is created, the [Test Workflow](https://github.com/verilog-to-routing/vtr-verilog-to-routing/blob/master/.github/workflows/test.yml)
673
680
will be triggered. The following tests are included in the workflow:
0 commit comments