Skip to content

Commit f0ae8c3

Browse files
committed
Add link to headers
1 parent 2df395a commit f0ae8c3

File tree

1 file changed

+51
-44
lines changed

1 file changed

+51
-44
lines changed

README.developers.md

Lines changed: 51 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -183,52 +183,57 @@ Python files are automatically checked using `pylint` to ensure they follow esta
183183

184184
VTR has a variety of tests which are used to check for correctness, performance and Quality of Result (QoR).
185185

186+
## Tests
186187
There are 4 main regression testing suites:
187188

188-
* `vtr_reg_basic`: ~1 minute serial
189+
### vtr_reg_basic
189190

190-
**Goal:** Fast functionality check
191+
~1 minute serial
191192

192-
**Feature Coverage:** Low
193+
**Goal:** Fast functionality check
193194

194-
**Benchmarks:** A few small and simple circuits
195+
**Feature Coverage:** Low
195196

196-
**Architectures:** A few simple architectures
197+
**Benchmarks:** A few small and simple circuits
197198

198-
This regression test is *not* suitable for evaluating QoR or performance.
199-
It's primary purpose is to make sure the various tools do not crash/fail in the basic VTR flow.
199+
**Architectures:** A few simple architectures
200200

201-
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
202-
Occasionally, code changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
201+
This regression test is *not* suitable for evaluating QoR or performance.
202+
It's primary purpose is to make sure the various tools do not crash/fail in the basic VTR flow.
203203

204-
* `vtr_reg_strong`: ~20 minutes serial, ~15 minutes with `-j4`
204+
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
205+
Occasionally, code changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
205206

206-
**Goal:** Broad functionality check
207+
### vtr_reg_strong
207208

208-
**Feature Coverage:** High
209+
~20 minutes serial, ~15 minutes with `-j4`
209210

210-
**Benchmarks:** A few small circuits, with some special benchmarks to exercise specific features
211+
**Goal:** Broad functionality check
211212

212-
**Architectures:** A variety of architectures, including special architectures to exercise specific features
213+
**Feature Coverage:** High
213214

214-
This regression test is *not* suitable for evaluating QoR or performance.
215-
It's primary purpose is try and achieve high functionality coverage.
215+
**Benchmarks:** A few small circuits, with some special benchmarks to exercise specific features
216216

217-
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
218-
Occasionally, changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
219-
220-
* `vtr_reg_nightly_test#, #:1-3`:
217+
**Architectures:** A variety of architectures, including special architectures to exercise specific features
221218

222-
**Goal:** Basic QoR and Performance evaluation
223-
224-
**Feature Coverage:** Medium
225-
226-
**Architectures:** A wider variety of architectures
227-
228-
**Benchmarks:** Small-medium size, diverse. All include:
219+
This regression test is *not* suitable for evaluating QoR or performance.
220+
It's primary purpose is try and achieve high functionality coverage.
221+
222+
QoR checks in this regression test are primarily 'canary' checks to catch gross degradations in QoR.
223+
Occasionally, changes can cause QoR failures (e.g. due to CAD noise -- particularly on small benchmarks); usually such failures are not a concern if the QoR differences are small.
229224

230-
* VTR benchmarks
231-
* Additional benchmarks for each suite.
225+
### vtr_reg_nightly_test1-3
226+
227+
**Goal:** Basic QoR and Performance evaluation
228+
229+
**Feature Coverage:** Medium
230+
231+
**Architectures:** A wider variety of architectures
232+
233+
**Benchmarks:** Small-medium size, diverse. All include:
234+
235+
* VTR benchmarks
236+
* Additional benchmarks for each suite.
232237

233238
QoR checks in these regression suites are aimed at evaluating quality and run-time of the VTR flow.
234239
As a result any QoR failures are a concern and should be investigated and understood.
@@ -237,24 +242,26 @@ There are 4 main regression testing suites:
237242

238243
These suites comproise a single large suite, `vtr_reg_nightly` and should be run together to test nightly level regression. They are mostly similar in benchmark coverage interms of size and diversity however each suite tests some unique benchmarks in addition to the VTR benchmarks.
239244

240-
| suite | wall-clock time| Additional benchmarks|
241-
|-------|----------------|----------------------|
242-
|vtr_reg_nightly_test1|~4.5 hours with `-j8`|ISPD and MCNC20 |
243-
|vtr_reg_nightly_test2|~6 hours with `-j8`|Titan23 and Titan `other`|
244-
|vtr_reg_nightly_test3|~5.5 hours with `-j8`|none|
245-
246-
* `vtr_reg_weekly`: ~42 hours with `-j4`
245+
| suite | wall-clock time| Additional benchmarks|
246+
|-------|----------------|----------------------|
247+
|vtr_reg_nightly_test1|~4.5 hours with `-j8`|ISPD and MCNC20 |
248+
|vtr_reg_nightly_test2|~6 hours with `-j8`|Titan23 and Titan `other`|
249+
|vtr_reg_nightly_test3|~5.5 hours with `-j8`|none|
250+
251+
### vtr_reg_weekly
252+
253+
~42 hours with `-j4`
247254

248-
**Goal:** Full QoR and Performance evaluation.
255+
**Goal:** Full QoR and Performance evaluation.
249256

250-
**Feature Coverage:** Medium
257+
**Feature Coverage:** Medium
251258

252-
**Benchmarks:** Medium-Large size, diverse. Includes:
259+
**Benchmarks:** Medium-Large size, diverse. Includes:
253260

254-
* VTR benchmarks
255-
* Titan23 benchmarks
261+
* VTR benchmarks
262+
* Titan23 benchmarks
256263

257-
**Architectures:** A wide variety of architectures
264+
**Architectures:** A wide variety of architectures
258265

259266
QoR checks in this regression are aimed at evaluating quality and run-time of the VTR flow.
260267
As a result any QoR failures are a concern and should be investigated and understood.
@@ -671,8 +678,8 @@ k6FracN10LB_mem20K_complexDSP_customSB_22nm.xml lstm.v common 38901.96
671678
### Example: CI Tests QoR Measurement
672679
Once the changes are pushed into the remote repository, or a PR is created, the [Test Workflow](https://github.com/verilog-to-routing/vtr-verilog-to-routing/blob/master/.github/workflows/test.yml)
673680
will be triggered. The following tests are included in the workflow:
674-
* [vtr_reg_nightly_test#](https://docs.verilogtorouting.org/en/latest/README.developers/#running-tests), #:1-3
675-
* [vtr_reg_strong](https://docs.verilogtorouting.org/en/latest/README.developers/#running-tests)
681+
* [vtr_reg_nightly_test1-3](./README.developers.md#vtr_reg_nightly_test1-3)
682+
* [vtr_reg_strong](./README.developers.md#vtr_reg_strong)
676683
* vtr_reg_yosys
677684
* vtr_reg_yosys_odin
678685
* odin_tech_strong

0 commit comments

Comments
 (0)