Added Customized Heap and Occupancy Profiling for MQ

[ueqri]

1. Added core affinity support for parallel router.
2. Added a customized heap with indexing from one optimization and the ability to drain/clear the heap directly.
3. Added a heap occupancy profiling method to gain insight into the MQ heap occupancy and workload.

Note: To assign the core affinity (i.e., pin threads to specific cores), please set the environment variable export VPR_CORE_AFFINITY=0,2-8. Formats such as 0,1,2,3,4,5,6,7, 0-7, 0-3,4-7, and 0,1-2,3-6,7 are all supported.

For multi-socket systems (i.e., with more than one NUMA node), please run numactl command for VPR execution. For example, numactl --cpunodebind=<NUMA node> --membind=<NUMA node> vpr_wrapper.sh (or simply numactl --cpunodebind=0 --membind=0 bash and run VPR or testing scripts inside the new bash).

[AlexandreSinger]

Hi Hang, these changes look good. I have left some comments that should be resolved. Specifically I would like you to IFDEF the tryPopWithPrio function so we can turn it ON/OFF. For convenience, can you please make if OFF by default.

[ueqri]

The deterministic version with queue draining (min-prio) optimization has been tested and works perfectly with Dijkstra settings (i.e., --astar_fac 0.0 --post_target_prune_fac 0.0 --post_target_prune_offset 0.0) on vtr_chain_largest benchmark. However, the deterministic version with A* settings doesn't work well, probably due to the invalid factor and offset numbers.

[AlexandreSinger]

Hi Hang, I reviewed again. Was this code tested with the options turned on and off? I found an undeclared variable that should not have compiled.

[AlexandreSinger]

The deterministic version with queue draining (min-prio) optimization has been tested and works perfectly with Dijkstra settings (i.e., --astar_fac 0.0 --post_target_prune_fac 0.0 --post_target_prune_offset 0.0) on vtr_chain_largest benchmark. However, the deterministic version with A* settings doesn't work well, probably due to the invalid factor and offset numbers.

@ueqri These results make sense. In order for the draining to be deterministic, the ordering heuristic needs to be an under-estimate (exactly like the pruning heuristic). When astar_fac is 0.0, the ordering heuristic is guaranteed to be an under-estimate. However, to make the A* version an under-estimate, you need --astar_offset to be some number such that it becomes an under-estimate. Since the option is off by default we can do this in a different PR, but I recommend cherry-picking this commit from VTR upstream verilog-to-routing#2601

We will need to modify some code in the parallel connection router to allow this. Once we have astar_offset we need to ensure that the ordering and pruning heuristics are both under-estimates in order to make this change deterministic. This in itself is a tradeoff and will require testing.

[ueqri]

The final version is ready. Deterministic A* works perfectly with --astar_fac 1.0 --astar_offset 3.05e-10 --post_target_prune_fac 1.0 --post_target_prune_offset 3.05e-10 on vtr_chain (verilog-to-routing#2601 was cherry-picked). Some other settings (e.g., deterministic Dijkstra/A*, non-deterministic Directed) with/without min-prio queue draining are all tested on MCNC and work as expected.

The commit (e7108) may be confusing: It was caused by a weird issue which I spent a long time investigating and still not sure why. The parent commit (fa8ba) cannot guarantee determinism. Only applying this commit (e7108) makes the deterministic version work, which doesn't quite make sense based on the changes in that commit. However, that's merely the only workaround I found to make determinism work.

[AlexandreSinger]

I ran into an interesting issue when using core affinity. When running multiple instances of the parallel router at the same time, they all get locked to the same exact cores. This greatly reduces their performance.

This is to be expected, but its odd since it is the default behaviour. I had to disable core affinity for my quick tests.

[ueqri]

When running multiple instances of the parallel router at the same time, they all get locked to the same exact cores. This greatly reduces their performance.

There is a workaround for this case: We can wrap the launch of parallel routers into different shell scripts. In each script, we specify the core affinity list, which will only affect that exact script running, such that different instances can be pinned to different sets of cores.

However, ultimately, the configuration should be put into the CLI options instead of the environment variables.

[AlexandreSinger]

When running multiple instances of the parallel router at the same time, they all get locked to the same exact cores. This greatly reduces their performance.

There is a workaround for this case: We can wrap the launch of parallel routers into different shell scripts. In each script, we specify the core affinity list, which will only affect that exact script running, such that different instances can be pinned to different sets of cores.

However, ultimately, the configuration should be put into the CLI options instead of the environment variables.

That makes sense! We can update our testing script to be aware of this! That's a very good point.

In the meantime I will keep core affinity off (just for my testing), but will turn it on once the testing script is updated.

[ueqri]

@AlexandreSinger Hey Alex, just added four command-line options for MQ-based parallel router:

• --multi_queue_num_threads <# threads>
• --multi_queue_num_queues <# queues>
• --multi_queue_direct_draining <on/off>
• --thread_affinity <off (meaning no affinity, leave OS schedule) or set a list of CPU core ID (the first one is for the main thread), e.g., 0,1,2,3 or 0-3 or 0-1,2-3 or 0,1-2,3>

All the options are tested and work as expected.

[AlexandreSinger]

@AlexandreSinger Hey Alex, just added four command-line options for MQ-based parallel router:

• --multi_queue_num_threads <# threads>
• --multi_queue_num_queues <# queues>
• --multi_queue_direct_draining <on/off>
• --thread_affinity <off (meaning no affinity, leave OS schedule) or set a list of CPU core ID (the first one is for the main thread), e.g., 0,1,2,3 or 0-3 or 0-1,2-3 or 0,1-2,3>

All the options are tested and work as expected.

Hi @ueqri , these changes look good to me. I have modified the testing script to have it work with these changes locally and tested them on VTR_chain. I have not verified determinism since I assume it should still be deterministic, but I have verified that at least the queue draining is working. I have started a full test suite of Koios_large which I will run all night. It will test Directed (with draining), A* (without draining), A* (with draining), and Dijkstra's (with draining) all on 12 threads; I expect this run to complete within 24 hours. I will then be able to compare the results with the results I have collected previously. If everything turns out green we can merge these changes and lock the code.

Thanks for helping with this change.

[AlexandreSinger]

@ueqri The performance test finished for Koios Large.

At 12 threads, 4 queues per thread:
Directed: Improved by 8.6%
A* (without direct draining): Slowed by 8.83%
A* (with direct draining): Slowed by 30%
Dijkstra's: Slowed by 7.5%

This is being compared to a run I did last week on SAVI4. This was run on SAVI3. Perhaps changing machines caused a slight difference, but I doubt it.

Since A* without direct draining slowed down by around 9%, it appears as though just the act of attempting to drain the queues may be causing some issues. I wonder if this line in the MQ may be the culprit:

I wonder if this should be memory_order_relaxed. But tbh I am not sure. Its very likely that this extra compare in Pop (which is a very very hot function) could be causing the small degredation.

In all honesty we should just merge this since it tells a good story and its not worth busting our heads over this any further. We could hypertune the implementation and squeeze performance by changing how ifs and else blocks are taken; but its not worth our time right now. I will merge this in; perhaps in the future we can better understand why this caused such a difference.

[AlexandreSinger]

LGTM, good job on this.