Add P/D scheduler #115

mayabar · 2025-05-04T14:22:53Z

Ref #114
*Still working on tests

…or decode. P/D scheduler is enabled by environment variable value, list of scorers and their weight are defined by environment variables + delete pd-filter

* dev: Added a simple unit test for the PostResponse plugin invocation Invoke the PostResponse handlers and send any added headers to the user Added code to scheduler to enable running the PostResponse plugins Use an init() function instead of modifying the scheduler code to inject our config Added PostResponse to scheduler config Fixed scorer tests

pkg/epp/scheduling/pd_scheduler.go

pkg/epp/scheduling/plugins/filter/pd_filter.go

pkg/epp/scheduling/pd_config.go

…ptLengthThreshold local Add function for schedulerContext creation

shmuelk

I reviewed this PR with Maya and after the changes she made it LGTM

* Add P/D scheduler - use 2 schedulers in it, one for prefill and one for decode. P/D scheduler is enabled by environment variable value, list of scorers and their weight are defined by environment variables + delete pd-filter * Remove unused variable * Update readme file with envirnment variables relevant to P/D scheduler * Fix problem caused by merge * Add documentation for PDScheduler.Schedule function * Update names of prefill and decode filters to avoid spaces * Update comment for prefill/decode fitlers * Change IsPDEnabled to PDEnabled * Fix typo in readme * Fix pd scheduler behavior for short promprts * Fix prefill/decode related text in readme * Remove redundant filter creation of prefil/decode filters + make promptLengthThreshold local Add function for schedulerContext creation * Fixes in readme * fix compilation prblem * add pd scheduler test * add postResponse plugins array to prefile and decode config * fix comment in test * fix pd-scheduler test

mayabar added 4 commits May 4, 2025 16:45

Add P/D scheduler - use 2 schedulers in it, one for prefill and one f…

bf593dd

…or decode. P/D scheduler is enabled by environment variable value, list of scorers and their weight are defined by environment variables + delete pd-filter

Remove unused variable

a694eb7

Update readme file with envirnment variables relevant to P/D scheduler

3ac82b2

mayabar requested review from elevran, shmuelk and vMaroon May 4, 2025 14:22

Fix problem caused by merge

f43d9c7