Code

Add P/D scheduler#115

Code

Merged

mayabar merged 20 commits intoneuralmagic:devneuralmagic/gateway-api-inference-extension:devfrom

mayabar:pd-schedulermayabar/gateway-api-inference-extension:pd-schedulerCopy head branch name to clipboard

May 5, 2025

Commits

Commits on May 4, 2025

• Add P/D scheduler - use 2 schedulers in it, one for prefill and one for decode. P/D scheduler is enabled by environment variable value, list of scorers and their weight are defined by environment v…

  Show description for bf593dd
  mayabar

  committed
  bf593dd

  Copy full SHA for bf593dd

• Remove unused variable

  mayabar

  committed
  a694eb7

  Copy full SHA for a694eb7

• Update readme file with envirnment variables relevant to P/D scheduler

  mayabar

  committed
  3ac82b2

  Copy full SHA for 3ac82b2

• Merge branch ‘dev’ into session_affinity_scorer4

  Show description for b6f4062
  mayabar

  committed
  b6f4062

  Copy full SHA for b6f4062

• Fix problem caused by merge

  mayabar

  committed
  f43d9c7

  Copy full SHA for f43d9c7

Commits on May 5, 2025

• Add documentation for PDScheduler.Schedule function

  mayabar

  committed
  9948464

  Copy full SHA for 9948464

• Update names of prefill and decode filters to avoid spaces

  mayabar

  committed
  b649586

  Copy full SHA for b649586

• Update comment for prefill/decode fitlers

  mayabar

  committed
  16e4496

  Copy full SHA for 16e4496

• Change IsPDEnabled to PDEnabled

  mayabar

  committed
  d58cb14

  Copy full SHA for d58cb14

• Fix typo in readme

  mayabar

  committed
  542984e

  Copy full SHA for 542984e

• Fix pd scheduler behavior for short promprts

  mayabar

  committed
  2ef8e19

  Copy full SHA for 2ef8e19

• Fix prefill/decode related text in readme

  mayabar

  committed
  c8e2aa6

  Copy full SHA for c8e2aa6

• Remove redundant filter creation of prefil/decode filters + make promptLengthThreshold local

  Show description for 5fb506e
  mayabar

  committed
  5fb506e

  Copy full SHA for 5fb506e

• Fixes in readme

  mayabar

  committed
  a993bc7

  Copy full SHA for a993bc7

• fix compilation prblem

  mayabar

  committed
  205f34a

  Copy full SHA for 205f34a

• add pd scheduler test

  mayabar

  committed
  c443a6b

  Copy full SHA for c443a6b

• merge from dev

  mayabar

  committed
  d83c5f6

  Copy full SHA for d83c5f6

• add postResponse plugins array to prefile and decode config

  mayabar

  committed
  e8d9f64

  Copy full SHA for e8d9f64

• fix comment in test

  mayabar

  committed
  2d0de9e

  Copy full SHA for 2d0de9e

• fix pd-scheduler test

  mayabar

  committed
  6c41600

  Copy full SHA for 6c41600