Cleanup - organize scheduler plugin by their functionality instead of type

[liu-cong]

PURE REFACTOR. NO LOGIC CHANGE.

This was discussed in #801 (comment)

The new way of organizing the plugins avoids confusion as some plugins can implement multiple plugin types (extension points).

Option 1: (Current) A "hybrid" organization based on plugin type (if a plugin only implements one plugin interface) and plugin functionality (if it implements multiple pugin interfaces)

plugins/
|--filter/ (plugins that only implement the Filter interface)
|----lora_affinity.go
|----least_kv_cache.go
|--scorer/ (plugins that only implement the Score interface)
|----queue.go
|----kv_cache.gpo
|--picker/ 
|----random_picker.go
|----max_score_picker.go
|--prefix/ (prefix plugin implements multiple plugin interfaces)
|-----plugin.go
|-----indexer.go

** Option 2: (This PR)** Organize based on the functionality of the plugin, not which interface it implements.

plugins
|--queue/ (queue filter and scorers)
|----queue_scorer.go
|----least_queue_filter.go
|--lora/ (lora affinity filter)
|----lora_affinity.go
|--kvcache/ (kv cache related filter and scorers)
|----leat_kv_cache_filter.go
|----kv_cache_scorer.gpo
|--prefix/ (no change to existing structure)
|-----plugin.go
|-----indexer.go
|--filter-util/
|----decision_tree_filter.go
|--random_picker.go
|--max_score_picker.go

** Option 3: (An improved version of Option 1, credits to @nirrozenbaum ) **

plugins/
├── filter/(...filters that only implement the Filter interface...)
├── scorer/(...scorers that only implement the Scorer interface...)
└── multi/ (plugins that implement multiple interfaces)
└──── prefix/(... your prefix files...)
└──── MY_PLUGIN_THAT_IMPLEMENTS_MULTIPLE_INTERFACES

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	d9db194
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/6823755f247dea0008ce1767
😎 Deploy Preview	https://deploy-preview-828--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: liu-cong
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liu-cong]

@nirrozenbaum @oglok

[oglok]

I think it makes sense to just call them "plugins" and organize them by folders. It'd be good to have developer documentation with the set of interfaces a plugin can implement, and what's the workflow (which stage is first, and what it's for...)

[nirrozenbaum]

@liu-cong Thanks for working on this and sorry for the delayed response (I was on a day off).
overall the PR looks good, but I'm having seconds thoughts on whether we should go that direction.

The new way of organizing the plugins avoids confusion as some plugins can implement multiple plugin types (extension points).

my second thoughts are a result of this sentence.
when seeing the plugins list now, I was much more confused.
I was looking on the list and trying to think - which plugin implements filter? which is scorer?
(I assume for you it was clear, but that could be because you're the author of the plugins :) )
I think this makes it harder to get familiar with scorer/filter/other plugins for a newcomer.
it made me think twice about previous discussion we started on #801.
an even more confusing part is that the picker and filter directories remain (which I agree with that choice if we do change the structuring).

on the other hand, it's also confusing to put under /scorer dir a plugin that implements multiple extension points.

I'm thinking about having a mixture to try and simplify (which I think it will, happy to get your opinion here).
maybe something along these lines:

• if a plugin implements only one extension point, it's located under the matching dir
  (e.g., capacity filter is under /filter dir). I believe that at least on this early stage of GIE, most plugins will go into this category.
• for plugins that implement more than one extension point, we create a new dir (maybe something like multi or multi-function) and under that dir we use directories for plugins as suggested in this PR.

so at the end the structure will look like:

/plugins/
├── filter/(...filters that do no implement other interfaces...)
├── scorer/(...scorers that do no implement other interfaces...)
└── multi/
└──── prefix/(... your prefix files...)
└──── MY_PLUGIN_THAT_IMPLEMENTS_MULTIPLE_INTERFACES

wdyt?

[liu-cong]

Thanks @nirrozenbaum for the suggestion. I updated the description to include all 3 options, please take a look.

I see both option 2(this PR with some improvements, more aligned with kube-scheduler) and 3(your idea) work. I don't really mind which one we pick, we just need to pick one and add a README to document the rules. @ahg-g Can you help break the tie?

I was looking on the list and trying to think - which plugin implements filter? which is scorer?

One way is to just name the files like lora_affinity_filter.go.

[nirrozenbaum]

keep in mind that in kube-scheduler the majority of the plugins are implementing more than one extension point.
so while option 2 might be best for kube-scheduler it doesn't mean it's the best here also.
I think that either structure we choose, we're going to have at least one argument for why the other option is better.
I believe we should just follow the pattern that fits the majority of the plugins, which atm is splitting by the extension point (option 3), while still having an easy to understand structure for multi plugins (plugin that implements multiple extension points).

I think option 3 also aligns very nice with the other ongoing PR #835, which allows setting WithFilters, WithScorers, etc.. and also WithPlugins (which is intended for multi plugin, but can be used also for a single interface plugin).

[nirrozenbaum]

@liu-cong btw, if both works for you, we can proceed with option (3) 🙂

[liu-cong]

great, just opened a new PR for option 3: #837

@nirrozenbaum pls review the new PR, thanks!

[ahg-g]

My preference is to organize the plugins based on the feature they represent, not the exact extensions they implement, so option #2, but going with option 3 is not a deal breaker for me.

[oglok]

I see the Pr has been already merged, but I agree with @ahg-g , I like the feature approach (option 2). I think it'll be more clear for developers, as long as we document the interfaces a plugin can implement.