Rollup merge of #131652 - compiler-errors:modifiers, r=Nadrieril,jieyouxu

Move polarity into `PolyTraitRef` rather than storing it on the side

Arguably we could move these modifiers into `TraitRef` instead of `PolyTraitRef`, but I see `TraitRef` as simply the *path* part of the trait ref. It doesn't really matter -- refactoring this further is much easier now.

Author: matthiaskrgr