Increasingly prominent latency issues lately (Not sure if Xbox specific)

[Carnivol]

For the last couple of seasons, it feels like the game's gotten worse and worse in terms of latency, and I'm not sure what it stems from.

Basically, way too often does my group experience that we're in a fire fight and flee around a corner, through a door, safely reach cover, etc. and then drop dead after we're very clearly out of harms way. Or, sometimes we'll be firing at an opponent and can audibly hear our weapons connect multiple hits, but no damage counter pops, and then we drop dead and then team mates firing at them also or spectator mode shows that said opponent has clearly not taken any damage from the hits we audibly connected.

Initially, I thought this problem came from extra system layering after the addition of cross-play, but I'm not entirely sure anymore -- seeing as it feels like this problem is just getting increasingly worse and worse.

I usually play with the same group all the time, with one player on Xbox One X and two players on Xbox Series X.

All of us are on 100/100 mbps (or better) fiber connections (with SpeedTest giving us a ping of 2~6 ms). We're all connected to the same data centre, and we all have a reported ping of roughly 20~24 ms to the nearest/best data centre (and we all use the same data centre).

In-game, our ping counters usually rest at 30 ms during most gameplay. One of us has the occational spike up to 45, but it usually drops down to 30-ish after the spikes.

In general, it "feels" like there's another large delay or something going on. And this often gets very apparent if we're eliminated and spectate a team mate -- as we can often hear in our Xbox Party chat that our team mates respond and react to something in-game before we even get to see it as spectaters. We're not talking seconds or anything, but the delay's large enough to notice.

If we encounter opposing players with stuttering movement, it also often feels like they're the ones with an advantage -- as shooting them will often trigger audible connects, but ultimately result in no actual damage.

We do not experience this level of latency issues in other online multiplayer games And whilst we've experience the occational incident like this before, these issues have been increasingly more prominent during the past few months and feels like the constant state of Apex Legends when we play it now. Kinda frustrating, especially when the non-existant(?) matchmaking frequently puts us in queues with people who seem to permanently reside within the game. (Matchmaking is of course a completely different issue, but it's clearly an issue when we get matched with people who have more playtime, kills and assists for the current season than our entire squad have collectively since launch). 

Not sure if there's anything we can try on our end, if there's a problem with our party setup, or anything else we can try to remedy this latency issue. The only thing we've tried, but which was entirely futile and we couldn't verify if improved things, was to turn off crossplay.. However, since there's no easy way to turn that off in-game on the Xbox version (you need to disable cross play on a system level, which is a convoluted mess, instead), this results in no one actually playing "Xbox Only" and you're stuck in an endless queue of never finding players.