Servers: Troubleshooting for EA/Respawn and Multiplay [ Edited ]

[apostolateofDOOM]

Let's put the boot on the other foot...

I'll put myself out there and say i think it's pretty unanimous that performance isn't that great in the games current state that's only getting worse as each season is added and before i start, i would just like to say that this isn't a rant (your servers are trash) thread and networks, servers, code and game development aren't my area of expertise.

I'm only mentioning things I've read briefly (some of which will be copy/pasted from online sources for my own convenience) or mentioned through discussions here that could be a cause or possibly be a part of the problem and i understand from a backend point of view that this is a process of evaluating costs and benefits. We need some benefits now as more than enough users have given plenty over the past 2 years to cover those costs. No excuse when you charge 150+ for an heirloom that's 3x the price of a typical AAA title. If you're to be so bold to expect us to pay that amount shoving these events down our throats each season then you can bet i'll have the audacity to request a decent live service when it's pretty sub par in comparison to most FPS titles either current or in the past that don't charge anywhere near that amount.

I'm trying to find a better experience than what i currently have (beyond the realms of user troubleshooting that i have already done multiple times) and hopefully for others too. Let's exclude the bare min specs users playing on wifi or bugs like sound issues they can't replicate as that isn't what i'm getting at nor that games are ok for most users to play at 30fps when we live in an age of 240hz + monitors and EA, the global leader in digital interactive entertainment officially advertising Apex as a competitive shooter in tournaments with a large prize pool.

Now, some of the problems that we're experiencing could be because of some of these things mentioned or totally unrelated. Point of this discussion isn't for replies of why they can't do this or improve that, but a workaround, compromise or solution that might actually resolve or improve some issues for a lot of users that they (EA/Respawn or Multiplay) can do on their end.  So, if you apparently know what you're talking about then i don't want excuses.. Problems are there in abundance...

They need addressing. ASAP.

Matchmaking Server - Server selection - Ping limitations per match.

From my limited understanding, Respawn has a dedicated matchmaking server that puts you through to Multiplays Hybrid Cloud solution that can place you into different servers that's integrated with the 3 biggest public clouds, Google Cloud, Microsoft Azure and AWS in over 190 data centers but the cloud machines are more unstable, due to weaker processing power, differently tuned buffering, maybe less optimal/reliable routing, etc..

SBMM, region, ping and other metrics to determine where to put you in a match and they can also migrate players to other data centers because of these things or to decrease que time which they're trying to keep very low for obvious reasons also effecting your matches.  Developers want to provide a consistent, curated, experience for players while some players very much want complete control over their experience.

As selecting our data center and instant play (ready now) is our ONLY option which will take the above into consideration when joining a match, the priority of providing you a smooth experience will decrease with longer que times and the problem made worse in off peak hrs in your region. This doesn't just apply to you, but all 60 players in a match so once you wait x amount of time then it will fill these spots with players that have high ping from another country and possibly even another continent to you.

Playing with a high ping means game lags, which adversely affects the player's game. Most times, with high ping, many servers of FPS games automatically disconnect the player and kick them out of the game but we don't have this in Apex.  Latency is the time measurement of exchanging information (ping) from one computer to the server.

What makes this worse is every time you or i join a new game it could be with a different server or provider and also potentially under load effecting performance causing micro stutter, slow mo, hitreg and delayed response (getting shot through doors or behind cover) and whatever else. These will already exist but be made much worse and this inconsistency is infuriating for those playing the competitive mode with RP on the line when trying to climb in the higher tiers of ranked.

Here's brief explanation of high ping, peekers advantage and lag compensation.

Guide to playing with or against high ping in FPSs that have lag compensation (kovaak.com)

With our performance display with limits pre set by what Respawn thinks is acceptable like ping 170+ until it shows the ping icon for example, it would be nice to set our own limits with more information and have network round trip included like Valorant.

A quote from Dave Heironymus, technical director of Valorant at Riot Games.

“The Valorant team has put extensive effort into determining the best combination of tick rate and latency that will minimize peeker’s advantage, and those tests showed that a 128hz tick rate and 35ms (or less) latency would be best for our players.” and “Blind tests also showed that elite FPS players … can reliably detect when the game is running at a lower tick rate.”

Most Adept FPS players look for performance which has led to Valorants success as let's be honest the movement, abilities, characters and all round generally, Valorant is an inferior game to Apex with performance being its main attraction. If Apex goes along these lines it would bring some of those players back and be good for the game overall.

With the addition of series one ranked in Season 2 and it becoming more refined for competitions over time with tweaks to the RP system, legend balance and further map releases and changes, the foundations are there for Apex to rise in esports. Many more improvements can still be made to ranked that i'll push for but that's beside the point.. Having a server selector and instant play is doable and i think having the extra option or choice, especially for those climbing the higher tiers or potentially wanting to play Apex professionally is a good thing imo.

Perhaps that's complicated in the current structure through Multiplay (i don't know) but could that be changed? Having random and unstable experiences at that level is causing unnecessary frustration for those future stars that could potentially be advertising the game and end up sending them to other games that have more focus on aspiring competitive players.

Network, packets, servers and server tick rate.

We all heard this before and it's the worst amongst ALs biggest competitors. I don't understand the whole sending packets with pigeons but just remove the crap, streamline and optimise it. Same with the ''netcode''. Just as you can remove all the fliers, prowlers and such if it's draining server resources. Have that content in the game by all means for those that want PvE but if it's hindering competitive PvP matches then remove it or have it separate.

Most games that have upgraded server ticks has been noticed by most just as you would notice playing on 30hz TV then going to a 60/75hz or from that to 144/240hz. Also, give us the option for higher client CPU/bandwidth usage. Clients could just choose their own rates. You apparently have a 10 year plan? Invest in it.

Here's some old videos from Battle(non)sense that are still relevant today.

The most important part to take away from this is THEY CAN improve it but either the developers don't get the resources or don't feel the need to make these changes.

There is also the fact pay-to-enter servers like ESEA, FACEIT, and CEVO, the last of which has locations in New York, London, France, Sydney, and more, all run on 128-tick environments, but with the added catch of being paid services.

Rumour is (so take with a pinch of salt) Respawn have a 6 year contract with Multiplay for their hybrid cloud server solution. Now i understand the benefits of this originally to address scaling issues but now the player base has settled into what could potentially be roughly estimated levels for new seasons release and throughout, could this be optimised? Or have separate agreement/contract for Ranked servers?

Crossplay

Intentional limits to accommodate inferior hardware - I don't know how much weight this carries or any of the insides of this but have separate versions of the game if it's true or let us toggle it. This is assumed to be worse with switch this month and the already broken ''lag compensation'' in conjunction with AA (aim assist) sounds like it's not fostering a very fair competitive environment.

We already have enough exploits in this regard.

Edit: Recently particle FX count was lowered for Caustic Gas (tac and ult), Bangalore Smoke, and Gibraltar ult likely to accommodate Switch users and reduce GPU load.  

https://twitter.com/GH057ayame/status/1367702094426173442?s=20

Word was around that other features had been changed long before this that weren't included in the patch notes and likely more in the future too.  It was mentioned in a separate thread that Respawn can tailor the game to specific platforms so why they have done this across the board that will negatively effect most users is unknown.

One size clearly doesn't fit all.

Code refactoring

It's always done to optimize things, it's usually happening where/when the code gets too messy because of a lot of patching or when there's new tech or device related features available that the game could benefit from. As the server issues are more prevalent when a new season or large update comes out which usually don't get fully fixed, i would imagine the pressure of keeping up with these content update schedules has had a negative impact on the game over time and while i appreciate they will constantly work on this, it's evident that many QoL issues have been neglected because of time, effort or resources required to fix them properly.

Lower the amount of players in a normal match, different game mode or LTM.

I think most have noticed a sluggish response at the beginning of the match on all platforms at some point. This issue has been prevalent since release when it was a very common occurrence and although it has improved since then (even if it's minor, credit where it due) we still have the same problem occasionally. This is also tied to the CODE: X,Y and Z names Respawn gave to address where the problem is so they can locate and fix it. Due to lack of communication we don't know what progress has been made but it still occurs, even if infrequently. 

Just to note, Respawn have highlighted in Trello about the slow matches that's been happening a lot recently. 

I'll keep updating this but feel free to add anything in the meantime...

An article that mentions Respawn are aware and that they'll address or at least communicate about servers issues.

Apex Legends Season 8: Fans Upset With Servers and Connection Issues (dbltap.com)

[apostolateofDOOM · Accepted solution]

@Shertgun Respawn... https://twitter.com/i/status/1379338747846688769

[DoYaSeeMe]

@apostolateofDOOM *cracks knuckles* :D

[dougieee]

This game like all other BR`s is designed with casting the net as wide as possible in mind, to gather the biggest player base possible, everything youve mentioned would alienate segments of that player base.

Improving server performance would kill the potatoe pc`s about 30% of players

Ping limit is something i feel could be tweaked (even it evven exists), there is a big difference between trying to play with someone a couple of thousand miles away and squads that VPN in from china with a probable 400 ms, i suspect its those squads that are hurting server performance.

All in all there is a complete lack of transparency from devs in all BR`s and i suspect that is because if you knew just how poorly the servers were running you would be disgusted.

If you go back to BF3 days and third party server rental, they were managed independantly, high pingers (that trashed server performance) could be kicked, servers were managed for the good of the players. Now servers are manged for the good of devs bottom line

[BaldWraithSimp]

@apostolateofDOOM I have never experienced that the game has put me on any server other than the one I selected. Here is how you can check:
Go to PC Bahrain servers
Try to find a game, you will be stuck in infinite matchmaking cuz the server is dead.

[DoYaSeeMe]

@BaldWraithSimp Bahrain is a pretty isolated data center, if considering those that are accessible to Apex. I think passing players to other data centers happens only within a certain radius, like 50-100ms from the original data center.

[hayhor]

@BaldWraithSimp I experience it daily. Pings range from 24ms to 70 ms because I get moved around the US like a cheap trick.

[apostolateofDOOM]

@dougieee Lack of transparency really doesn't help the matter and i think more are not only becoming aware of some of these issues but also educating themselves to find where the problem lies as most are having them or just moving on.

As to servers improvements killing potato PC's i don't know if that's entirely true, i mean in some instances maybe but in others you would see a difference as these are issues effecting everyone irrespective of platform from what i can tell.  Why i also suggested clients could just choose their own rates so everyone would still have what they have now but those that do have the hardware or internet to push themselves to a more acceptable level for most modern games, can do. If you're running a 10 year old rig or old console then it's time to upgrade now anyway. 

Privately run servers has been suggested before but Respawn have said they don't want to pass the buck on to the community. I don't know if that will change or even an option in the future with our current instant play being the only choice but i'll put that into the thread later.

@BaldWraithSimp I often get put into mainland Europe servers with London selected.

@DoYaSeeMe I don't need your negativity and excuses here! ;D

I welcome the input but i want to hear improvements or solutions and not reasons to justify why this current state is the best they can do. I understand with the launch of Apex of how things were but time has moved on and by their own admission 40% of players are in the competitive mode so i'm looking at it from that and a performance lens.

There's other reasons why they play ranked instead of casual but that's another issue..

[DoYaSeeMe]

@apostolateofDOOM Maybe I'm trying to tell you that Santa doesn't exist. Is that negativity? :P

[apostolateofDOOM]

@DoYaSeeMe If he does or not still doesn't mean i want coal for Christmas!

[apostolateofDOOM]

@DoYaSeeMe I thought you would be in your element with this.

Are you trying to say there's nothing they could do on their end?