4 weeks ago
4 weeks ago - last edited 4 weeks ago
I'm in the process of looking over the crash logs in this post, starting at the end and working my way back. So far, all these crashes are the Intel CPU crash.
As mentioned in another thread, these are always fixed by lowering your clock speed, even if you aren't overclocking. That is the one and only known fix. It appears to be a timing flaw in Intel hardware executing a specific instruction sequence. It isn't related to the program being too intense for the CPU, or to Apex doing anything wrong.
We are trying to change this function to not expose this crash on Intel CPUs in a future patch, but I can't promise when that change will come out.
@OrioStorm , you can't blame Intel CPUs for bad coding in the game. Honestly, this is the first game ever that crashes for me like that on an overclocked CPU.
My chip is brand new - I7 9700K, stable overclock since I bought it at the beginning of this year.
5.1GHZ rock solid stable, playing also Battlefield 5 and Metro Exodus. I also play other games, but those two are the pioneers of "stress testing".
As a fellow developer I will ask you this - why is my system stable in all other games, that are way more CPU demanding than Apex Legends, and programs 24/7, but it's not stable in Apex Legends? Because of a flaw in Intel's chips? Come on, man.
On 5.1GHZ, I need to put an AVX offset to x4 in order for the game to work. If I use the stock MCE overclock of 4.6GHZ there are no crashes apparently as well. Problems begin to happen after 4.6 GHZ, so I need to do the above for the game to work.
I hope the above helps in some way. This is a problem on your side, not a problem on Intel's side. If you don't fix it soon, there will be more and more players quitting the game.
I understand that those issues are hard to locate and fix, but this is fundamental to the player fanbase and the game's future growth.
4 weeks ago
Thank you very much I did not expect personal feedback I really appreciate its nice to know what is the cause of the crashes, it's unfortunate that it's a chip problem but still thank you for your time!
4 weeks ago
Product: Apex Legends
Please specify your platform model. PC
AMD or Nvidia Model Number Geforce RTX2070
Enter RAM memory size in GB 16 GB
What is your gamertag/PSN ID/EA Account name? DevinpepperZ
Where did the issue occur? In a match
Which part of the map or menu were you in? If you don't remember the exact name, please describe the area or what you were trying to do in the menu. Any
What were you doing when the issue occurred? E.g. respawning a squad mate at a beacon, using an ability, a squad mate left the game. Normal gameplay just walking around
Did your squad mates also experience the issue? Unclear
How many matches had you played in a row before the issue occurred? 1
When did this happen? (dd.mm.yy hh:mm) 26/04/2019
How often does the bug occur? Every 1-2 games
How severe is your issue? Major impacts to gameplay
What happens when the bug occurs? Crash to desktop and it tells me there is a crash report in documents
What should be happening instead? Not crashing
Steps: How can we find the bug ourselves? Unclear
4 weeks ago
@S7ORM_BR1NGR, I understand your frustration. I'd be just as frustrated if one program crashed constantly on my PC.
I also completely agree with you that I can't blame Intel CPUs for flaws in the game.
So, let me explain why I make this conclusion. It is not a conclusion I reach lightly.
First, the TL;DR version:
Now, let me go into more detail.
First, I don't blame the CPU lightly. If you'll notice, I don't even blame faulty CPUs for all the crashes reported in this thread. But, I do think I'm qualified and have done the investigation to conclude that this is an Intel flaw. I have a Master's degree from Stanford University in Electrical Engineering, specializing in microprocessor design, so I have a better understanding of how CPUs work than most people. I've also worked in the game industry for 20 years, and have fixed many bizarre bugs, some of which relied on how CPUs handle rare edge cases. I've also found and worked around several compiler bugs, but I still trust the compilers, and don't assume it's a compiler bug until the disassembly proves it to me. With all that experience, this is the very first time I've ever considered the possibility that the cause of a bug was the CPU itself malfunctioning.
Also, I implemented the crash log feature so that I could find out why Apex was crashing and then fix it. I was frustrated that we didn't get real information from Microsoft crash reports, and hated that Apex had a reputation for crashing, so implemented a way for me to get the crash info myself. On the first pitch that included this log, we found that 90% of the crash bugs were bizarre, but could be explained by one thread stomping another thread's memory. That was a really difficult bug, but I tracked that down and fixed it, and we had patched the game in under a week. Of the remaining bugs, there were 1 or 2 other minor one-off crashes we fixed, then about 4% were crashes in other modules (not our code), and about 5% were crashes in this same function that still crashes on Intel chips. Those last crashes were impossible as well; usually, an instruction at one address (based on RIP) was complaining that it didn't have permission to execute an instruction at a different address (based on the crash exception type and address). That other address was always the address of the start of the next instruction, with some or all of the lower 32-bits forced to 0. This is impossible for a properly functioning CPU. I can't come up with any way to get this to happen, even with hand-written assembly language running at the highest privilege level. The instruction to execute is RIP by definition, so execute violations should only ever happen at an address equal to RIP.
In the latest patch, the most common version of the Intel crash is that the instruction "test r15,r15" is interpreted as "int 3", but sometimes that instruction also complains it can't write to a memory address that isn't even in any register. If you look up Intel's x64 instruction manuals for the "test" instruction, you'll see that this form of it shouldn't be able to trigger any exceptions, much less these. Again, the evidence shows that this crash is impossible for a properly functioning CPU.
This crash is also in the middle of some instructions that run every iteration of a loop. From matching the disassembly to the code, I can find the loop count in a register, so I can tell that the loop count is always greater than 100 and sometimes over 3000 at the time of the crash. I estimate that it crashes about 0.0000001% of the time through the loop for people who are experiencing the bug. But this loop runs so much, that translates to crashing once every 2 to 3 games, or even every game if you're unlucky.
So, rigorous investigation of the evidence points me to this bug lying with Intel CPUs, and not our code.
Now, you bring up a legitimate question. Why does Apex crash, when other more demanding software doesn't? You mentioned that Battlefield 5 and Metro Exodus run fine. Other people experiencing this crash have passed 24-hour stress tests designed to expose rare overclocking errors. Why would these more demanding things work, but Apex fail?
Simply put, this crash is not caused by overworking the CPU. This crash appears to be caused by the CPU not finishing part of one instruction before a following instruction uses its results. That could depend on the exact sequence of instructions and branches that were encountered preceding the crash, and with out-of-order chips it may even depend on future instructions. Even though Intel is very, very good at manufacturing chips (in addition to being very, very good at designing chips), not all chips come out of the manufacturing process identical, and not even all cores on one chip. That's why they rigorously test each CPU to determine what clock speed it's stable at and how many cores came out working right. So, it's very possible that one person with this chip will have no problems, and another person with this chip will have constant crashing. It's even possible that one core will work fine, but another core on the same chip will crash frequently. It could also show up in a patch that shouldn't have changed the code at all, just because the compiler's link time code generation made different choices.
Finally, I really don't like that this is a bug in Intel's CPUs. If it was a bug in our code, I could just fix it. Since it's a bug in their chips, it's not that easy. But I'm not content to just give up and let our players crash all the time. So, I've locally modified this function in our code to do the same thing in a slightly different way. This will cause the compiler to generate different instructions, which will hopefully avoid this Intel flaw and stop the crashing. This change will come out in a future patch.
So, hopefully this satisfies you that we're not just blaming Intel's CPUs to avoid the hard work of fixing our own code. The only explanation consistent with all the evidence is a flaw in Intel's CPUs, and we're changing our code anyway to try to work around it.
4 weeks ago - last edited 4 weeks ago
Thank you, @OrioStorm , I respect that. It's always good when there's a logical answer and transparency when it comes to issues like these. It wouldn't be that of a problem if other games were crashing like Apex, but since Apex became popular and many people play it, it's a different story.
I really hope your workaround will fix this, it would be great if I and other players can play on 5GHZ and even more.
When are you planning to release the patch?
4 weeks ago
Hello everyone, while I'm playing apex my game crash every match, I noticed so far that there is 2 types of crash, first It freeze and then the game closes itself and the second it just closes the game and send me an error that says something about my game crashing and to go and check my documents folder to find the crash report, I'm gonna attach my crash report.
4 weeks ago
@Mewinator, that crash report is from the first patch that had the crash log feature. It's the crash that we fixed in a patch later that week.