@slink oh yes, we have tools for that. First however I'd point you to my thread about memory errors because those are even more common when analyzing crashes: https://fosstodon.org/@gabrielesvelto/112407741329145666

For crash analysis we have a rust crate to analyze minidumps, which we generate when Firefox crashes. The crate can be used both as a tool and as a library:

GitHub - rust-minidump/rust-minidump: Type definitions, parsing, and analysis for the minidump file format.

Type definitions, parsing, and analysis for the minidump file format. - rust-minidump/rust-minidump

GitHub (github.com)

@slink this crate can detect patterns that suggest a memory error was encountered or that the crash was inconsistent and thus most likely due to a hardware bug. If you check out the output schema of the tool you'll find two fields called "possible_bit_flips" and "crash_inconsistencies" that capture this information: https://github.com/rust-minidump/rust-minidump/blob/main/minidump-processor/json-schema.md

@gabrielesvelto yes, i know the memory error thread, thank you. ECC absolutely is a must and in this regard i am glad that my code (usually) does not run on consumer devices. fwiw, relying on every single bit in a multi-tb ram system still feels scary at times, and it is amazing that these machines actually work.
thank you for the links!

@gabrielesvelto I always thought that the entire Intel CPU architecture was doomed with its over-complicated instruction set. I preferred Motorala designs. Do you have insights on how PowerPC and ARM (Mac) CPUs fare in in this regard? I suspect they use microcode as well, but it may be less complex? But then again, that may allow them to optimize in other areas more, which in turn raises the complexity just as well. (Oh, seeing you replied already, i.e. that your ARM base is much smaller)

@grumble209 @gabrielesvelto Sun cheaped-out on the external cache pathway, using only parity protection rather than the ECC protection that direct competitors (HAL/Fujitsu) were using.

This made the US-II external cache vulnerable to environmental factors (alpha-particle emissions from common packaging materials).

@tempelorg every modern core is like what I described in this thread, regardless of the ISA. The ISA only contributes to the complexity of some parts of a specific design, but the bulk of it comes from these being very high performance cores. The machinery required to reach the current performance levels is what makes these designs very complex.

At lower performance level an ARM core can be simpler than an x86 one all else being equal, but not in the desktop/server space.

@gabrielesvelto x86 instruction complexity alone is unmanageable. A single instruction can be up to 15 bytes long. That's 2^120 possible bit combinations for instructions. So it's already physically impossible to test every instruction individually, let alone test every *sequence* of instructions to find problematic execution sequences.

@gabrielesvelto Intel/AMD had an opportunity to create a clean, easy to decode instruction layout with the transition to 64bit but they failed. http://www.emulators.com/docs/nx05_vx64.htm

@hyc ISA complexity is just part of it. The issue stems from the combination of very large instruction sets and operation modes with very high performance implementations. If you look at something as old and simple as the Cortex A9, even that came with a pretty significant amount of issues: https://documentation-service.arm.com/static/608118315e70d934bc69f13d

@gabrielesvelto yes, it's only a part, but it starts there. The irregular instruction sizes caused problems when instructions straddled cacheline boundaries, etc. Everything after that: superscalar execution, OOOE, all got harder because the simplest case, single instruction in-order, was already non-deterministic.

@hyc it's definitely an added source of complexity for x86 implementations. I remember reading this a few years ago: https://blog.trailofbits.com/2019/10/31/destroying-x86_64-instruction-decoders-with-differential-fuzzing/

@gabrielesvelto seriously, GET A FUCKING BLOG.

#getafuckingblog

@gabrielesvelto this was fascinating, thanks!

@krzysdz @gabrielesvelto Back in my day at least there was lots of latch-based design. The time borrowing through the transparencies was used to make up for timing miscorrelation on the datapath. I remember timing limiters that could be 5+ cycles long.

However, that presumes you have tighter constraints on the clock path. Even a faster-than-model clock path could slow you down.

@gabrielesvelto great thread! Thanks!

@shelldozer @grumble209 @gabrielesvelto I've probably had more of those UltraSPARC-II's pass through my hands than any other CPU. (I had four maxed-out E4000's at home at one point.)

I had a friend in the 90s who had a job at DEC one summer writing a program that output random but legal C, to stress-test their compiler.

@gabrielesvelto The 6502 had no bugs. Just some undocumented features.

@mdione @gabrielesvelto The small flash chip that holds the UEFI and other firmware components also has the µcode patch on it. The chip sits on a simple bus (usually SPI), so it can be directly wired into the CPU and accessed immediately after the system comes out of reset.

@gabrielesvelto how about a blog post ?

@usul je n'ai pas l’énergie pour fair un blog post, ça serait très long