@mdione @gabrielesvelto in the same flash that contains UEFI. There's a set of headers that describe what is in the flash. That typically includes microcode for the chip generations supported by the motherboard. For example, a board that supports Zen2 and Zen3 will have two microcodes in the flash and the one that matches the CPU installed will be used

@gabrielesvelto @eniko I shouldn’t have read that while sick. Now every time I’m between sleep and wake, I have one of these feverish hallucinations that I’m a little worker inside a CPU core, waiting for a branch prediction to resolve, my hand on the button that dumps everything that was wrongly preloaded.

That’s a very boring job.

@gabrielesvelto #getafuckingblog

@gabrielesvelto thank you for this great, informative overview.
numerous times, i had asked myself if a reported crash could be caused by a hardware bug, and so far i would think i never saw a real case - possibly due to the software i work on running in more controlled environments.
but i would be curious how a crash from a real hardware bug could be classified automatically. do you have pointers to foss tools?

@gabrielesvelto let’s assume .1 major bug per 1kByte binary code. For a 6502 or Z80, you get 6.4 bugs any given time. Now with 16 GByte main memory … it’s the scale that ruins it.

@gabrielesvelto that was super fascinating. Thanks for the thread!

@slink oh yes, we have tools for that. First however I'd point you to my thread about memory errors because those are even more common when analyzing crashes: https://fosstodon.org/@gabrielesvelto/112407741329145666

For crash analysis we have a rust crate to analyze minidumps, which we generate when Firefox crashes. The crate can be used both as a tool and as a library:

GitHub - rust-minidump/rust-minidump: Type definitions, parsing, and analysis for the minidump file format.

Type definitions, parsing, and analysis for the minidump file format. - rust-minidump/rust-minidump

GitHub (github.com)

@slink this crate can detect patterns that suggest a memory error was encountered or that the crash was inconsistent and thus most likely due to a hardware bug. If you check out the output schema of the tool you'll find two fields called "possible_bit_flips" and "crash_inconsistencies" that capture this information: https://github.com/rust-minidump/rust-minidump/blob/main/minidump-processor/json-schema.md

@gabrielesvelto yes, i know the memory error thread, thank you. ECC absolutely is a must and in this regard i am glad that my code (usually) does not run on consumer devices. fwiw, relying on every single bit in a multi-tb ram system still feels scary at times, and it is amazing that these machines actually work.
thank you for the links!

@gabrielesvelto I always thought that the entire Intel CPU architecture was doomed with its over-complicated instruction set. I preferred Motorala designs. Do you have insights on how PowerPC and ARM (Mac) CPUs fare in in this regard? I suspect they use microcode as well, but it may be less complex? But then again, that may allow them to optimize in other areas more, which in turn raises the complexity just as well. (Oh, seeing you replied already, i.e. that your ARM base is much smaller)

@grumble209 @gabrielesvelto Sun cheaped-out on the external cache pathway, using only parity protection rather than the ECC protection that direct competitors (HAL/Fujitsu) were using.

This made the US-II external cache vulnerable to environmental factors (alpha-particle emissions from common packaging materials).

@tempelorg every modern core is like what I described in this thread, regardless of the ISA. The ISA only contributes to the complexity of some parts of a specific design, but the bulk of it comes from these being very high performance cores. The machinery required to reach the current performance levels is what makes these designs very complex.

At lower performance level an ARM core can be simpler than an x86 one all else being equal, but not in the desktop/server space.

@gabrielesvelto x86 instruction complexity alone is unmanageable. A single instruction can be up to 15 bytes long. That's 2^120 possible bit combinations for instructions. So it's already physically impossible to test every instruction individually, let alone test every *sequence* of instructions to find problematic execution sequences.

@gabrielesvelto Intel/AMD had an opportunity to create a clean, easy to decode instruction layout with the transition to 64bit but they failed. http://www.emulators.com/docs/nx05_vx64.htm

@hyc ISA complexity is just part of it. The issue stems from the combination of very large instruction sets and operation modes with very high performance implementations. If you look at something as old and simple as the Cortex A9, even that came with a pretty significant amount of issues: https://documentation-service.arm.com/static/608118315e70d934bc69f13d

@gabrielesvelto yes, it's only a part, but it starts there. The irregular instruction sizes caused problems when instructions straddled cacheline boundaries, etc. Everything after that: superscalar execution, OOOE, all got harder because the simplest case, single instruction in-order, was already non-deterministic.

@hyc it's definitely an added source of complexity for x86 implementations. I remember reading this a few years ago: https://blog.trailofbits.com/2019/10/31/destroying-x86_64-instruction-decoders-with-differential-fuzzing/

@gabrielesvelto seriously, GET A FUCKING BLOG.

#getafuckingblog

@gabrielesvelto this was fascinating, thanks!

@krzysdz @gabrielesvelto Back in my day at least there was lots of latch-based design. The time borrowing through the transparencies was used to make up for timing miscorrelation on the datapath. I remember timing limiters that could be 5+ cycles long.

However, that presumes you have tighter constraints on the clock path. Even a faster-than-model clock path could slow you down.