> Tight loops come from the uop cache, and branchy code gets 3-way decoding.
First, there is no uop cache on the "mont" cores.
Second, Intel aren't decoding both sides of the branch.
That wouldn't actually help much, as modern branch predictors are correct well over 99% of the time. It would be a waste of silicon and power to have an extra decoder producing work which simply decoded most of the time, and an even bigger waste to have two extra decoders.
Intel's actual approach is way more clever; They run the branch predictor ahead of the decoders by at least 3 branches (probably more). The branch predictor can spit out a new prediction every cycle, and it just plops them on a queue.
Each of the three decoders pops a branch prediction off the queue and starts decoding there. At any time, all three decoders will each be decoding a different basic block. A basic block that the branch predictor has predicted that the program counter is about to flow through. The three decoders are leap frogging each other. The decoding of each basic block is limited to a throughput of three instructions per cycle, but Skymont is decoding three basic blocks in parallel.
The decoded uops get pushed onto three independent queues, and the re-namer/dispatcher merges these three queues back together in original program order before dispatching to the backend. Each decoder can only push three uops per cycle onto its queue, but the re-namer/dispatcher can pull them off a single queue at the rate of 9 uops per cycle. The other two queues will continue to fill up while one queue is being drained.
The branch prediction result will always land on an instruction boundary, so this design allows the three decoders to combine their efforts and maintain a throughput of 9 uops per cycle, as long as the code is branchy enough. It works on loops too, as far as I'm aware, intel doesn't even have a loop stream buffer on this design; The three decoders will be decoding the exact same instructions in parallel for loop bodies.
But Intel have a neat trick to make this work even on code without branches or loops. The branch predictor actually inserts fake branches into the middle of long basic blocks. The branch predictor isn't actually checking an address to see if it has a branch. Instead it predicts the gap between branches, and they simply have a limit for the size of those gaps. Looks like that limit for Skymont is 64 bytes (was previously 32 bytes for Crestmont)
Thank you for that explanation, I was confused as to what was happening with the multiple decoders. That's a wild way to implement a processor front end.
Oh, I thought the uop queues were uop caches when I looked at the diagram. Not having loop handling does seem off, but I guess with long loops they will just alternate between the decoders.
The whole 99% branch prediction thing is sort of misleading - most branches are loops taken for a constant amount of time, so most are perfectly predictable, and most others are error checks which are also easy to predict. However, a large amount of comparative wall time in code is spent on sequences of a short piece of code and a branch that is hard to predict. Without hyperthreading, I would assume that decoding both sides of the branch would actually help a lot in these circumstances. It sounds like Intel is possibly capable of doing that.
The synthetic basic blocks are also an interesting idea given how hard it is to figure out where an x86 instruction boundary is. It's easy to split a basic block when you have a branch going to that basic block, but if you just synthetically insert a split some distance down, you may be misaligned with the actual instruction stream. That can be self-synchronizing at points, but it's hard.
I do agree. The fact that those uop queues are already there and Intel isn't using them as a loop buffer does make me ask questions. Have they just not gotten around to it? Have they decided its not worth the power savings? Maybe they are aiming for simplicity?
> However, a large amount of comparative wall time in code is spent on sequences of a short piece of code and a branch that is hard to predict.
The thing is, any time the branch predictor has at least one correct prediction, the decode throughput doubles to 6 IPC. And if it gets two correct predictions in a row, the IPC triples to 9.
I'm not sure how many cycles the "execute both sides of the branch" would save on a misspredict, but your basic blocks would need to be very short and the prediction accuracy would need to be very low (like, 50% or lower) before it can actually out-preform the leapfrogging decoder approach on those sequences of code.
> but if you just synthetically insert a split some distance down, you may be misaligned with the actual instruction stream
It only inserts the splits after decoding, so they will always be at the correct alignment.
Branches that are hard to predict should be “hand-optimized” through specially written code, compiler intrinsics/annotations, or profile guided feedback to tell the compiler to emit the conditional using unconditional branchless instructions like cmov. Expecting a CPU to detect this at runtime may be asking it to do too much.
That necessitates executing both sides fully. Very often, those are "business logic" branches that are very long, and you would only prefer to cover branch mispredict penalty.
Yes you would only do this for hot loops that have short basic block branches (binary search being the canonical example). That’s why I said hand annotate vs having the CPU try to detect and distinguish these situations at runtime.
Yes, we're talking about different things. Those are able to be optimized in software doing what you mentioned. The branches I am talking about are not. They often sit in "business logic" code.
I fancy myself of having a good understanding of modern uarch. But i have to agree with @Marthinwurer. This branch predictor structure with parallel predictor and fake branch address is quite wild.
Do you know how this compare to what AMD/AppleM/Qualcom is doing ? This seems super effective, but seems pretty power hungry as opposed to just increasing the chase size and predictor precision. Plus i would assume it makes the cost of miss-predict even higher.
I'm pretty sure the patten of allowing the branch predictor to run ahead is pretty common.
At least, it's common to have multi-level branch predictors that take a variable number of cycles to return a result, and it makes a lot of sense to queue up predictions so they are ready when the decoder gets to that point.
But I doubt the idea of parallel decoders makes any sense out side of x86's complex variable length instructions.
It (probably) makes sense on x86 because x86 cores were already spending a bunch of power on instruction decoding and the uop cache.
> Plus i would assume it makes the cost of miss-predict even higher.
It shouldn't increase the miss-predict cost by too much.
The new fetch address will bypass the branch-prediction queue and feed directly into one of the three decoders. And previous implementations already have a uop queue between the decoder and re-name/dispatch. It gets flushed and the first three uops should be able to cross it in a single cycle.
Thanks, that's a nice explanation. I hadn't looked in details of how the multiple decoders in the *monts worked. Relying on branches and prediction to find the instruction boundaries is quite a nifty trick.
If the branch predictor to predict branches ahead it needs to know where the branches instructions are. Is there a mini decoder tasked to just decode the instruction stream just enough to handle the variable length instructions and figure out where the branches are? Or am I fundamentally misunderstanding how branch prediction works (which likely I am)?
There seems to be a very common misconception about branch prediction, that its only job is to predict the direction of the branch.
In reality, the problem is so much deeper. The instruction fetch stage simply can't see the branch at all. Not just conditional branches, but unconditional jumps, calls and even returns too.
Even a simple 5 stage "classic RISC" pipeline takes a full two cycles to load the instruction from memory and decode before it can see it, and your instruction fetch stage has already fetched two incorrect instructions (though many RISC implementations cheat with an instruction cache fetch that takes half a cycle, and then adding a delay slot).
In one of these massive out-of-order CPUs, the icache fetch might take multiple cycles, (then length decoding on x86), so it might take 4 or 5 cycles before the instruction could possibly be decoded. And if you are decoding 4 instructions per cycle, that's 20 incorrect instructions fetched from icache.
To actually continue fetching without any gaps, the branch predictors needs to predict:
1. The location of the branch
2. The type of branch, and (for conditional branches) if it's taken or not.
follow up question: if the branch is predicted to not be taken, why does the predictor have to use resources to record its location and the destination?
These predictors change their prediction (both direction and destination) based on the history of the last few hundred branches and if they were taken or not-taken. So the predictor needs to know where those branches were, even if they aren't taken.
Indirect TAGE predictors are very powerful. They can correctly predict jump tables and virtual function calls.
In general, branch predictors don't utilise their tables very efficiently. Cheap and fast lookups are way more important than minimising size.
First, there is no uop cache on the "mont" cores.
Second, Intel aren't decoding both sides of the branch.
That wouldn't actually help much, as modern branch predictors are correct well over 99% of the time. It would be a waste of silicon and power to have an extra decoder producing work which simply decoded most of the time, and an even bigger waste to have two extra decoders.
Intel's actual approach is way more clever; They run the branch predictor ahead of the decoders by at least 3 branches (probably more). The branch predictor can spit out a new prediction every cycle, and it just plops them on a queue.
Each of the three decoders pops a branch prediction off the queue and starts decoding there. At any time, all three decoders will each be decoding a different basic block. A basic block that the branch predictor has predicted that the program counter is about to flow through. The three decoders are leap frogging each other. The decoding of each basic block is limited to a throughput of three instructions per cycle, but Skymont is decoding three basic blocks in parallel.
The decoded uops get pushed onto three independent queues, and the re-namer/dispatcher merges these three queues back together in original program order before dispatching to the backend. Each decoder can only push three uops per cycle onto its queue, but the re-namer/dispatcher can pull them off a single queue at the rate of 9 uops per cycle. The other two queues will continue to fill up while one queue is being drained.
The branch prediction result will always land on an instruction boundary, so this design allows the three decoders to combine their efforts and maintain a throughput of 9 uops per cycle, as long as the code is branchy enough. It works on loops too, as far as I'm aware, intel doesn't even have a loop stream buffer on this design; The three decoders will be decoding the exact same instructions in parallel for loop bodies.
But Intel have a neat trick to make this work even on code without branches or loops. The branch predictor actually inserts fake branches into the middle of long basic blocks. The branch predictor isn't actually checking an address to see if it has a branch. Instead it predicts the gap between branches, and they simply have a limit for the size of those gaps. Looks like that limit for Skymont is 64 bytes (was previously 32 bytes for Crestmont)