Part 4 · Where the fictions break
Latency versus throughput
The distinction that makes instruction counting useless. One number tells you how long an operation takes; the other tells you how many you can start. They are not the same and rarely close.
What this lesson is trying to break (3)
- An instruction has "a cost" — one number.
- A 3-cycle instruction always takes 3 cycles of your time.
- Slower instructions should always be avoided.
This is a threshold concept: troublesome, transformative, and irreversible. You should expect to be genuinely stuck here, and being stuck is the mechanism rather than a sign that you are failing. Budget several sessions. Do not move on because you are bored — move on when you can explain it out loud without notes.
This is the lesson that makes “how many instructions?” stop being a meaningful question.
Two numbers, not one
Every instruction has at least two costs.
Latency — cycles until the result is usable by a dependent instruction.
Throughput — how often you can start a new, independent one. Usually quoted as reciprocal throughput: cycles between issues, so 0.5 means two per cycle.
The reason both exist is pipelining. A multiplier might take three stages, so any single multiply takes three cycles end to end. But the stages are independent hardware, so a new multiply can enter stage 1 while the previous is in stage 2. Three-cycle latency, one-per-cycle throughput.
Varies: which of the two numbers dominates
| Situation | What limits you | Effective cost of a 3-cycle multiply |
|---|---|---|
a = a * b in a loop (dependent) | Latency | 3 cycles each — nothing overlaps |
x[i] = y[i] * z[i] (independent) | Throughput | ~1 cycle each — three in flight at once |
Same instruction, same chip, three times the cost depending only on whether the results feed each other. This is why a single 'cost' number for an instruction is meaningless.
So the answer to “how expensive is a multiply?” is it depends on what surrounds it, and that is not evasion — it is the actual answer.
Both compute a product of 1024 numbers. Assume no vectorisation, to isolate the effect.
// A
long a(long *p) { long r = 1; for (int i=0;i<1024;i++) r *= p[i]; return r; }
// B
long b(long *p) {
long r0=1, r1=1, r2=1, r3=1;
for (int i=0;i<1024;i+=4){ r0*=p[i]; r1*=p[i+1]; r2*=p[i+2]; r3*=p[i+3]; }
return r0*r1*r2*r3;
}B, by roughly the latency of a multiply — around 3×.
In A each multiply waits for the last: 1024 multiplies × latency. The multiplier sits idle two thirds of the time, because it could accept a new operation every cycle but nothing independent is available.
B keeps four chains going, so three or four multiplies are in flight simultaneously. Same instruction count, same arithmetic, roughly a third of the time.
The general rule, and it is worth memorising: to saturate a pipelined unit, you need at least as many independent chains as the latency in cycles. A 3-cycle, 1-per-cycle multiplier needs 3–4 chains. Fewer, and you are latency-bound. This is exactly what compilers do when they unroll-and-split accumulators, and it is why they unroll by 4 rather than 2.
Integer multiply is associative, so a compiler may do this for you. Floating
point is not, so for double it will refuse without -ffast-math
(3.4) — which is the case where you may
genuinely need to restructure by hand.
Rough shapes worth carrying
Specific cycle counts are microarchitecture-specific and go stale, so learn the
shape and look numbers up in uops.info when they matter.
| Operation | Latency | Throughput | Notes |
|---|---|---|---|
| add, sub, and, or, xor | ~1 | several/cycle | effectively free when independent |
lea |
~1 | high | why it is used so freely |
| integer multiply | ~3 | ~1/cycle | cheap unless in a chain |
| integer divide | ~20–40 | poor | bad on both axes |
| L1 load | ~4–5 | ~2/cycle | see 4.4 |
| FP add/mul | ~4 | ~2/cycle | pipelined |
| FP divide / sqrt | ~10–20 | poor | like integer divide |
| predicted branch | ~0 | ~1/cycle | genuinely almost free |
| mispredicted branch | ~15–20 | — | a pipeline flush |
Two entries deserve attention.
Division is the only common instruction that is bad on both axes — high
latency and not pipelined, so a second division cannot overlap the first. That
is why 3.3’s magic-multiply trick
exists, and why the compiler will spend five instructions to avoid one div.
A correctly predicted branch costs approximately nothing. People are surprised by this and it changes how you think about branchless code (4.3).
Ports: the third constraint
Even with no dependencies, you can be limited by which units exist. Execution units sit behind ports, and each port handles particular kinds of work. If a core has two load ports and your loop needs three loads per iteration, it takes at least 1.5 cycles per iteration regardless of everything else.
That situation is called port pressure, and “there’s port pressure on the load
units” is a normal sentence in this field. llvm-mca reports it directly, which
is the fastest way to discover you are fighting the wrong constraint.
A colleague proposes replacing a multiply with two shifts and an add, “because shifts are cheaper than multiplies”.
Give the two situations in which they are right and wrong respectively, and say what you would need to know about the surrounding code to decide. Then say what you would actually do first.
What you should now be able to say
- Every instruction has latency and throughput, and they can differ by an order of magnitude.
- Latency binds inside a dependency chain; throughput binds across independent work.
- To saturate a pipelined unit you need as many independent chains as its latency.
- Division is bad on both axes — hence the magic-multiply trick.
- A predicted branch is nearly free.
- Port pressure is a third limit, independent of dependencies, and
llvm-mcareports it.
Next: 4.3 — Branch prediction.