Agner`s CPU blog

Each core is lightweight, based on an extension of the Silvermont low power architecture. Each core runs slower than a desktop CPU, but with a large number of cores we can still get a high overall performance. It has 32 kB of level-1 code cache and 32 kB level-1 data cache per core; 1 MB of level-2 cache shared between two cores each; and 16 GB of MCDRAM inside the package. The MCDRAM can be configured as a level-3 cache or as main memory.

The predecessor, Knights Corner, was not very impressive and it had its own instruction set. The Knights Landing is the first processor with the new AVX512 instruction set. It is expected that AVX512 will be the standard for future x86 processors so that the Knights Landing will be binary compatible with mainstream microprocessors. It also supports the previous instructions sets AVX2, etc.

The AVX512 instruction set seems to be quite efficient. It has 32 vector registers of 512 bits each, where AVX2 has only 16 registers of 256 bits each. It also has a new set of eight mask registers that can be used for conditional execution of each element of a vector. Almost all vector instructions can be masked. This works quite efficiently. The latencies and throughputs of vector instructions are the same with or without a mask, and independent of the value of the mask register. The Gnu compiler optimizes this quite well, so that for example an addition and an if can be merged together into a single add instruction with a mask.

On the positive side, the Knights Landing has true out-of-order processing (unlike Knighs Corner and Silvermont). It has a good memory throughput. It can do two 512-bit vector reads, or one read and one write, per clock cycle. The throughput for simple vector instructions is two 512-bit vectors per clock.

The Knights Landing has an instruction set extension, AVX512ER, with some quite impressive math instructions. It can calculate a reciprocal, a reciprocal square root, and even an exponential function, on a vector of 16 floats in just a few clock cycles. The manual has a somewhat confusing description of the accuracy of these instructions. My measurements showed that these instructions are accurate to the last bit on single precision floats, while they give only approximate results for double precision. These instructions are useful for neural networks and other large low-precision math applications.

On the negative side, all vector instructions have a latency of at least 2 clock cycles, where earlier processors have a latency of 1 for simple vector instructions. Integer instructions on general purpose registers have a latency of 1. A possible explanation for this difference is that the integer reservation station can hold source data, while the floating point reservation station cannot. This means that an integer ALU can write its result directly to any subsequent micro-op in the reservation station that needs it, while results in the floating point unit have to go via the floating point register file. The size of the vector operands is simply to large to make it practical to store the values in the reservation station.

Almost all instructions that generate more than one micro-op are microcoded. The performance of microcode is not good. All microcoded instructions take 7 clock cycles or more. This includes most of the legacy x87 floating point instructions. You should avoid legacy x87 code. Floating point division is also relatively slow (32 clock cycles for a vector division).

The instruction decoder is likely to be a bottleneck. It can decode a maximum of two instructions or 16 bytes of code per clock cycle.

When AVX was introduced with 256-bit vector registers, we were told to use the instruction VZEROUPPER to avoid a severe penalty when switching between VEX and non-VEX code. Four generations of Intel processors had such a penalty (Sandy Bridge, Ivy Bridge, Haswell, and Broadwell). AMD processors and later Intel processors (Skylake and Knights Landing) do not have such a state switch. They have no need for the VZEROUPPER. Unfortunately, the VZEROUPPER is quite costly on Knights Landing. The recommendations from Intel are conflicting here. The Intel optimization manual recommends VZEROUPPER when switching between AVX and SSE code, but elsewhere in the same manual the say that you should not use VZEROUPPER on Knights Landing. This conflict is currently not resolved (see my discussion in Intels developer zone).

I am somewhat sceptical about the extensive use of hyperthreading - Intel's word for running multiple threads in the same core. What is the point of running four threads in a CPU core with a limited bandwidth of two instructions per clock cycle? This wouldn't be useful for CPU intensive code, but perhaps for code that is limited by memory access, branch mispredictions, or long dependency chains. Hyperthreading has a hazard that is often ignored. If four threads are running in the same core then each thread gets only a quarter of the CPU resources. I have seen a high-priority thread running at quarter speed because three other low priority threads were running in the same core. This is certainly not an optimal use of resources, and current operating systems are unable to avoid this problem. There is little you can do in a multi-user or multi-process system to prevent low priority threads from stealing resources from high priority threads. It may actually be better to turn off hyperthreading completely in the BIOS setup. There is also a security issue here: One thread will be able to detect what kind of code is running in another thread in the same core by detecting which CPU resources are fully used and which ones are unused.

My optimization manuals have been updated with test results and instruction timings for the Knights Landing and some more general information about AVX512 (link).

My assembly function library has been updated with memcpy, memmove, memset, and memcmp functions optimized for AVX512 (link).

My vector class library has been updated with improved support for AVX512 (link).

> The Knights Landing has full out-of-order capabilities

It may be worth mentioning that the Memory Execution Cluster is still in order. The Intel Architectures Optimization Reference Manual says "The MEC has limited capability in executing uops out-of-order. Specifically, memory uops are dispatched from the scheduler in-order, but can complete in any order. " More specifically, they mean "in order with regard to other memory operations": unlike other modern Xeons, a memory operation with an unfulfilled dependency will block successive memory operations from being dispatched even if their dependencies are ready.

> The reservation stations have 2x12 entries for the integer lines, 2x20 entries
> for the floating point and vector lines, and 12 for the memory lines.

The manual says "The single MEC reservation station has 12 entries, and dispatches up to 2 uops per cycle." I think this means that both 'memory lines' share a single 12 slot queue this is used for both scalar and vector operations? If so, might be good to be more explicit.

> The Knights Landing has two decoders. The maximum throughput is two
> instructions or 16 bytes per clock cycle.

The manual says "The front end can fetch 16 bytes of instructions per cycle. The decoders can decode up to two instructions of not more than 24 bytes in a cycle. ", and then later says "The total length of the instruction bytes that can be decoded each cycle is at most 16 bytes per cycle with instructions not more than 8 bytes in length. For instruction length exceeding 8 bytes, only one instruction per cycle is decoded on decoder 0." I haven't figured out when the 24B limit would apply. Have you? Also, is it correct that these limits are not affected by alignment?

> The throughput is limited to two instructions per clock cycle
> in the decode and register rename stages.
> the average throughput is limited to two μops per clock cycle.
> Read-modify, and read-modify-write instructions generate a single μop
> from the decoders, which is sent to both the memory unit and the execution unit.

By emphasizing that it's a 'single µop', do you mean that the same µop is first sent to the memory unit and then, when the data is ready, (Figure 16-2) sent to the Integer or FP rename buffer as appropriate? And thus since the renamer can handle only two instructions per cycle, this means that unlike other Xeon the use of 'read-modify' (aka 'load-op') instructions does not usually help to increase µop throughput?

> There is no penalty for the 2- and 3-byte VEX prefixes and 4-byte EVEX prefixes
> unless there are additional prefixes before these.

It's fairly easy to check that all the instructions are encoded to less than 8B, but I'm not sure I know how to correctly count the prefixes. Do you have any examples of common cases where this would be a problem?

> The Knights Landing has no loop buffer, unlike the Silvermont. This means
> that the decoding of instructions is a very likely bottleneck, even for small loops.

The corollary to this is that loop unrolling can be a win on KNL even when it would be counterproductive on a Xeon with a loop buffer.

> These are forked after the register allocate and renaming stages into two the integer unit

I think there is a missing word after 'two'?

> A 64-bit register can be cleared by xor'ing the corresponding 32-bit register with itself.

Hmm, interesting find. Do you see any reason they would choose to recognize the 32-bit idiom but not the 64-bit? Or just oversight?

> The processor can do two memory reads per clock cycle or one read and
> one write with vector registers of up to 512 bits. It cannot do two reads with
> general purpose registers in the same clock cycle, but it can do one
> read and one write.

I can't tell from this (or the Intel manual) which "vector plus scalar" memory operations are possible in the same cycle. Do you know if it can read a vector and a scalar in the same cycle? Read one and write the other?

> The latency from the mask register, k1, to the destination is 2 clock cycles.
> The latency from zmm1 input to zmm1 output is 2 clock cycles.

I'm not understanding what you are measuring here. Are you saying that if have an instruction that changes the mask or the input, immediately followed by a masked read, that you will have an effective load latency of 2 + 5? Or something else?

> The {z} option makes the elements in the destination zero, rather than
> unchanged, when the corresponding mask bit is zero. It is recommended
> to use the zeroing option when possible to avoid the dependence on the previous value.

Does this dependency occur only when explicitly using a mask, or does it matter in other cases too? That is, does {z} make a difference without a mask?

> The best solution in such cases may be to turn off hyper-threading in the BIOS setup.

Do you think there is a difference between turning off hyper-threading in BIOS, rather than telling the OS to disable someo f the cores at runtime? The manual says "Hard partitioning of resources changes as logical processors wake up and go to sleep", which makes me think that OS level would be fine, although I'm not sure exactly what it means by 'sleep'.

> 15.11 Bottlenecks in Knights Landing

As shown in your Instruction Table, the extreme cost of some of the "byte oriented" vector instructions might be worth calling out. The ones that stood out for me were VPSHUFB y,y,y with latency 23 and reciprocal throughput 12 (versus 1 and 1 for Skylake) and PMOVMSKB r32,y with latency 26 and reciprocal throughput 12 (versus 2-3 and 1 for Skylake). Specific mention of some of these might be helpful, since while these are still technically supported, it's unlikely that you'd want to use an algorithm that depends on them.

There are a couple passages in the manual that I haven't been able to decipher. Do you perhaps understand this one, which (among other things) pertains to shuffles and permutes? "Some execution units in the VPU may incur scheduling delay if a sequence of dependent uop flow needs to use these execution units, these outlier units are indicated by the footnote of Table 16 2. When this happens, it will have an additional cost of a 2-cycle bubble."

The other line that scares me in the manual is this one: "The decoder will also have a small delay if a taken branch is encountered." Did you happen to figure out how long this delay is?

Thanks again for making this great research available! I'm sure a lot of people will greatly appreciate it.

As for why the chip has 4 threads per core - I'm the guy that persuaded KNF (and thus KNC) to have 4 threads, and the reason is first to hide memory misses, second branch mispredicts, and third instruction latencies. Those are all huge bottlenecks in real-world performance. Yes, you can also hide them with huge OOO machines, wide decoders, and long pipelines, but when flops/watt is your efficiency metric, those aren't the first choice. The Knights line of chips already open the book with "we assume you have 70+ threads of stuff to do", and while getting from 1 thread to 4 is agony, and 4 threads to 16 is hard, getting from 70 threads to 280 is actually pretty simple.

Looks like a typo in the KNL recip-throughput number for FMA - it's currently 3. KNF and KNC get 1, and this chip is a real FMA machine - it's designed around that unit. Pretty sure the correct number for KNL is 0.5 (like VADDPS and VMULPS).

As for why the chip has 4 threads per core - I'm the guy that persuaded KNF (and thus KNC) to have 4 threads, and the reason is first to hide memory misses, second branch mispredicts, and third instruction latencies. Those are all huge bottlenecks in real-world performance. Yes, you can also hide them with huge OOO machines, wide decoders, and long pipelines, but when flops/watt is your efficiency metric, those aren't the first choice.

Nathan Kurz wrote:

The Knights Landing has full out-of-order capabilities

It may be worth mentioning that the Memory Execution Cluster is still in order. The Intel Architectures Optimization Reference Manual says "The MEC has limited capability in executing uops out-of-order. Specifically, memory uops are dispatched from the scheduler in-order, but can complete in any order. " More specifically, they mean "in order with regard to other memory operations":

The manual says "The front end can fetch 16 bytes of instructions per cycle. The decoders can decode up to two instructions of not more than 24 bytes in a cycle. ", and then later says "The total length of the instruction bytes that can be decoded each cycle is at most 16 bytes per cycle with instructions not more than 8 bytes in length. For instruction length exceeding 8 bytes, only one instruction per cycle is decoded on decoder 0." I haven't figured out when the 24B limit would apply. Have you? Also, is it correct that these limits are not affected by alignment?

Read-modify, and read-modify-write instructions generate a single μop from the decoders, which is sent to both the memory unit and the execution unit.

By emphasizing that it's a 'single µop', do you mean that the same µop is first sent to the memory unit and then, when the data is ready, sent to the Integer or FP rename buffer as appropriate?

There is no penalty for the 2- and 3-byte VEX prefixes and 4-byte EVEX prefixes unless there are additional prefixes before these.

It's fairly easy to check that all the instructions are encoded to less than 8B, but I'm not sure I know how to correctly count the prefixes. Do you have any examples of common cases where this would be a problem?

A 64-bit register can be cleared by xor'ing the corresponding 32-bit register with itself.

Hmm, interesting find. Do you see any reason they would choose to recognize the 32-bit idiom but not the 64-bit? Or just oversight?

I can't tell from this (or the Intel manual) which "vector plus scalar" memory operations are possible in the same cycle. Do you know if it can read a vector and a scalar in the same cycle?

The latency from the mask register, k1, to the destination is 2 clock cycles. The latency from zmm1 input to zmm1 output is 2 clock cycles. I'm not understanding what you are measuring here. Are you saying that if have an instruction that changes the mask or the input, immediately followed by a masked read, that you will have an effective load latency of 2 + 5? Or something else?

The {z} option makes the elements in the destination zero, rather than unchanged, when the corresponding mask bit is zero. It is recommended to use the zeroing option when possible to avoid the dependence on the previous value. Does this dependency occur only when explicitly using a mask, or does it matter in other cases too? That is, does {z} make a difference without a mask?

The best solution in such cases may be to turn off hyper-threading in the BIOS setup. Do you think there is a difference between turning off hyper-threading in BIOS, rather than telling the OS to disable some of the cores at runtime?

Do you perhaps understand this one, which (among other things) pertains to shuffles and permutes? "Some execution units in the VPU may incur scheduling delay if a sequence of dependent uop flow needs to use these execution units, these outlier units are indicated by the footnote of Table 16 2. When this happens, it will have an additional cost of a 2-cycle bubble"

The decoder will also have a small delay if a taken branch is encountered. Did you happen to figure out how long this delay is?

what's the latency of the MCDRAM when used as main memory?

It is also important to remember that the latency can be different for each physical address, depending on the location of the requesting core, the location of the coherence agent responsible for that address, and the location of the memory controller for that address. Intel has not publicly disclosed the mapping of core numbers (APIC IDs) to physical locations on the chip or the locations of coherence agents (CHA boxes) on the chip, nor has it disclosed the hash functions used to map physical addresses to coherence agents and to map physical addresses to MCDRAM or DDR4 memory controllers. (In some modes of operation the memory mappings are trivial, but not in all modes.)

The Knights Landing system at TACC uses the Xeon Phi 7250 processor (68 cores, 1.4 GHz nominal). For operation in "Flat" mode (MCDRAM as memory, located in the upper 16 GiB of the physical address space), with the coherence agent mapping in "Quadrant" mode (addresses are hashed to coherence agents spread across the entire chip, but each cache line is assigned to an MCDRAM controller in the same "quadrant" as the CHA responsible for coherence), my preferred latency tester gives values of 154ns +/- 1ns (1 standard deviation) for MCDRAM. These values are averaged over many addresses, with the variation mostly from core to core (with a few ns of random variability). My latency tester uses permutations of even-numbered cache lines in various sized address range blocks, so it is not guaranteed that my averages are uniformly distributed over all the coherence agents.

For the same system in "Flat" "All-to-All" mode (addresses are hashed to coherence agents spread across the entire chip, with no special correlation between the location of coherence agents and the MCDRAM controller owning an address), the corresponding value is 156ns +/- 1ns (1 standard deviation).

For the same system in "Flat" "Sub-NUMA Cluster 4" mode, the corresponding values are 150.5ns +/- 0.9ns (1 standard deviation) for "local" accesses, and 156.8ns +/- 3.1ns for "remote" accesses. Variability across nodes is not entirely negligible, in part because different nodes have different patterns of disabled tiles. (Four of the 38 tiles are disabled on each Xeon Phi 7250 processor.) Run-to-run variability is typically small when using large pages, but there are certain idiosyncrasies that have yet to be explained.

Note that even though the average latency differences are quite small across these modes of operation, the sustained bandwidth differences are much larger. The decreased number of "hops" required for coherence transactions in "Quadrant" and "SNC-4" modes reduces contention on the mesh links and thereby allows higher sustained bandwidths. The difference between sustained bandwidth in Flat-All-to-All and Flat-Quadrant modes suggests that contention on the non-data mesh links (address, acknowledge, and invalidate) is more important than contention on the data transfer links (which should be the same for those two modes of operation). I will post more details to my blog as they become available....

Without knowledge of the tag directory hash, the processor placement, the MCDRAM hash, etc, it is challenging to make a lot of sense of the results. On KNC the RDTSC instruction had about a 5 cycle latency, so I was able to do a lot more with timing individual loads, and the single-ring topology made the analysis easier.

There are more performance counters in the "untile" on KNL, but there is no documentation on the various box numbers are located on the mesh. There is some evidence that the CHA box numbers are re-mapped -- on a 32-tile/64-core Xeon Phi 7210 all 38 CHAs are active, but the six CHA's with anomalous readings are numbered 32-37. The missing core APIC IDs are not bunched up in this way.

The stacked memory modules have slightly higher latency because they are typically run in "closed page" mode, and because there is an extra set of chip-to-chip crossings. HMC (and Intel's MCDRAM) have an extra SERDES step between the memory stack and the processor chip. There are many different approaches used to error-checking on SERDES, but it is probably safe to expect that error-checking will require at least some added latency.

From this discussion

stackoverflow.com/questions/41303780/why-is-this-sse-code-6-times-slower-without-vzeroupper-on-skylake

it appears that VZEROUPPER is even more important in some situations with Skylake then e.g. with Haswell.

it appears that VZEROUPPER is even more important in some situations with Skylake then e.g. with Haswell.

I just made some experiments on a Haswell. The dirty state makes every non-VEX vector operation dependent on the previous value of the destination register. For example

vpor ymm1,ymm2,ymm3
...
pxor xmm4,xmm4 ; Has false dependence on ymm4


I have no access to a Skylake at the moment, but it may behave the same. This means that VZEROUPPER is still needed on all Intel processors, except Knights Landing.

I have one question regarding the permutations on KNL. As you can see in your instruction manual and corroborated in "Intel Xeon Phi Processor High Performance Programming", there is around 2X difference in throughput and 33% in latency when using a two source or one source permutation/shuffle with the one source variant being superior. I am interested in permutations on double precision operands to be more precise.

Scanning through your instruction table, I see that you have added vpermpd as only working on ymm registers which presumably is a one source permutation. However, in the intrinsics guide, I can see that you can generate a vpermpd using zmm's with _mm512_mask_permutex_pd where you can use a mask to keep results from the source argument (which I think gets overwritten) and shuffles across 256bit lane in the other operand. This decays to vpermpd zmm {k}, zmm, imm. As far as I can tell, this is a one source operand permutation which will have a latency between 3-6 cycles and a rec throughput of 1, am I correct? From my understanding, the two source operands are the intrinsics that decay into vpermpd zmm {k}, zmm, zmm such as _mm512_maskz_permutexvar_pd or the VSHUFF64X2 instructions.

Your clarification between these would be very helpful.

Kind regards,
Ioan

InstLatx64 for Silvermont says 1 per clock INC r32 throughput, but 0.5c ADD r32, imm8. Same for Goldmont (except that ADD r32,imm8 is 0.33c).

I haven't found InstLatx64 results for actual KNL, but Intel's optimization manual describes KNL as having the same flag-merging extra uop for INC/DEC as Silvermont, so I expect that INC is worth avoiding on KNL as well when you're bottlenecked on issue throughput instead of decode. (Is there any IDQ between decode and issue to let the expansion from 1 uop decoded / 2 uops issued fill decode bubbles?) If so, it could still have an apparent cost of 0.5c in the right circumstances.

According to Intel's optimization manual, 17.1.2 Out-of-Order Engine (KNL):

Additionally, some instructions in the Knights Landing microarchitecture will be decoded as one uop by
the front end but need to expand to two operations for execution. These complex uops will have an allocation
throughput of one per cycle. Examples of these instructions are:

• POP: integer load data + ESP update, PUSH: integer store data + ESP update.
  
• INC/DEC: add to register + update partial flags
  
• Gather: two VPU uops
  
• CALL / RET: JMP + ESP update
  
• LEA with 3 sources
  

(lightly edited the list to group related instructions better.)

This means there's *always* a flag-merging uop, even when nothing reads INC's flags. And it explains why the sustained throughput of INC/DEC is only 1, not 0.5, according to measurements other than yours, and according to Intel's published tables. It's nice that the integer register update itself doesn't have a false dep on flags, though, so they made it a lot less bad than P4.

Everything except your table says 1c throughput for INC/DEC, not 0.5c, for both KNL and Silvermont