How do I programmatically disable hardware prefetching?

Question

I would like to programmatically disable hardware prefetching.

From Optimizing Application Performance on Intel® Core™ Microarchitecture Using Hardware-Implemented Prefetchers and How to Choose between Hardware and Software Prefetch on 32-Bit Intel® Architecture, I need to update the MSR to disable hardware prefetching.

Here is a relevant snippet:

"DPL Prefetch and L2 Streaming Prefetch settings can also be changed programmatically by writing a device driver utility for changing the bits in the IA32_MISC_ENABLE register – MSR 0x1A0. Such a utility offers the ability to enable or disable prefetch mechanisms without requiring any server downtime.

The table below shows the bits in the IA32_MISC_ENABLE MSR that have to be changed in order to control the DPL and L2 Streaming Prefetch:

Prefetcher Type MSR (0x1A0) Bit Value 
DPL (Hardware Prefetch) Bit 9 0 = Enable 1 = Disable 
L2 Streamer (Adjacent Cache Line Prefetch) Bit 19 0 = Enable 1 = Disable"

I tried using http://etallen.com/msr.html but this did not work. I also tried using wrmsr in asm/msr.h directly but that segfaults. I tried doing this in a kernel module ... and killed the machine.

BTW - I am using kernel 2.6.18-92.el5 and it has MSR linked in the kernel:

$ grep -i msr /boot/config-$(uname -r)
CONFIG_X86_MSR=y
...

Answer 1

From the Intel reference:
This instruction must be executed at privilege level 0 or in real-address mode; otherwise, a general protection exception #GP(0) will be generated. Specifying a reserved or unimplemented MSR address in ECX will also cause a general protection exception.

...
The CPUID instruction should be used to determine whether MSRs are supported (EDX[5]=1) before using this instruction.

So, your fault might be related to a cpu that doesn't support MSRs or using the wrong MSR address.

There are lots of examples of using the MSRs in the kernel source:

In the kernel source, for a single cpu, it demonstrates disabling prefetch for the Xeon in arch/i386/kernel/cpu/intel.c, in the function:

static void __cpuinit Intel_errata_workarounds(struct cpuinfo_x86 *c)

The rdmsr function arguments are the msr number, a pointer to the low 32 bit word, and a pointer to the high 32 bit word.
The wrmsr function arguments are the msr number, the low 32 bit word value, and the high 32 bit word value.

multi-core or smp systems have to pass the cpu struct in as the first argument:
void rdmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 *l, u32 *h);
void wrmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 l, u32 h);

Answer 2

You can enable or disable the hardware prefetchers using msr-tools http://www.kernel.org/pub/linux/utils/cpu/msr-tools/.

The following enables the hardware prefetcher (by unsetting bit 9):

[root@... msr-tools-1.2]# ./wrmsr -p 0 0x1a0 0x60628e2089 
[root@... msr-tools-1.2]# ./rdmsr 0x1a0 
60628e2089

The following disables the hardware prefetcher (by enabling bit 9):

[root@... msr-tools-1.2]# ./wrmsr -p 0 0x1a0 0x60628e2289 
[root@... msr-tools-1.2]# ./rdmsr 0x1a0 
60628e2289

Programatically, you can do this as root by opening /dev/cpu/<cpunumber>/msr and using pwrite to write to the msr "file" at the 0x1a0 offset.

Answer 3

In 2014 Intel published info about h/w prefetcher disabling with 0x1a4 msr (1a4 msr) for Nehalem, Westmere, Sandy Bridge, Ivy Bridge, Haswell, Broadwell (and probably newer cores). Link was found by bholanath here:

https://software.intel.com/en-us/articles/disclosure-of-hw-prefetcher-control-on-some-intel-processors Disclosure of H/W prefetcher control on some Intel processors - Vish Viswanathan (Intel), September 24, 2014

This article discloses the MSR setting that can be used to control the various h/w prefetchers that are available on Intel processors based on the following microarchitectures: Nehalem, Westmere, Sandy Bridge, Ivy Bridge, Haswell, and Broadwell.

The above mentioned processors support 4 types of h/w prefetchers for prefetching data. There are 2 prefetchers associated with L1-data cache (also known as DCU DCU prefetcher, DCU IP prefetcher) and 2 prefetchers associated with L2 cache (L2 hardware prefetcher, L2 adjacent cache line prefetcher).

There is a Model Specific Register (MSR) on every core with address of 0x1A4 that can be used to control these 4 prefetchers. Bits 0-3 in this register can be used to either enable or disable these prefetchers. Other bits of this MSR are reserved.

They are local to every CPU core and can be changed by root with help of msr linux kernel driver. They are used by Intel to measure memory latency in NUMA with Intel MLC tool:

For example, Intel Memory Latency Checker tool (http://www.intel.com/software/mlc) modifies the prefetchers through writes to MSR 0x1a4 to measure accurate latencies and restores them to the original state on exit.

Answer 4

I am adding an answer here, because the previous ones may not be applicable to all Intel processors.

For my Intel Xeon 5650 (06_2CH family) processor the manual chapter 35 specifies that bits 10 to 8 of the register IA32_MISC_ENABLE at adress 0x1A0 are reserved. I guess that this means I can't toggle prefetcher on and off trhough MSR.

According to an answer from an Intel employee here: "Intel has not disclosed how to disable the prefetchers on processors from Nehalem onward.You'll need to disable the prefetchers using options in the BIOS."