Why is PCD bit set when I don't use ioremap_cache?

Question

I am using ubuntu 12.10 32 bit on an x86 system. I have physical memory(about 32MB,sometimes more) which is enumerated and reserved through the ACPI tables as a device so that linux/OS cannot use it. I have a Linux driver for this memory device. THe driver implements mmap() so that when a process calls mmap(), the driver can map this reserved physical memory to user space. I also sometimes do nothing in the mmap except setup the VMA's and point vma->vmops to the vm_operations_struct with the open close and fault functions implemented. When the application accesses the mmapped memory, I get a page fault and my .fault function is called. Here is use vm_insert_pfn to map the virtual address to any physical address in the 32MB that I want.

Here is the problem I have: In the driver, if I call ioremap_cache() during init, I get good cache performance from the application when I access data in this memory. However, if I don't call ioremap_cache(), I see that any access to these physical pages results in a cache miss and gives horrible performance. I looked into the PTE's and see that the PCD bit for these virtual address->physical translation are set, which means caching on these physical pages is disabled. We tried setting _PAGE_CACHE_WB in the vma_page_prot field and also used remap_pfn_range with the new vma_page_prot but PCD bit was still set in the PTE's.

Does anybody have any idea on how we can ensure caching is enabled for this memory? The reason I don't want to use ioremap_cache() for 32 MB is because there are limited Kernel Virtual Address on 32bit systems and I don't want to hold them.

Answer 1

Suggestions:

1. Read linux/Documentation/x86/pat.txt
2. Boot Linux with debugpat
3. After trying the set_memory_wb() APIs, check /sys/kernel/debug/x86/pat_memtype_list