Short answer: Because Linus decided to do it that way.
Long answer: Actually, it think it's wrong that you should be able to access even that area without a driver signing up for it. But never mind that.
In the old days, sometime in the 1990's, PCI didn't exist, and graphics cards, network cards, etc, were using something called ISA bus. It didn't have fancy features that allow you to request information from the card, or configure things onto the card with regards to where it thinks it's hardware addresses should be. All of the ISA memory is between A0000 (640K) and FFFFF (1MB-1). So, in the early days of Linux, this was where graphics and such things lived, and the kernel couldn't really know where these things were. Either way, we don't have these sorts of hardware these days, thanks to evolution of better hardware. Good riddance!
For compatibility reasons, this memory space is still used during boot, since until you get the drivers loaded and PCI hardware set up, it behaves in "legacy mode", so you can still run really old DOS and other old software on the machine without it behaving weird.
However, once your graphics card, for example, is configured, it will have BAR's (Bus Address Range) set up to tell the world where it is located in the bus address space (physical address). ioremap will map the physical address to a virtual address that you can use in the kernel - baseptr in this example. [I could go into much more detail about ioremap, as I've been working on some code derived from that recently].
As the other answer says, you will need to ask the device what it's BAR is (which is part of the PCI configuration space), and then map it's memory to a virtual address. You can see where things are using lspci -v|grep Memory
(of course, using just lspci -v
will give you a lot more information, including which device has what memory). These addresses are the physical addresses.