Yes it really does work that way. Any PCI device in your system can read/write to any location it can address. If the device only has 32-bit PCI then it is limited to the lower 4G of memory space, if it is 64-bit PCI then it can go anywhere.
There is an IOMMU (http://en.wikipedia.org/wiki/IOMMU) but I am not very familiar with it. More modern machines than I was working with would probably implement this for protection from the device.