Privileged guests (usually the domain0 guest) are an exeption to this because they do have direct access to the real hardware (and the hypervisor doesn't do any work here, privileged domains write *directly* to the device's registers). Privileged domains also runs the virtual device backend drivers, which does the work for virtual devices in the guest OSes.
So unprivileged Xen guests don't think they have access to the real hardware, instead they use virtual devices though custom Xen drivers.
Also, another reason why you need Xen-modified kernels is virtual memory: guest OSes (privileged and non-privileged) don't have direct access to page tables, CR3 register, etc.
When they want to update a page table entry or do a context switch they have to ask the hypervisor to do it; so the VM system needs to be changed to this.
NetBSD on Xen is a port in the NetBSD terminology, the same way NetBSD/mac68k and NetBSD/sun3 are 2 different ports. The machine NetBSD/Xen runs on is different from the machine NetBSD/i386 (which should really be NetBSD/pci386