I did some work for a local bank, and their ATM's were running Windows XP (not embedded), IIS (can't remember the version), and IE. This was to allow them to serve "rich content" (movies, images, animations, etc), without having to write it all themselves. The ATM just had IE talking to IIS, and displaying the results in "kiosk mode". The buttons on the sides of the screen were mapped to keys on the keyboard (I think), and that's how it ran.
I specified a full set of ports that needed to be accessible to the ATM controllers, and that was all that was supposed to be accessible from the network.
However, if you can get access to the back of the machine, it has a second monitor, keyboard and mouse, and you can access the OS, and do whatever you want to do. I *THINK* that the keyboard and mouse were locked away in the vault (or at least behind a door), but the hardware itself is pretty standard PC, so I don't imagine that it would be particularly difficult to add a USB keyboard or mouse and gain access when rebooting the device. Maybe even boot from a USB disk or similar.
The reality is that if you have physical access to practically anything, it is game over.
Personally, I would have been a lot happier to see a stripped down Linux kernel + minimal OS, BIOS passwords, bootloader passwords, etc than the entire Windows stack. Less to verify == more security.