A common theory to explain Venus' slow and backwards rotation is that it suffered a large impact similar to the one that formed the Earth's moon, only in a direction counter to it's original rotation, so much that it put the brakes on Venus so hard that it's now slowly spinning in reverse.
Also it's thought that Venus is simply too close to the Sun, there was a time when the Sun wasn't as bright and water may have been on the surface but as the sun matured the "goldilocks" zone shifted outwards and Venus got cooked.
The rotation idea doesn't really hold water. It would have needed to happen very early in planet formation, or the whole planet would still be molten today, plus the details of the impact bringing angular momentum to 0, which requires the pieces that escaped the collision to have just precisely the right parameters post-impact, are "finely tuned", which is the polite way scientists say "BS".
Keep in mind that, at least with Earth, the surface isn't rigidly coupled to the core. While the difference in rotation is only significant in geological terms, it means that you can't stop the core spinning just by smacking the crust around, and you can't really put enough energy into the crust to counterbalance the spin of the core. A single large impact just can't transfer enough momentum with a glancing blow - too much of the transferred energy ends up thermal, liquifying, even vaporizing, crust and magma. An impact while planets were still forming, or at least before the iron catastrophe might do it, but again the odds of the numbers working out exactly right to reach near-0 angular momentum are, well, astronomical.
Venus getting cooked goes beyond the oceans boiling - the vast majority of carbon is in a planet's crust, and you don't get an atmosphere like Venus's without melting all the crust. The atmosphere is a side-effect of the real mystery under the surface. The simplest assumption is that Venus's lack of rotation and repeated crust overturning are symptoms of the same weirdness, and we're unlikely to guess what that is from studying Earth geology.