How does the router know not to work before you've even added the masquerade rule?
Because my LAN uses RFC1918 IPs, which are not routable over the internet. You cannot connect to 10.0.0.2 on my LAN.
Similarly, without the masquerade/snat rule the router passes an outgoing packet with source IP 10.0.0.2, which then gets dropped by the ISP because the ISP has IP filtering or, if the ISP allows the outgoing packet, a reply packet will not come.
So, let's say I have a typical home internet connection. My router gets a public IP and my LAN is one of the RFC1918 subnets. My ISP does not have a static route to my LAN.
With just ip.forward enabled and no iptables rules (allow all), in theory my LAN would be accessible from outside, but for that the ISP would have to have a route to my LAN and even then, my LAN would only be accessible from my ISP, not the rest of the internet.
A subscriber to my ISP could add a static route to his router to access my LAN, but for that he would need to be on the same VLAN as me and the ISP has to not have IP filtering or something similar implemented.