Actually, the reason Google knows that bit more about sites people visit, is that Firefox, Chrome and Safari all send each and every domain you visit to Google's Safebrowsing servers before they connect to it.
That is not how SafeBrowsing works. Firefox downloads a large database of hash prefixes. If the hashes of the domain and url are not in the list you go to the site and nothing is sent to Google. If the first bit of the hash matches an entry in the list Firefox asks Google for the list of complete hashes that start with that prefix. If the site's hash matches then you're blocked, if it doesn't you're not, but nothing more is sent.
To further obfuscate things, when Firefox finds a prefix match it doesn't just ask for the hashes matching that prefix, it also asks for the hashes matching a couple other random prefixes from the list.
Google may still know all the sites you visit through cookies on google-analytics or AdSense, but they're not getting that information from SafeBrowsing.