You only have to look at the balanced chemical equation to see (which I got wrong, also not a chemist, and you're right about the bicarbonate being soluble!)
CaCO3 + H2O + CO2 = Ca(HCO3)2
This will certainly reduce the acidity of the water, increasing the pH, because carbon dioxide is being removed from the water and is in equilibrium with the carbonic acid ions.
CO2 + H2O = HCO3- + H+
There is a reason, however, that I know just from a thought experiment that this cannot be a positive feedback loop. If it were a positive feed back loop and calcium carbonate reacted with carbon dioxide in the water to increase the acidity of the water, then I could take some limestone, throw it into some water, and over time the limestone would totally dissolve and leave me with a cup of very strong acid. In other words, if this were the case, calcium carbonate in normal water would be unstable. You only have to look at some old quarries to see that this is not the case!