Apple PR failed to remove a lot of the misconceptions about the little location file on the device, so let me take a crack. The location information on the iPhone is not YOUR location, but rather a collection of location data points that includes the cell towers in the local vicinity, some of which could be up to 100 miles away. As a result, the phone is not storing your location, but instead just downloading a bit of cache data so it can look up your location faster when you want it to. That responds to the 'storing' part of your post. With respect to tracking, if you know a way for a navigation app to give you directions and locate you, but not track you, I'd like to hear it. In the meantime, if you would like the phone to NOT know where you are, period, just switch off 'location services', which, as of 4.2.3, also deletes the local cache database.
Meanwhile, all sorts of information about your location is leaked by all the devices you use. One trivial example is your IP address, which, while not a precise locator, gives the other end some idea about where you are (assuming no proxies, etc etc)