We had a similar experience with Flex 2/3. Flash would hold onto objects through invisible references that were almost impossible to track down and release. We even went so far as setting all object properties for all objects to null before unloaded, and still nothing. It got so bad at one point that each user click increased memory usage by 10mb without ever returning it.
This problem occurred both with our own custom code and with the base components provided by Adobe/Flex. This leads me to believe that it wasn't not our code that was the problem, but a fundamental problem with the Flash runtime. Oh yeah, objects instantiated through MXML are even harder to track down and release.