I had a similar situation once in which I was working away as a contractor, but the manager wasn't really aware of everything I was doing.
The best advice I received, which came from an outside source, was to start emailing the team leader and the manager a quick "status" update every week. Just a quick email about what I was working on that week, what I accomplished, and any issues they should be aware of or handle. It worked very well, and it tended to cut down any interruptions from them wandering by asking me "how's it going?" As time went on, they learned to trust me more as a professional, and it became less of an issue.
Now, I hate mandated weekly status reports as much as anyone, but if the perceived problem on their end is that they don't know enough about what you're doing, I would much rather start sending them email with the relevant information. Otherwise, you might find you have to start filling out detailed weekly status reports, attending regular status update meetings, or something else more painful that a quick email.