That is one advantage, yes. If one is perceived to have earned their position through hard work/talent then one is more likely to continue to advance because they will be considered for future promotions/opportunities. If one is perceived to have gotten their position as a handout, then one is less likely to advance further because they will be less likely to be considered for future promotions/opportunities.
Another disadvantage of being perceived to have gotten a position through affirmative action is that it's really easy to internalize those perceptions which will further hinder performance. Everyone has experienced impostor syndrome or feelings that they are in over their head even when they are absolutely capable and doing a great job, but when those feelings are being reinforced by others attributing success to handouts rather than ability, or that any good work they do is luck (or due to someone else helping), it's a very difficult thing to dismiss that.
Personally, I loathe affirmative action because it leads to exactly this kind of bullshit. There are better solutions to the inequality caused by systemic prejudice than affirmative action, and it would be a good thing to find them.