The truth is probably a lot more complex, and honestly, much more human.
My guess is that he really wanted to make a spaceship, he got funding, got in over his head by showing dual use for his rockets, and then was pretty much co-opted into the war program.
He certainly was subject to arrest, and was arrested at one point. Only his particular position allowed him to avoid it becoming permanent. It is difficult to believe that after that, he was not actively trying to keep himself useful to the regime so that he wouldn't be arrested again.
Was he an Oskar Schindler? Almost certainly not. I'm guessing he simply saw the people assigned to him as what was needed to get the job done and that complaining about their fate would do nothing more than allow him to join them. I imagine he simply kept going and probably used his dream of building a spaceship to put a silver lining on the situation.
However, was he a committed Nazi? Also quite unlikely. If he could be accused of actual abuses, those were likely a mix of his overwhelming belief in the value of what he was doing and fear of what failure would expose him to. It is unclear what the mix was, but it is unlikely he was ideologically motivated. Instead, he was a technocrat, and like many technocrats, humane concerns can often fall by the wayside, a mindset that would certainly be enhanced by his need to survive.
In Wernher von Braun: Crusader for Space, numerous statements by von Braun show he was aware of the conditions but felt completely unable to change them. A friend quotes von Braun speaking of a visit to Mittelwerk:
It is hellish. My spontaneous reaction was to talk to one of the SS guards, only to be told with unmistakable harshness that I should mind my own business, or find myself in the same striped fatigues!... I realized that any attempt of reasoning on humane grounds would be utterly futile.[34]
When asked if von Braun could have protested against the brutal treatment of the slave laborers, von Braun team member Konrad Dannenberg told The Huntsville Times, "If he had done it, in my opinion, he would have been shot on the spot."[35]
The unsuspecting von Braun was detained on March 14 (or March 15),[40] 1944 and was taken to a Gestapo cell in Stettin (now Szczecin, Poland),[9]:38–40 where he was held for two weeks without knowing the charges against him.
Others claim von Braun engaged in brutal treatment or approved of it. Guy Morand, a French resistance fighter who was a prisoner in Dora, testified in 1995 that after an apparent sabotage attempt that von Braun ordered a prisoner to be flogged,[36] while Robert Cazabonne, another French prisoner claimed von Braun stood by as prisoners were hanged by chains suspended by cranes.[37] However, these accounts may have been a case of mistaken identity.[38]
The quotes show that he was certainly not invulnerable, and his actions are in line with the survival of a technocrat. This doesn't make him a hero, but there may be extenuating circumstances. We almost certainly would not have condemned him if he was some simple farm overseer expected to make use of slave labor to make a quota, but von Braun gets a lot of criticism in proportion to his future prominence in the US Space Program.