That heavily depends on the military in question and the circumstances. Not to invoke Godwin here or anything, but that was a recurring theme among rank-and-file when it came to the engineered destruction of around 12,000,000 lives in mass executions and graves at the hands of Nazis and their sympathizers during WWII. "Just following orders" is equivalent to saying "I'm just a robot and incapable of independent thought". How is that any different than an insanity plea which indicates that you do not have the mental capacity to tell the difference between good and bad?
Sure, as a lowly soldier you're less likely to receive seriously harsh punishment as [hopefully] your personal involvement is limited and some legal issues might not be known to you. But in some militaries - and this I know for a *fact* - you are NOT to obey a known illegal order even if that order comes from the Chief of Staff himself.