In special relativity lets say I'm sitting next to someone and then I go for a walk and come back. When we compare clocks they will be the same since otherwise there would be symmetry breaking and we could establish a preferred inertial frame.
You don't need to factor in acceleration to have the clocks get out of sync, just remember that changing directions means you won't be at rest the whole time in any reference frame:
Inertial frame of your friend: First you move away with say 1 m/s. Halfway through you move towards him at 1 m/s. You're moving, so your watch will be slow when you return.
Inertial frame of you walking away: First your friend moves away at 1 m/s. Halfway through you start moving in the same direction at 2m/s. You're moving faster, so your watch will be slow when you catch up with him.
Inertial frame of you walking back: First you move at 2m/s, while your friend falls behind at 1 m/s. Halfway through you stop moving, and wait for your friend to catch up. You've been moving faster, so your watch will be slow when he reaches you.