The boss sees you with a smartphone, he assumes you're always on duty. Because he can. Because YOU can.
Well, you could:
(1) Not give your boss the number. Grapple the expectation head-on.
(2) Give your boss the number with the caveat that at some places the reception really sucks. "Gee, boss, sorry I missed your call. I must have been in a dead spot!"
(3) Get a Google Voice number, and give that to your boss. It's up to you whether or not you forward it to your phone.