Sure but dry wall has paper on both sides. To prevent frost/mildew/water building up you need something to insulate the drywall from the elements. Otherwise the paper will rot and you you'll start to get mold etc. Which will not be good.

"Seal Toi" (Dan Acroyd, Coneheads reference.)