It's certainly true that the brain does not exist in isolation, and that any tightly-coupled system essentially acts as a single system.
However, smartphones don't necessarily offload the right stuff from the brain. You ideally want the brain to do the thinking stuff and your smartphone to offload all the mundane stuff that interferes with coherent thinking. This is because of how the brain works.
If you push specific parts of your brain hard, you can extend them by up to 20% in the number of neurons involved (and much more than that in the number of synapses), but only by sacrificing neurons from other functions in the brain.
The ideal is to have the smartphone do the stuff you need to shrink in order to grow the stuff you want to actually utilise, so that your functionality hasn't worsened anywhere but you can still actually become a mega-genius at some tasks.
This is not how smartphones are designed and this is not how smartphones are used. Smartphones are designed stupidly and used stupidly. This is a serious problem, leads to phone addiction, and a net redunction in mental capacity. Now, off with ye, I need to get back to that minesweeper app.