It's not quite that simple, unfortunately.
The EU operates what is termed a "single market" or "internal market", which actually includes the EU member states plus a few others via separate international agreements. This is a region in which the "four freedoms" apply: goods, services, labour and capital may be moved freely between the participating states as if within their own country.
This relatively close relationship is generally seen as good for trade between members of that single market. It means there are no government-imposed tariffs on imports/exports, there are common standards and regulations for what you're allowed to sell throughout the market, and so on. This is why some people in the UK are currently arguing that on leaving the EU as a whole, we should seek an agreement to remain within the single market (a form of "soft Brexit").
However, membership of that single market isn't necessarily a win in all respects.
One issue is that the freedom of movement of labour means member states can't limit immigration from other member states. This has been controversial recently for a number of reasons. In the UK specifically, some people argue that immigration is putting an unsustainable burden on our national infrastructure. Others argue that immigrants are both helpful and in some cases necessary to keep our economy running and support that very infrastructure. Some point out that while we receive many immigrants from elsewhere in the single market, many of our own citizens also choose to work or retire abroad, and that travelling within the EU without visas is beneficial. Across the EU more widely, there is an issue at the moment with the number of refugees from elsewhere in the world who are entering member states close to troubled areas but then able to move around within the EU relatively freely. And on top of all of this, there are all the "free movement, but with strings attached" arrangements where the politicians and diplomats have been trying to dance around the problems without giving up the benefits.
There has probably been more objectively wrong nonsense said about immigration than any other issue around Brexit, but unfortunately it's long been a difficult subject and a certain part of the population in most EU states, including the UK, isn't very nice when it comes to foreigners. And just to throw one more ingredient into the mix, of course the UK also has people moving to and from non-EU states, but our visa and immigration system is overcomplicated, dysfunctional and a huge burden on those people and businesses involved. The natural assumption is that the same currently awful system would apply to those coming from the EU in the event of a "hard Brexit" where we cut ties like single market membership as part of leaving the EU, which some people see as too high a price to pay pragmatically, even if they don't in principle mind immigration from the EU being subject to the same rules as from anywhere else.
Another issue with the single market is that it is also what is called a "customs union". That means that while trade within the market is free, any member state importing from outside the market is required to impose a certain level of tariffs, regulations, and so on. That is usually seen as bad for trade with partners outside the EU single market, for much the same reasons that trade within the market is good. For the UK specifically, although it does a lot of trade with the EU, it actually does a bit more now with other partners outside the EU, and the external trade is also growing a bit faster. And of course a lot of goods and services are both provided and consumed internally within the UK. As long as the UK is within the scope of the EU arrangements, it therefore has to apply the EU rules even to internal matters and to trade with non-EU partners. Depending on who you ask and what line of business they're in, this is either no big deal or a crippling burden on trade and our national economy.
Crucially, the UK is also not free to negotiate its own trade deals for more favourable terms with non-EU partners, because the rules say that only the EU itself can negotiate trade deals on behalf of the bloc as a whole. This goes along with the whole single market/customs union deal, but if you're looking at increasing trade with, say, North America or Asia, it's a big barrier. And as we've seen recently with proposed trade deals like TTIP and CETA, being in the EU is no guarantee that your diplomats will actually close good trade deals on behalf of the member states. Apparently negotiating on behalf of the whole EU bloc, when in the real world those member states naturally have different priorities and goals and when they also have varying levels of veto powers, isn't always easy!
In the end, a lot of the controversy around Brexit is whether the known, established benefits of being an EU member state outweigh (and would continue to outweigh) the potential benefits of being free to negotiate independently with non-EU partners and to set our own rules for our home market. It's not really about "losing access to the single market" or "preventing immigration". Trade between the UK and EU member states would still happen even if the UK left the single market, just as obviously the UK trades with many other nations around the world. Likewise, people would still come and go. But there would potentially be significant extra barriers to trade and movement within the EU, and potentially lower barriers to trade and movement elsewhere, and the long term pros and cons of those arrangements are hard to predict.