This has already happened. Many conferences are on a yearly Asia - Europe - USA cycle (one year in Asia, next year Europe, next year USA, repeat), for instance several large IEEE conferences and workshops. Some of these have already decided to cut out the USA, and go on a Asia - Europe cycle. The number of registrants for this year's conferences in the USA has dramatically declined, with many countries simply no longer bothering to attend at all. Several Pakistani and Iranian colleagues already had major trouble getting visa for attending conferences in the USA (as in, they never get any), and now they simply gave up completely. Others who work at US universities no longer dare to go home for vacation or family visits, afraid that they cannot return. These are some brilliant scientists by the way.
Even many European colleagues no longer bother, as they have simply become afraid of the USA border and security theater. Last year in Los Angeles, two out of five of my colleagues got held up for 12 hours at the airport for no apparent reason (it turned out one of them had a name familiar to someone on a list, and the second's passport was erroneously on some list as well). They had Israeli and UK nationalities, and nothing that could flag them as suspicious (well behaved, well spoken, well dressed white folks, on an incoming European flight). These stories move quickly in the conference crowd, drawing nothing but disgust, and going to a USA conference is now almost considered a liability.
Also the opposite seems to happen: Recent workshops in Europe seem to draw lots of American attention now, with many American researchers now traveling to Europe, which they didn't do in the past. Even these American researchers no longer plan to attend the USA conferences anymore, and I am not talking about some random PhD student, these are the big shots that organize the conferences and give the plenaries.