Let's not forget the the Concorde program was eventually scrapped...because there just wasn't enough interest in shortening an 8 hour flight...nobody in the end really cared for the faster trip.
No. The Block 2 Concorde was scrapped due to the oil crisis in the 70s, but it was a tragic accident caused by runway FOD from a Boeing that took the original fleet off the flight-line after 30-some years of service. Branson/Virgin even offered to buy and operate BA's planes, but due to past animosity they declined.
If you discount the R&D paid for by the French and British governments, on a day-to-day basis Concorde was operationally a profit-maker for their respective flag-carrier airlines as, it turned out, there were enough people for whom the trade-off between cost and time was worthwhile. After all, the difference in price for the NYC-London run between 1st class in a widebody and a seat on Concorde was surprisingly small.
The noise issue is an interesting one - the boom from a plane at 40,000+ feet is almost insignificant compared to a clap of thunder, and yet no-one complains about broken windows/doors/etc from that. As an aside, it's also worth pointing out that the whole "sonic boom" hysteria in the USA didn't actually exist until Boeing scrapped their own SST project. Coincidence?
I don't see this service as ever being economically feasible.
If they can fly at Mach 2+ for most of a 3,500nm range, then there will be people wanting to use it for the NYC->London/Paris route, or Dubai->Singapore, etc, as long as the ticket is price-competitive with taking a Lear.