The impact of missing TRIM depends a lot on your usage. When the FS starts to reuse blocks from delete files, performance starts to go down because the SSD doesn't know that part of the block can be discarded and will have to do extra work to keep that unused part intact (as Bengie explained in more detail in his answer).
So if you mostly use big files, the impact will be negligible because the FS won't use partial blocks very often.
Or if you don't delete files often, the impact will be negligible because there isn't many blocks to partially reuse.
But think of it, if TRIM was added, it's not just for the fun of it, people saw the performance degradation first. Just read the comments on the web before TRIM was added. Just read the benchmarks when the first drives with TRIM came out. Yes some user will never notice it, but that doesn't mean that nobody does and that TRIM isn't useful.