Vegetarianism is about the minimization of cruelty and suffering.
Plant life does not factor into it because they can not suffer. They can’t suffer because they have no nervous system with which to think. They also have no physical mechanisms with which to feel pain. And even if they did, they have no thoughts, so the pain would mean nothing. They have no fear, panic, or sadness. They live, but they live without consciousness. So you can not torture a plant or make it suffer.
On the animal spectrum, not all animals are the same since some animals have small brains and simple thoughts and other animals have complex brains and complex thoughts. At the top of the animal spectrum you have humans with the most complex brains and abstract thoughts and intense sensations of fear. Humans have a high capacity to suffer. On the other end of the spectrum you have animals like spiders with comparatively simple nervous systems and simple thoughts. They have a much smaller capacity to suffer. That’s why it would feel more painful to watch someone rip the legs off a spider than watch someone rip the legs off a cat or horse or chimpanzee. So there’s a spectrum of animals ordered by how self-aware they are and how complex their thinking is: spiders, fish, chickens, ravens, octopus, cats, dogs, pigs, cows, horses, dolphins, gorillas, chimpanzees, humans... roughly something like that. Everyone draws a line on the spectrum, whether consciously or unconsciously, what they are comfortable with. Some people are fine eating fish and chicken, but not pigs and cows. Other people are fine eating pigs and cows, but not chimpanzees, who are almost human. Some people are even fine eating chimpanzees and feel no empathy when they shout and panic. Almost everyone at least agrees that it’s not ok to eat humans. But some people even do that. A vegetarian draws the line at it being not ok to eat any animal.
Some people argue that oysters, despite being animals, are vegetarian. They aren’t, by definition of the word vegetarian, but it is true that the argument for plants applies to oysters. Oysters do not have a central nervous system, no consciousness, and no thoughts. So they can not suffer.
Not all vegetarians are vegetarian for the same reasons. Some people have a spiritual belief that all life is sacred and equal, but that’s not my belief and not something that’s supported by any facts I’ve seen. What I outlined above, though, is simple fact and simple reasoning.