As a concrete example: suppose P != NP. Then no simulation can solve NP-hard problems in polynomial outside time, so most simulations will have the same constraint as the parent universe (where we reside) and also be unable to solve NP-hard problems in polynomial inside time.
The "hey, this universe looks like a simulation too!" argument would be: every simulation we make has P!=NP. Our universe has P!=NP... sure looks like what a simulation would be like. Obviously, every simulation's rules is heavily correlated with the rules of the parent universe. But that doesn't mean that the parent universe is also a simulation.
To take this example with respect to time: it might be that time is fuzzy in our universe. If so, it puts a constraint on how "sharp" time in a simulation can be as well without excessive penalties to runtime (like the poly inside time = exponential outside time example above for any simulation with P=NP). But if time is fuzzy inside and time is fuzzy outside, that doesn't mean that the outside is also a simulation, at least not unless you've controlled for the effect that every simulation in this universe will be greatly biased towards being like the universe.
Now, the outside (our universe) might seem to be like the inside (a simulation) in surprising ways, e.g. things seeming to be lazy-evaluated with how quantum mechanics weirdness works. But mathematical patterns might persist in seemingly disparate structures. So it may be, say, that it's easier to lazy-evaluate small systems (like simulations in RAM) because it's hard to change large amounts of information at once, and it's hard to change large amounts of information at once because that's how our universe works.
In a way, not taking this correlation into account when speculating about whether the universe is a simulation begs the question.