Simple definition usually is that if two organisms can't produce viable, fertile offspring then they're of different species.
Even in sexual organisms, this definition of species has some problems. You can easily have organism A that can breed with B, and B can breed with C, but A and C are incompatable. How do your divide up the species lines in this case?
In general, when a new species forms, each organism has to have others it can mate with, or else they would just die out without any offspring. The speciation process is a gradual one, and so, theoretically, there is probably a path you could follow between any two sexual organisms where any pair on the path could theoretically mate.
In Avida, for simplicity, we determine species by testing each orgasnism against the species of its parent. If it can cross-over at most points with the prototype of that species, it is marked as being part of that same species. If it cannot, we create a new species for it where it is the prototype. Not an ideal method, but it works in most cases (and we rarely need to resort to the species concept).
What's fun, is that this even works for asexual organisms. We can force all possible crossovers (in isolation of course -- this never feeds back into the system) to see if they would have any ability to mate if they has been sexual.
Dr. Charles Ofria
Director, MSU Digital Evolution Lab