I didn't mean necessarily mean specific memories, but at least a more vague continuity of "sense of self," the "I" that is being reincarnated. Without the present "I" sensing a bit about having been the past "I," in what sense would they be the same, i.e. why would the new "I" be a reincarnation of the old, instead of a whole new thing? Something like whatever sense we have when, waking up each morning, we "know" we were the same person who went to sleep the last night. Having concrete, verifiable memories would certainly help prove the reincarnation.
If this continuity "fades away" after the first few years of an infants life, then this is an awfully weak sense of the word "reincarnation" --- at most, one would say that the life of the previous person is extended a couple years past their previous bodily death, to flail around in a barely-self-aware infant for a bit longer before permanently vanishing into the void ("replaced" by a new "I" that, lacking the self-identified continuity with the old, can hardly be said to be the same thing). Anyway, in this scenario, reincarnation would not provide a "chain" of mind that extends into the far past and indefinite future, because this would be broken (basically the same as an entirely non-reincarnational system) at every generation, only with a bit more time as an infant (perhaps returning the "lost" time from the previous infancy occupied by another, now lost and disconnected, mind).
So, what part of the "persona" do you think survives --- and why does this part deserve to be called the "same" persona reincarnated, rather than simply a "similar personality" (which is hardly a controversial claim, that younger people will grow up with the same distribution of general personalities as the preceding generation).