After reading through James with reference to Matthew I would say that James has a is speaking about the ideas from the sermon on the mount. He tends to blend references along with personal view points. This seems to coorespond with whether he was there or not. Ie. He is more likely to quote a OT scripture and use his own words to describe things found in the NT.

I do not get the impression that he gave the sermon on the mount, but that he is expressing the ideas as he heard them from Jesus.