Freedom of speech doesn't mean speech without consequences. See: slander. How is that different? If the first amendment is absolute, then we should let people lie and destroy other's reputation at will.
"If truth is beauty, how come no one has their hair done in the library?" -- Lily Tomlin