The problem is that harassment is subjective, so you can only ask for people's opinions rather than setting an absolute standard.
Take your nice sweater example. If said casually it would just be friendly small talk. If said leeringly while standing uncomfortably close and trying to look down it that would be something else. In between there is a whole spectrum of behaviour and annoying as it is I'm afraid there is no ISO standard to compare it with.
Looking at very specific examples and asking if people felt uncomfortable is all we can do. Of course, we make a judgement, simply feeling uncomfortable is not in itself harassment.