A fugitive is the antithesis of the organisation, conference and attendees. It's a conference for and about the legal profession. As far as I'm aware, Assange has zero legal qualifications whatsoever.
He's involved in whistleblower activity detailing alleged abuses of the legal system and his status as a fugitive is likely to be linked to this more than the supposed rape accusations. But even assuming the rape accusation to be sound, his activity in whistleblowing makes him definitely relevant "for and about the legal profession".
That's like saying you should invite a convicted paedophile to your school safety talk, or a rapist to your rape counselling group. Maybe it SOUNDS good and fair and balanced, but the practicality is insanely stupid.
Why insanely stupid? It might give some insight on why these people did what they did. It's not a matter of fairness, it's a matter of having a look at the other side to try to understand. You seem to imply that merely listening to them would mean justifying them or giving them reason, which is not the case.
Criminals (and Assange is one, legally speaking, in the UK for skipping UK court bail) DO NOT get a say in how their justice system handles them, or invited to conferences about the legal profession. Reasonable outsiders make sure the law is fair and balanced for all, but the criminals themselves? No.
Talking at a conference does not in any way imply "having a say" in anything. Whoever wants to write laws fair, balanced and effective has to meet the criminals too to be able to better understand why some crimes happen, how to prevent them and/or mitigate them.
So, why not?
It's not libel if it's true
This might be the case in the US, but in other countries the law might be different. Not sure about Japan, but countries in which proving "truth" is not a valid defence against an accusation of defamation do exist.
Of course they might be immature, since what they might consider "sexually inappropriate" might be perfectly fine within that specific working environment, making them the problem. If you want to work as a waitress in a topless bar and consider working topless "sexually inappropriate" the problem is yours: nobody forces you to work there as nobody forces you to contribute to this specific project.
Basically "inappropriate" sometimes just means "I personally don't like it so it should not exist at all".
You're still missing the point. The amount of sheet metal surrounding you doesn't determine your worth but it surely determines the outcome of a collision with something else, be it your fault or not or even without a fault at all. A pedestrian has to accept reality: if he gets hit by a car he easily risks injury or death, which means that for him being careless when crossing is much more dangerous.
Common advice for "soft" street users, be it pedestrian, cyclist or motocyclists, is to assume that everyone else is either completely incompetent or actively trying to kill you since in case of accident you dont want to be right but dead, you want to avoid the accident in the first place.
You cannot force other drivers to always pay attention or always "do their jobs": it would be nice but reality begs to differ. Being defensive is the best survival strategy.