Part of this is an unintended effect of the present culture in America.

As a white dude, if I say something, rude or in approriate to another white dude, maybe they will chastise me for rudeness or even think less of me.

Still if I say the same rudeness to a woman or a minority, I run the risk if being of being labelled as the anti-Christ of our age: sexist or racist.

I am not saying that is wrong, but it has a strong effect. If you are smart, you mind your Ps and Qs in those contexts. The only place one can relax on this point is when there is no risk of either of those labels.

More challenging, you are limited in signaling that you would not take offense. Even if you say so, your collegues would be running a real risk to take you up on it. Words and actions can come back to haunt YEARS later, and the "he seemed fine with it" will never be an acceptable defense.

Of course if we were just never ever rude or inappropriate to anyone then there would be no issue.... and if pigs could fly we could all use them for transportation too.

This is simply how things are right now.