When someone expresses displeasure at something someone else said, the immediate gut reaction is instead to get defensive.
"Oh, lighten up."
"It was just a joke, you precious snowflake."
"You're too thin-skinned."
Thus putting the onus (or blame) on the victim rather than the aggressor. Here's the problem with this entire notion.
Sure, words themselves have as much meaning and power as the listener ascribed to them. But over time, when one hears negativity frequently or constantly, it breaks down the listener's resolve and defenses. Consider an interrogation room. The suspect is placed in an uncomfortable physical environment, already affected by the negativity and pressures surrounding the circumstances of being in there in the first place. Law enforcement officials -- symbols of authority -- apply various techniques of pressure to extract confessions or ascertain guilt.
But they're just words, right?
This is the power of words. A single water drop might not overcome a stone barrier. But relentless waves of countless pounds per square inch will.
This is why "you can't let what they say affect you" is useless, and is either the default of those who've never been subjected to it (possibly through lifelong positions of social power) or are using it as a defense mechanism by externalizing their own inner conflicts, and living in denial of their own vulnerabilities.