When someone expresses displeasure at something someone else said, the immediate gut reaction is instead to get defensive.
"Oh, lighten up."
"It was just a joke, you precious snowflake."
"You're too thin-skinned."
Thus putting the onus (or blame) on the victim rather than the aggressor. Here's the problem with this entire notion.
Sure, words themselves have as much meaning and power as the listener ascribed to them. But over time, when one hears negativity frequently or constantly, it breaks down the listener's resolve and defenses. Consider an interrogation room. The suspect is placed in an uncomfortable physical environment, already affected by the negativity and pressures surrounding the circumstances of being in there in the first place. Law enforcement officials -- symbols of authority -- apply various techniques of pressure to extract confessions or ascertain guilt.
But they're just words, right?
These words are like the sea. Seawalls are built as defenses, and many will stand over long periods of time because the effect of words aren't enough to overcome the strength and resolve of the construct. But amplify the power of the sea, and even the most resolute concrete behemoths will crumble.
This is the power of words. A single water drop might not overcome a stone barrier. But relentless waves of countless pounds per square inch will.
This is why "you can't let what they say affect you" is useless, and is either the default of those who've never been subjected to it (possibly through lifelong positions of social power) or are using it as a defense mechanism by externalizing their own inner conflicts, and living in denial of their own vulnerabilities.