I’m above-average attractive and almost never face aggression. I’m also above-average height and broad across the shoulders, though, which probably has more to do with it.

That said, in my experience, customer service workers are far more likely to be the recipients of aggression and abuse than to give it out. Like others, I’m very curious as to where you live that you’re experiencing otherwise; I’m thinking it’s gotta be a place where it’s very difficult to fire people, a place with very strong labor protection laws.