Is happiness always at the expense of others?

Happiness is something we all want. But sometimes I wonder, is the only way people can be happy in life, is at the expense of others? Since America's founding, white people led more comfy lifestyles and had a happier experience compared to minorities, at the expense of Natives and Blacks. Even today, whites enjoy white privilege simply for being their skin color, at the expense of minorities, especially blacks. Naturally most don't want to give that up, so most tend to be very condoning of racism and tend to turn a blind eye because of it.

I start to think about each individual person's happiness. Surely enough we are a society that believes in material possessions. And if it takes a material object to make one happy, wouldn't one be doing that at the expense of all those who make them in third world countries, underpaid?

It seems like, regardless of what makes us happy, as much as one person is happy, another is sad over the same thing. Is it only possible for their to be happiness at the expense of others?

Are those that have to live unhappy and miserable existences, to thank for your happiness in the end, are they the underdog hero, not only given a bad rep, but treated unfairly considering the situation?