I don't think that it necessarily damages a person's self esteem to have to work hard. Succeeding at difficult tasks is one way that people build self-esteem and a sense of self-efficacy... and being around people who get your jokes is priceless.