It's not elitism, it's pragmatism. Work is a collective, social endeavor. Getting along better with others is going to help, whether that's by communicating better, being generally more likable, or being tall and good looking (yeah, it sucks that this is a thing, but who are we kidding?)

Most of these things are under your control, so work at getting better at your soft skills, they're at least as important as your technical expertise.