Yeah, there's no word to describe something negative that can't be turned into an insult. Whatever words we use to describe people on the fringes of society will be turn against those we don't like to treat them as if they are on the fringes of society.
If it somehow became popular to describe people of a certain type using the phrase "people on the fringes of society," then within a generation or two kids would taunt each other in schoolyards with "No, you're on the fringes of society," and "No, you are!"
Just like Special Education students in schools, students who require one-on-one teaching from trained teachers. It was abbreviated as sp.ed., and where I grew up, "sped" was a common insult, as in "you're such a sped", or "that sped is so retarded".
I once got into a debate with someone on Reddit where they claimed "Middle East" is an offensive term, which was news to me. They preferred "West Asia and Northern Africa", or something. I claimed that this is just taking the same path that "oriental" took, which is now taboo in some countries, even though it literally means "from the east" and is the opposite of "occidental".
If it somehow became popular to describe people of a certain type using the phrase "people on the fringes of society," then within a generation or two kids would taunt each other in schoolyards with "No, you're on the fringes of society," and "No, you are!"