If you have a goal of "diversity" that implies that you are going to treat people differently depending on their race and gender. Race and gender becomes part of the selection criteria. Isn't that pretty much the definition of racism and sexism?
A system where the speakers are just randomly chosen from a pool of people who submitted papers would definitely not be racist or sexist, but it probably wouldn't meet your definition of diversity. (Probably wouldn't make for a very good conference either.)
A fair system is one where race and sex do not influence the selection criteria. It cannot have diversity listed as a goal. If it is a goal then you have created a racist, sexist system for selecting who gets to speak.
A system where the speakers are just randomly chosen from a pool of people who submitted papers would definitely not be racist or sexist, but it probably wouldn't meet your definition of diversity. (Probably wouldn't make for a very good conference either.)
A fair system is one where race and sex do not influence the selection criteria. It cannot have diversity listed as a goal. If it is a goal then you have created a racist, sexist system for selecting who gets to speak.