I think the best possible change to the App Store (aside from complete transparency) would be a button in the submission queue labelled "resubmit under different reviewer." Every time it was pressed, the reviewer who had rejected the app would be banned from reviewing it again; thus, we'd be able to get over humps like thirty dictionary apps being accepted and one rejected, or the same app being accepted and then rejected later in an update for things it has contained since the original version. The number of bannings a reviewer collected would be a very helpful metric for Apple come review-time. Of course, if you ended up banning the entire review staff from looking at your app, it's not going anywhere (thus avoiding people bullying the "market" for reviewers into only ever accepting apps. If you're unilaterally rejected, all the reviewer-bannings on your app are lifted—and thus no longer count in the HR process—and you're the one who gets banned instead.)

That defeats the point, since the guys who accept all apps will get a score of zero, while not doing their main job description.

That, in my mind, is the most annoying thing about it. I don't care what the rules are (within reason), but make sure we know what they are and apply them consistently.