One only needs to read Stevens' dissent in hindsight to recognize that all of his concerns were and are legitimate, and the worst of which have come to pass.
Around 2015, my boss got permission from corporate to take me and a few of my coworkers to a leadership conference for a day. It was setup at a church and had religious theming around it, and a lot of "leadership values" were communicated. It was the usual fare of responsibility, accountability, truth and trust you'd expect, mixed with the religious overtones you'd also expect. But what I didn't expect is how trite this fictional narrative would prove to be.
I think about that a lot now, and how my religious upbringing was bombarded with this messaging (in a good way). And how much of the same world that previously claimed espouse it has complete turned their backs on it in such a short time.
Is it dead or did it ever exist at all? At least we don't seem to pretend to have higher values anymore. If we even have leaders at all, they are not measured by the yardsticks we used to make.
If you look at papers on benchmarks, they're usually created to expose gaps in how models are trained. It should be no surprise that models get better on them over time, because you can't get better at what you don't measure.
Cherry picking the benchmarks you present is where the falsehoods lie.
Another thing that sort of puzzles me about benchmarks is that LLMs are not deterministic and do not always complete a problem. So what are the results actually representing? The best run? The average? It is all in some ways a falsehood
That's a good point and conventionally if benchmarks aren't run as "one shot", it is denoted as "benchmark@K". Inference time scaling has historically shown improvement.
Generally though, many of these fairness complaints do go away if there is "3rd party testing". Right now, companies reporting their own benchmarks has all the problems that 3rd party testing resolves in many other industries.
I think it's hilarious. LinkedIn is rushing to de-legitimize themselves so hard that they're inventing a new market for someone else to step into. Apparently indeed doesn't want to take it... not sure what's going on there.
https://en.wikipedia.org/wiki/Citizens_United_v._FEC#Dissent...
reply