A focused newsletter on model evaluations and benchmarks, for people who want to know what a new model can actually do
Weekly
Every week: a new model's evaluation results with the benchmark used and its known limits, new or revised benchmarks and what they leave out, and results later disputed or independently checked. For someone who wants to know what a model can actually do, not a marketing chart.
Not a benchmark score reported with no context on what it measures, not marketing charts with no methodology attached, not model release announcements with no evaluation data, not general AI industry news better followed as its own subject.
Where the news comes from: LMArena blog, arXiv: machine learning, Hugging Face blog, Quanta Magazine, arXiv: computation and language.
Nobody gets this one yet. It starts when you do.