The Deploy Log | Breakthroughs
olmo-eval
Ai2 released olmo-eval, an evaluation workbench that builds on OLMES and supports agentic and multi-turn evaluation, pairwise model comparison, and per-question analysis with standard errors and minimum detectable effects.
The call on this one
What shipped is free. The call on it, what to do about it and the condition on that, opens with a signup: free, no card, every edition in full.
It becomes $50 a year, and signing up now keeps your first year free.