Today’s issue is an exception and instead of combining news and fun analysis, I will just do a full teardown of the now model OpenAI released as I believe it deserves it.
OpenAI released a new family of models - o1. These models had the much exploited internal codename Strawberry which turns out also is the later name of the Q* project. And if you remember that Sam Altman was fired from OpenAI last year because "he was not consistently candid in his communications with the board”. But also there was another rumor, project Q* was considered to be the root cause for disagreements on how to handle consequences from such powerful technology.
The highlights/benchmarks (also known as Evals) are impressive. And while those never really pan out the same in the real world, it is worth mentioning them and acknowledging their heights.
Programming - Codeforces is a competitive programming questions website where people solve hard problems and get ranked accordingly. Report says o1-preview model was in the 89%, which means it was better than 89% of the other competitors. And this is quite significant, compared to the o1-preview that we have in our hands currently, which scores 62%. Does this mean it will do better than 89% of programmers in real life? No. But it is massive and we will talk about the implications.
PHD physics/bio/chem - it scores better than PHD human experts on the GPQA diamond benchmark, which is a science “A Graduate-Level Google-Proof Q&A Benchmark”. Its accuracy is 78.3%
Math - math has always been the weak point of LLMs but now it places among the top 500 brightest high school students in US by solving correctly 93% of the problems on the Math Olympiad.
Also it outperforms GPT-4o in all the below metrics:
All in all, this is the biggest jump of capabilities we have seen in a while.