Google's Gemini 2.5 Pro in 4 Minutes

--- type: transcript date: 2025-03-25 youtube_id: bbkcQp5X3h0 --- # Transcript: Google's Gemini 2.5 Pro in 4 Minutes Google has just released Gemini 2.5 Pro Experimental, its latest state-of-the-art thinking model, which leads in a number of different benchmarks with notable improvements in enhanced reasoning as well as coding. This model has leaprogged above the other models by the biggest margin of points yet. So, 2025 is definitely shaping up to be a cutthroat year for AI Labs. Just yesterday, Deepseek released their V3 update with considerable improvements with Deepseek R2 as well as GPT5 are rumored to be just around the corner. You're going to be able to access this model on a studio.google.com. Now, just some notes with the model. The knowledge cutoff is January 2025. I believe this is the most recent knowledge cutoff across all of the Frontier AI labs. The model is it does have a million tokens of context that you can pass in, which is an absolute huge number of tokens. Now, the other great thing with this model is it has native multimodal understanding, real-time streaming, as well as native tool use. You're going to be able to use this within the API right now from AI Studio. And additionally, this is going to be rolling out to GCP on their Vert.Ex AI platform if you're looking to integrate this into an application. Additionally, if you are subscribed to Gemini, the gemini.google.com app, you will be able to access it within the interface. Now, the one thing to note with the Gemini app is you won't be able to access deep research with this model yet or their recent canvas feature which allows you to create these HTML games. Now, just to touch on the leap in performance, Gemini 2.5 Pro Experimental came out on February 5th, and just a little over a month later, we see that this has an ELO bump of 1380 all the way to 1443 with the next closest model on the LM Arena leaderboard being Gro 3 Preview with a score of 1404. Now, just to quickly go over some pieces within the blog post if you're not as familiar with reasoning models, but what the model will do before giving an answer is it has the ability to analyze information, draw logical conclusions, incorporate context and nuance, and make informed decisions. In terms of the specifics of the model, they mentioned that we've achieved a new level of performance by combining a significantly enhanced base model with improved post- training. To go through some of the benchmarks in terms of humanity's last exam, this is a recent benchmark from scale AI. This ranks in 18.8% on GPQA diamond. This ranks 84% just shy of claw 3.7 thinking as well as Gro 3 beta with their extended thinking mode enabled. In terms of mathematics, we have an 86.7. Now, if we compare that to Gro 3 beta, the difference here is that this is at one shot. This is with one query to the LLM. This ranks at the top. Now if we just go into some of the other benchmarks, the one thing that's interesting with the coding benchmarks, so live code, ADAR polyglot as well as Swebench is there are mixed results here. So for the ADR polyglot benchmark, this ranks at the top whereas with live codebench as well as Swebench, it doesn't quite perform quite as well as Sonnet 3.7 or Gro 3 beta with extended thinking. But an interesting debate is around the ADAR polyglot benchmark because this is arguably a benchmark that is more attuned to day-to-day tasks rather than things like competition code and agent coding. Here is a quick demonstration of them generating this dinosaur. So, if you've ever been on one of the loading pages for the Chrome browser, you might be familiar with this dinosaur where what they did is they generated the code for this and here they're putting it within an editor and they have this preview of this pixel dinosaur running. As we see here, it's that familiar game that you've probably played within your browser before. Just my first impressions with the model. I asked it to make a 3JS game where there was a snowman playing soccer. In just a couple prompts, I was able to generate this where it has the physics of a ball. And as we can see here, if we just take a look at the snowman, it does look like a pretty reasonable snowman. Obviously, there are some other requests that I would have to make to actually make this a bit more playable, like adding collision detection and some sort of mechanism to have the score potentially when the ball goes in or to actually simulate another player. But this is just to give you an idea in just a handful of prompts what it was able to do. I said create a playable soccer game with keys WD in 3JS and HTML and I went through a handful of prompts that ultimately generated this game for me. Otherwise, a huge kudos to the team over at Google for this release. I'm definitely going to be leveraging this model quite a bit more. Otherwise, if you found this video useful, please comment, share, and subscribe. Otherwise, until the next