OpenAI's o1: Has It Surpassed Claude 3.5 Sonnet? Testing with Cursor

--- type: transcript date: 2024-09-13 youtube_id: MV2OdVRQnew --- # Transcript: OpenAI's o1: Has It Surpassed Claude 3.5 Sonnet? Testing with Cursor in this video I'm going to be trying out the 01 preview as well as the 01 Mini model that just came out from open AI within cursor so at time of recording cursor doesn't yet have this available within their Pro tier and from open aai and it's only available for those that have access to tier five with an open AI so if you have tier five access you will be able to plug in your API key and get this working there are some rough edges there from what I found but that's one thing to be mindful of hopefully they will be rolling out this over the coming days and weeks to other TI as well what you can do is if you command shift p you go to cursor settings here what you'll be able to do is you'll be able to add a new model and you can just grab the model string here it's within the URL bar you can also just grab it from their documentation page if you just put in o1 preview add that model and then you'll have these checkboxes here where it will show you the different models that are turned on once you've added any model names you will have to add your openi API key I have this simple nextjs application here I thought I'd just play around with it a little bit and see what this new model can do first we're going to open up our composer command shift I here's our composer View and I'm just going to give it one of our files within our project let's just say the page I'm going to say I want to make this page a landing page for a brand called developers digest so we'll go ahead and submit that and if you haven't used the composer view before what's really cool with it is it will create net new files for you it will understand the context of what you've passed in if you select different files it will understand what's within those files and the one thing to know with this new model from open AI is it doesn't stream back so you're just going to get the response back in full the other thing to note is the 0 Mini model does actually outperform the 01 preview model on some task for coding and it is much cheaper as well as faster I think this is probably going to be the model that a lot of developers gravitate towards I'm going to go ahead and accept those changes now if we look at our page here we can see some of the changes I'll go ahead and click buev we'll start up our server here and and we'll take a look at what it's made so far right off the bat it did import a logo that I don't have so let's just go ahead and get rid of this logo here and for that sake let's get rid of all of that image component as well here is the page that it made for us welcome to developers digest your goto resource for the latest developer news and tutorials learn more privacy policy terms of service all right so let's go back to cursor let's open up the composer again and let's say I want to add three pages a Details page with YouTube thumbnails a GitHub page with some recent Pro projects as well as a contact page I also want to make this look much nicer we'll go ahead and submit that so I think there's definitely going to be some mixed feelings with the model even though it is more capable and competent I think the fact that it doesn't stream back is going to be something that a lot of people might not like but one anecdote that I've had from using cursor and the composer is it does seem to be really quick so I know originally they did have a deal with open AI so I'm not sure if maybe they have faster end points or something to that effect but it does seem to work pretty quick the other thing to note is for these small little changes it can feel like it does take a bit more time if I'm just asking it for a small block of code to resolve something it can take a little bit of time one thing to note with this is we can see that this is a pretty obvious error that we got here that I haven't seen Sonet 3.5 make it did put an invalid comment within the jsx here if I go ahead and look at this again we have hidden buttons we have our different pages and our navigation mind you I didn't give it much instruction but the navigation is on the bottom here if I just click that we see this page here and it's a mediocre page now mind you again I didn't really give it much detail here and then there's our contact form creating UI components just after just a handful of promps I'm not super impressed I think v0 is by far the best at generating UI components this is really not great but let's just continue on so no mind you this also just came out and the cursor team they might be able to do some magic with this model to really get the best out of it that's something to stay tuned to as well I'm going to say I want the navigation at the top let's make the nav bar black with white text also all of those pages look terrible let's add some color and make sure the is on each page I'll reference the layout as well so just to touch on the composer while we wait for it the really cool thing with composer is there's a ton of different things that you can select within here whether it's folders files code you can even index documentation that you commonly access as well which is really handy there's just a ton within here I encourage you to check out if you haven't already all right we see that it made changes to a bunch of different things head it to the layout the pages we'll go ahead and accept all of that and here we go now we have a another invalid comment that seems to be a bit of an issue but we do have a better looking page so see details here are our thumbnails we have our projects we have our contact page now this seems to be pretty rudimentary these aren't Shad cnii components or anything like that mind you I didn't ask for that so it might just be me not being used to like these sort of vanilla type of components but what kills me is this comment at the top here so I'm going to say why now we showing now just to touch on the models themselves if you are accessing it from the API it is relatively expensive especially if you're going to be using the 01 preview model1 Mini model it is a fair bit cheaper but it isn't one of the cheaper llms out there this is definitely going to cost you a little bit of money if you're playing around with it here we go here's our Pages again I'm going to say let's begin to really fill out these pages let's give them a lot of different elements and make all of them very professional one thing with this model when I'm using something like composer I'm not necessarily always wanting something to be like instantaneous right I have the expectation that it is going to take a little bit of time to go through and edit all of the changes even if I'm using something like CLA 3.5 Sonet while all these fast inference providers like sanova or cerebrus those are really great for a ton of different applications often times when I'm building out coding projects or I'm working with these tools quite heavily for different aspects I generally try and use CLA 3.5 Sonet or GPD 40 and now likely also 01 mini I'd expect is probably going to have a fair bit of time as well now if time isn't an issue it's definitely a good model obviously at reasoning as we can see from all of the benchmarks that they had all right so here we see it failed to compile we're importing a component that needs used State that's something as well that can be a little bit of a critique of the model is it does have training data up to 2023 but in terms of nextjs I find that it's not always great the open AI models I don't know if this is potentially let's just go ahead and resolve our error here we'll just use client at the top here next CH does change a fair bed having these used client use server a lot change right I've used react for over five years now and nextjs I do find that like year over-year things can change sometimes right for instance how you used to reference pages and everything within an EXs project and then app router came out there's a bit of confusion at times amongst llms in terms of how to use nextjs but this can be said about any modern framework anyone that's innovating and whatnot there can be these issues as well all right so we're getting a little bit closer it's not perfect or anything but if we play around with say just a little bit more I'm going to say I don't want to use any images on any Pages let's also add even more content to the internal Pages I'll submit that and I'll wait for it to load another thing to consider is while you're waiting for the response maybe it's going to take a minute or two minutes or whatever to get your response back you could in the meantime go ahead and leverage other llms to build out other aspects of your application right or maybe build out the infrastructure pieces like here it's saying implement the actual email sending logic using node mailer or an external service you can Branch out and start to do the other pieces of your application getting it already ready if you're trying to be one of these AI Builders and iterating quick and deploying your ideas I find right now the biggest limitation is really just the ideas it's not really the ability to execute there's a ton of different tools where if you are a decent developer or even just a determined developer you'll be able to ship things like definitely if you're using these AI tools so it did seem to run in terms of a little bit of an error here I'm not sure if this is an endpoint error maybe a potential cursor issue or something like that but that's just a quick one you can plug in both the 01 mini as well as 01 preview within this when I use o1 preview it's considerably longer than what you saw me demonstrate here but I just wanted to give you an overall sense and first look at 01 as well as cursor and how you can set it up from the API hopefully they really just start roll this out to anyone with an API key and hopefully also the cursor team rolls this out to at least their Pro panging members over the coming days and weeks but otherwise that's it for this video if you found this video useful please like comment share and subscribe otherwise until the next one