Fine-Tuning GPT-4o with Synthetic Data: A Step-by-Step Guide

--- type: transcript date: 2024-07-25 youtube_id: z8WHh5M8Knw --- # Transcript: Fine-Tuning GPT-4o with Synthetic Data: A Step-by-Step Guide in this video I'm going to be showing you how you can fine-tune GPD 40 minis what we're going to be doing in this video is I'm going to be showing you how we can leverage a more powerful model or a model that's good at a particular task and then we're going to generate some synthetic data that we're ultimately going to feed back into g540 and then the goal is to ultimately have better results with that fine-tuned version of GPD 40 so in this video I'm going to be showing you how you can dynamically generate synthetic we're going to be asking Cloud 3.5 to generate spgs for us as you can see here Cloud 3.5 Sonet is by far the best model that I've used that can generate svgs but if I ask the same question of gp40 mini and I say generate a SVG of a crab what we'll see here is definitely not a crab right we see like an abstract mess let's call it I'm going to show you how you can generate some synthe data from CLA 3.5 Sonet and then we're going to be normalizing that data uploading it to have a fine-tuning session on the open aai platform and then ultimately we'll see pretty quickly that we'll get a model back where we can see the results what we're going to be doing is we're going to be making a pretty simple script to essentially ask Claude 3.5 sonnet to generate these dynamically generated svgs for us and once we have these svgs we're going to put them in this svgs folder and then we can go through it and we can decide okay what are good svgs we can do a little bit of human intervention at this point the way that I set this up is say if there's an icon that you don't like that you don't want to fine-tune on you can just go ahead and delete that SVG here so you can just do a quick click through on all the different icons and assuming they're okay you can go and run the second script by the end of it so one caveat with this is a huge space right now there are definitely a ton of different approaches for this this is a relatively rudimentary way on how to generate some Dynamic svgs you can definitely get a more sophisticated approach to do this but this is really more for example sake the first thing that we're going to be doing is we're going to be importing a bunch of modules we're going to be importing the path and Fs to be able to interact with our file system but in this case I'm going to be leveraging Port key which essentially you could use an open AI or an anthropic as a substitute here essentially what port key does it does a handful of things but all I'm doing in this example is I'm just leveraging that because that has my AWS Bedrock API Keys all set up within the platform when I go project to project I don't have to go ahead and set up all those environment variables once that's set up we're going to set up a simple interface and then from there we're going to be setting up our configuration object so here we just have some examples of queries for svgs we have some configuration outputs the number of svgs that you want to generate so you can say 10 or 200 or whatever it is you can put in your retries here you can put in the delay and then things like the maximum tokens as well as the generation delay now finally because I'm using Port key I also have this thing called a virtual key which you don't necessarily need to do if you are going to be swapping out Port key for say the open AI SDK or anthropic or what have you and then from there what we're going to do we're just going to initialize Port ke and then once Port key is initialized the first thing that we're going to do is we're going to Define some arrays and within each of these arrays are going to be how we generate some Dynamic queries for our prompt which you'll see in just a moment here next what we're going to do is we're going to pick one of these words from each of these arrays and then we're going to concatenate them within a sentence here where it says create style of an spvg in this color shape related to this theme once we have that what we're going to do is we're going to generate an idea prompt and this is going to be the prompt that we use to generate a new idea and the the idea with this is hopefully that this is going to be generating something unique each time that's why we have a bunch of different arrays with a bunch of different items we have that Randomness generator where it's just picking at random different parts of those and then in addition to that we're also going to be passing in all of the use prompts that we've used just to ensure that we have as many unique ideas as possible for this synthetic data that we're generating once we have that we're going to generate a prompt for the SVG code generation itself we're going to be really descriptive and we're going to say generate an SVG code with the following description the SVG should be simple clean Etc so on and so forth once we have that we're going to have a simple sanitize file name function and all that this is doing is we're taking the prompt and then we're putting essentially The Prompt as well as an underscore wherever the spaces are and then of course it's going to end in SVG and why we're doing this is because we have a second script that we're going to run and this allows us to have a really simple way where we can just go through this folder quickly and delete the ones that we don't want and then at the end we can just generate what's going to be a Json L file that we're going to upload to open AI to ultimately fine-tune so we have this function name generate SVG with gp4 so this can be really whatever you want so it can be anthropic Sonic 3.5 but say if there's a model that does particularly well on another task like if you want to use llama 3.1b as well as their 400 B model for a particular task you can definitely generate good synthetic data from there as well so in this case what we're going to be doing is we're going to be setting up the retries just to make sure if we have any errors we're going to retry that attempt for our query what we're doing within this function this is going to be where we actually pass in that prompt to what you can think of it as the Endo for anthropic in this case I'm using AWS Bedrock cuz they host to sonit 3.5 but again you could swap out this portion for or whether it's open AI you could use Lang chain you could use llama index you could use the open AI SDK essentially whatever you want to generate all of those different svgs assuming we have a successful generation we're going to log out that it's completed otherwise it's going to Loop through and retry that query again from there we're going to look for that stop token and depending on the llm that you're using this stop token can be a little bit different the Finish reason can vary and if you're swapping out a different SDK just be mindful that that you might have to tweak this so you can just console log right above here if you need to do that and otherwise we have a little bit of air handling and the retry attempts if it does fail we have a simple helper function that's going to validate that we have an SVG it's a very simple validation all that we're going to be doing within the validation is we're going to check to see whether that string starts with SVG finally this is going to be our main function as well as the invocation of our main function within here first we're going to destructure a few different pieces from our configuration object that we have at the top then we're going to check to see if we have that directory of svgs if we don't have that directory we're going to create it we're going to console log out the generation output then from there we're going to create an array where we're going to store all of our used prompts which we're ultimately going to be feeding into that initial prompt to the llm we can keep track of all the prompts that we've already used then from there we have a simple Loop and what we're going to do within the loop is first we're going to generate a new idea once we have that new idea we're going to push it within the used prompts array that we had just declared from there we're going to log out that we've processed this item with this idea this is going to be where we pass it to the llm to generate that SVG and then from there we're going to pass that new idea to an llm and then we're going to wait for that result back once we have that result back we're going to validate that result is valid again we're going to check whether it starts with that SBG opening tag and then if it does what we're going to do is we're going to sanitize that file name so we're going to create these nice file names with underscores instead of spaces then we're going to join it together and then we're finally going to write out the SVG otherwise if we have an invalid output we're just going to skip that and say that it's an invalid or null SVG if you're using say a llm that does have a rate limit like maybe you're on a free tier and using something like grock or something just be mindful of the rate limits that you set here if you need them otherwise you can really dial this down but that's pretty much it and then from there what you can do is you can just open up your terminal the one thing to note is you will have to install Port key if you're using Port key otherwise you can also just install the open a SDK or the anthropic SDK or Lang chain or whatever you might be using once you've installed the port key. a if you're using portkey what you can do is I can run the script and then you can see here that it's running through and then we should see that the folder on the left hand side here is it starts to generate icons that it will actually save out that icon into that folder and we just saw that list got a little bigger here so I'm just going to stop the script there because I already have a bunch of different examples here then what we're going to be doing is we're going to convert the SPG folder everything that's in there so you can clean it up if you want and we're going to create a Json file which is going to be what we use to create that finetuning payload that we're going to upload into the open AI playground there so this is a really simple node script all that this is doing is is generating the Json l so we're going to be looping through that SVG folder then we're going to be parsing the title of the file name here we're going to be breaking that up and then here is going to be how we generate the message for the Json l so we're not actually sending this in anywhere just creating that object schema here it's going to have that system message where we're going to say you are an AI specialized in generating svgs we're going to put in the description which is going to be all of the words that we have within the file name here and then we're going to be passing in the contents of the SVG as well and then once we run that script we'll have something that looks like this now one thing with Json L is it is a very strict format similar to Json but it's a little bit different you have to follow this format exactly you can't have extra lines you can't have commas between the objects or anything like that it has to be essentially just like this so if you are tweaking within this file I'd encourage you to run this through a Json L validator you can just Google for one and you can see if your Json is valid otherwise once you upload that to open AI if it doesn't actually start fine-tuning it's likely a result of it not being able to correctly parse your Json L file within here we have our system message Mage and the content is going to be the same across the board then we have our unique messages for the user message and then we're going to have the content what we're doing in the fine-tuning process is to try and teach that llm how we want it to respond back to us so in this case we want it to be really good at responding with svgs and what have you but this could be used for a ton of different use cases you might have a business application or you need to fine-tune it for a particular brand and have it respond in a certain way so once we have that we're going to hop on over to the open AI playground and we're going to get started with the actual fine tuning all right so for fine-tuning you can head on over to platform. open.com sfind tune and then if you want to see if you have a gbd4 available you can simply click create here and then you can select whether you have that base model there so here you see that we have a gb4 o but you can also find tune the GPT 3.5 turbo models if you'd like and there's even the old older Da Vinci and babage models as well if you're interested in playing around with those from there it's really straightforward you can just upload the file there and then we can add a prefix if you'd like so we can just say svgs and then once you have that all set up and you have your model selected you can go ahead and click create the first thing that it's going to do is it's going to check that Json L file and it's going to make sure that file is going to be able to be used for the fine-tuning job then assuming that's all good to go it's going to proceed and actually start fine-tuning the model for you now if it does fail you will get an email letting you know that the fine-tuning job failed and then you can go back and double check your Json L file or whatever the issue might be so once the fine tuning started there's a ton of nice stuff in here you can see the epox you can see the bash size you can see the LR multiplier stuff like that you can also see the seed and then of course you can see the base model when you started creating it and then the other nice thing is you'll be able to see the time of completion so once it started to run the job a little bit you will be able to see the estimated time of completion here as well and then as it's going through you're going to be able to see the training loss the training loss you can read into it a little bit more if you'd like but you'd ideally like to see that sloped curve just gradually go down over time to ensure that model is doing what we intended to do here we see that we do have that training loss begin to appear and it's just going to start to stream back as it's running through the job for you so we also see now we have our estimated finish time and we see that we started it at 8:15 and it's already going to finish at 819 now one thing to know with this is I just had 18 rows or something right now you're going to need a ton more data if you want really good results if you're going to be doing this seriously you're definitely going to want hundreds of rows if not thousands of rows of good data and responses that you're going to feed into your fine-tuning session but the one thing to note is you do have up to two training tokens a day for free through September 23rd so if you're early on trying this out you can definitely experiment a ton with this get comfortable with the process start to flesh out some new ideas you can just try this out for the next couple months and not have to pay for too many tokens to actually fine-tune it now once the model is complete what you're going to have here is an output model so you can copy this and you can paste that string directly where you would put the string for GP 4 o mini or GG 3.5 turbo or whatever the model that you're using if you're using the open AIS SDK you can plug that in directly where your model goes you'll still obviously need your API key and all of that and that's going to be how you reference it and then the other nice thing with this is you'll be able to see all of your different models listed out within the playground and you can also play with them within the playground here you can go over to the playground and then you can select all your fine-tune models here you'll see them all within a list here here and you don't even need to necessarily set up a little app to test them out but that's pretty much it so while that's running through what I'm going to do is I'm just going to grab one of these generated messages that we got I just want to show you what this can potentially do here so here I have the styliz compass here and if I go over to our Json L file and I just search for Compass this will just give you an idea now ideally you'd have a ton of these queries here and you don't necessarily need to take the query verbatim obviously for this to work but in this case if I just take this over for example sake and I scroll down to my fine tune gb40 mini where I had put in that string like I just described and I paste in that prompt that we sent to the llm we hopefully will see something similar to that stylized Compass so if I go back to the stylized Compass we have something within this real here we see that it's generating and then there we go we have a pretty good representation of that stylized Compass so it looks more like this one to me and it looks pretty close right so this is just to give you an overall idea now if you put something in that was a little bit different it wouldn't quite necessarily render like this but it's just to give you an idea this is obviously just training with a handful of examples but you can imagine as you're putting in hundreds or thousands of examples that you can really improve the base model that you're fine tuning on that's it for this video I'll also do a tutorial on likely fine-tuning something like llama 3.1 in the near future so if you're interested in seeing that or fine-tuning any other models just let me know in the comments below but otherwise that's it for this one if you found this video useful please like comment share and subscribe otherwise until the next one