Meta AI's Code Llama 70B in 7 Minutes

--- type: transcript date: 2024-01-31 youtube_id: tHEug5blLwY --- # Transcript: Meta AI's Code Llama 70B in 7 Minutes in this video I'm going to walk you through getting started with Cod llama 70b the latest release from medic what's exciting about code llama 70b is its performance across a number of benchmarks first I'm going to go into some details on the model itself and then by the end of the video I'm going to be showing you a handful of ways on how you can access code llama 70p there's one tool in particular I'm excited to show you which integrates seamlessly with vs code so as you might expect this is the largest and best performing model in the code llama family and there's three ways you can get started with code llama so there's the foundational model there's a Code llama version specifically for Python and then there's a Code llama instruct model which is fine- tuned to understand natural language instructions so if you're looking to integrate this into a chat GPT like tool that would be one that you'd consider now for code llama specifically this is going to be one where you'd integrate it directly with in something like a code editor like the tool that I'm going to show you by the end of the video so just like all the other models that meta has released code llama is free for both research and commercial use and codee llama is built on top of llama 2 it can generate code both from natural language so where you actually instruct it like they describ here write me a function that outputs the Fibonacci sequence or you can use it with code completion across a number of different programming languages as it stands today there are four sizes for code llama there's a 7B model a 133b Model A 34b model as well as a 70b model one of the things that stands out for the 70b model is it's trained on twice the amount of tokens as the original 7B 13B and 34b models so if you have a piece of code that's relatively large that you're looking to refactor ask questions about or modify it will be able to input and output 100,000 tokens of context the thing that's going to be interesting with this is how well I can generate and make tweaks to particularly large pieces of code because users can provide so much context from their code bases it will be able to generate more relevant code when you think about coding assistance more broadly obviously it's helpful if you can have support on a particular function but it's going to give you much better results if it has the context of that particular file as well as potentially files that you're importing or leveraging where you're able to have a number number of different files and have relevant pieces of code that you're feeding to the llm this can be helpful in debugging large code bases so maybe instead of reading through hundreds of lines of code you feed it to the llm if it can identify particular bugs or suggestions and you might be able to do a comparison on what is generated and there's some tools that will give you the ability to go line by line and compare and do a diffing the vs code plugin that I'm going to show you that integrates with code llama 70b will be able to show you the differences so here's a bit of a lineage chart on how the different models were further trained or find tuned we can see in the python case that it was further trained on 100 billion tokens and then further fine tune on Long contacts of 20 billion tokens for the code llama models it was further fine-tuned for that longer context and then finally for the instruct model it was fine-tuned on 5 billion tokens of instructions to evaluate code llama's performance they use two benchmarks I'm going to focus mainly on the human ofal metric but they also plotted the mostly basic Python Programming metric or the mbpp for the python Developers for code llama instruct it does perform GPT 4's reported metric code llama does outperform all of the other stated models they don't have a gp4 metric for the mvpp model or the multilingual human eval model the code llama model doesn't quite reach those gbd4 levels but it does outperform in GPD 3.5 so it's pretty amazing that we now have these open source models that are comparable to these closed Source models by putting these open source models out there this is going to drive down the cost of some of these models like GPT 3.5 even further I think if you're looking for more information you can go ahead and look into the code llama research paper here where you can go ahead and see some of the specifics on how this model was made one of the easiest ways to get started is to head on over to labs. perplexity doai you can select the lamama 2 Sub model from the drop down in the right hand corner here there's also a handful of models here like you can see on screen and so we can just put in our prompt here so I'll just say generate me a helicopter game in a single HTML file write the full HTML CSS and JavaScript so I'll just go ahead paste that in here and we have that chat GPT like experience it's writing that CS s for us and then I assume it's going to go ahead and continue on with our JavaScript like we see here it's a pretty long piece of code that it generated for me from a basic line of text here if you're looking for a copy and paste solution this is an option for you the other nice thing with perplexity Labs is they do have this model also available from an API so say if you want to use this model but you don't want to run it locally you can go ahead and use a service like this and integrate it within your projects one thing to note with perplexity is if you are a pro member you do also get $5 worth of free credits every single month which is really great for hobbyists if you're looking to experiment with some of these models another great option is together if you haven't used them before they'll give you $25 worth of free credits at least at time of recording you can do a ton of stuff on their platform from fine-tuning Models to using them for inference and what you can do with this is you can have serverless endpoints that are going to be build per token if you want to go in here and try a number of different models see which one might be best for your use case and you can easily scale with a service alike together if I just go ahead and put in a prompt similar to what I had let's say generate a snake game in a single HTML file we see that it's going ahead and it's not giving me quite all of that HTML file it's giving me some of the elements of the JavaScript by the looks of it you can see here that it's limiting the response to 512 tokens where you can just dial that up with in the parameters here on the left hand side now I did notice the outut length does only go to 2048 within the GOI I would assume that the inference API would view a larger context window as this model can handle a significantly larger context the other nice thing is you can just go ahead within their playground click this button here and get the code for python JavaScript or the curl request where you can go ahead and plug this within your project now the last option I wanted to show you is an open source autopilot for your IDE continue is a cross between cursor which is an AI first code editor and GitHub co-pilot so I have videos on both of those which you can check out if you're interested now I'm not going to be doing a deep dive on continue within this video but if you get started with continue you could simply go to vs code go to the extension Marketplace or their website and you can go ahead and install it there here you're able to see that you have code llama gbd4 as well as a number of different models that you can interact with so as you see in their example here you can go ahead and select a piece of code you can ask questions about it or you can ask it to be tweaked and if I just demonstrated here so I just copied over that helicopter game from perplexity labs and if I just go ahead and highlight let's say some of the Cs you have the option that pops up here so it says command M to select code or command shift L to edit so let's say I'll edit and I'll say make this more colorful and you can see that line by line is going to go ahead and add in some different colors so it's giving a new background color it's giving you the color of the foreground you can go ahead and click the accept or reject buttons or you can go ahead and accept them all so here there's a number of different options so there's code Lama 70b there's gp4 Vision there's gp4 once you select a piece of codes you have the option to put it within the context window of the chat view here or you can go ahead and specify the edits that you want to make right in line so if I command shift L I can write what I want to have happen and I can say simplify this you can see line by line it's going to go ahead and make a recommendation on what you can do so in this case it's saying maybe I can simplify by removing this class name and you can go ahead and accept it so you can command shift enter or you can reject I just wanted to give you a quick demonstration on continue and you can play around with it further if you'd like leave a comment if you found it useful that's pretty much it for this video if you found this video useful please like comment share and subscribe and otherwise until the next one