Ollama Function Calling: LangChain & Llama 3.1 🦙

--- type: transcript date: 2024-08-01 youtube_id: H1fwL7TuEtc --- # Transcript: Ollama Function Calling: LangChain & Llama 3.1 🦙 we now have tool support within olama and what this allows you to do is have function calling locally on your machine now AMA is typically associated with running llms locally but you can also deploy AMA to the cloud if you'd like there are a number of really great benefits on having or running something like a llama because all of a sudden you have this tool where you can just pull down the model that you want to use and then you'll be able to interact now with this function calling capability one thing to note is this is supported on a handful of models right now but obviously over time this list is going to grow in this example I'm going to be leveraging llama 3.1 I'd also encourage you to check out the new model from Gro which is best in class for the function calling capability and there is even an open- Source version of that that you can find and pull down on the AMA website here you can just go ahead and AMA run and then pass in the string of the model that you want to have here if you don't have aama installed it's really easy to get set up you can download it and then as soon as you find a model that you want to use all that you need to do is AMA run and then pass in the name of the model with in your terminal so in the example of llama 3.1 you can just type O llama run llama 3.1 and the first time that it's going to run it it's going to pull down that model and then once it's on your machine you're going to be able to leverage that locally to be able able to have that as a model that you can use to interact with for chat applications or for function calling in this example I'm going to be leveraging bun Lang chain as well as typescript and then of course we're going to be leveraging AMA and then AMA 3.1 I'm going to show you what this does of the bat and then I'm going to run through quickly the coding portion you understand on how to implement this yourself I'm also going to put this within a GitHub repository that you're going to be able to pull down if you're interested in doing that and playing around with this example within this example we're going to have three different functions we're going to be asking what's the current weather in San Francisco can you open the calculator app lastly can you open up Chrome and ask CLA what is the capital of France if I just go and Trigger this what it will do is it's going to look through our query and then it's going to look to see if there are any matches for the functions that we have passed in so how function calling works is we're going to map the functions that we have within our application with natural language and then those are also going to be passed in to the context of the llm and that natural language query that we have it's going to have a bit of a different schema it's not going to be within the same place of where we put the prompt or the messages there's going to be a particular array on how we pass in all of the different functions that we have here we see that we do have a response back from cloud we also had the calculator app open and then if I just look within the terminal here we also have the results of the temperature which presumably would be from San Francisco to actually get the temperature we're using a no API Key weather endpoint and what it accepts is a coordinate in this case we're even asking the llm to get the coordinate of the particular City that we're asking for this is just to give you an idea on some of the different things that can leverage function calling for I'm going to run through this relatively quickly what we're going to be doing is we're going to be importing tools as well as chatama from Lang chain once we have that we're going to have a Zod and that's going to be how we validate the schema for the different types that we need within our function calling arguments and then from there we're going to be using the built-in child process Library which is built-in within node.js as well as bind and what the child process allows us to do is to essentially execute commands as if we were using the terminal directly from nodejs for our weather Tool It's relatively simple like I mentioned we're going to be using a free no API key end point so you don't need to worry about getting an API key for anything at all within this application and all that our function steps is the latitude as well as the longitude that we're going to be asking for from the llm in our requests once we have that we're going to be mapping this with natural language and you'll see that we do have this wrapped within the tool and that we do have it wrapped within this tool method here that we're getting from Lang chain the way that the tool method works is you can put the function that you're using directly within it just like this and then immediately adjacent to it you'll have that mapping of natural language of what that function does and then we're also going to have that schema this is going to be where we use that Zod validation library to say okay we want a number and then we're going to describe that type with natural language this is going to be how the llm knows exactly what we need for arguments as well as what the actual function does itself now for the functions where we're going to be using our child process it's going to be a little bit different depending on the system that we're using we do have a switch case here where all that we're doing within this switch case here is we're going to be passing mapping the particular command depending on the operating system that you're using for instance to open up a calculator on Windows is going to be different on Linux and then it's also going to be different on Mac or Darwin in this case if you see Darwin that's going to be the Mac OS implementation then from there we're just going to console log and then this is going to be how we execute the child process we're going to be passing in our Command if we have any errors we're going to log those out alternatively we're just going to say that the calculator has successfully opened when this case we're just going to pass in an empty object since we don't actually have any arguments that we're passing in just for this example but and then very similar thing for the Chrome tool a similar switch case to open Chrome depending on the system that you're using and then we're going to be using this encode URI component and what that allows us to do is instead of just passing in the query and it trying to pass that within our child process if we did that we'd get an object that is sending to the query Within CLA dot but by encoding that URI that's going to be how we're going to be able to send in longer strings where it doesn't have to worry about special characters and spaces and all of that you don't need to worry about it too much but that's just to give you an idea on what's going on there similar thing here we're going to follow pretty much the exact same pattern to send in that command to our terminal finally we're going to describe what the function does and then we're going to say the one argument that we have is going to be a string and then we're going to describe it for the llm here and we're just going to say the query to ask CLA Ai and then from there this is going to be where you can Define your models so once again you can just AMA run and then put in the model name you can just go on their site and search for the particular model that you like and then you can put it within the string here so now that we have all our tools declared and also we have the model declared that we're going to be using we're going to bind all of those different tools or function calls to our model and then once we have that we're ready to use our model before I dive into what's happening here I just want to first touch on what what the results log as when we get results back for the tool calls what they're going to look like is we're going to have this array of tool calls and then it's going to show the tool that we're going to use as well as the arguments that we're going to pass within the tool here you'll see the number of different tool calls here it could be one it could be two it could be three it all really depends on how many are being invoked and how accurately it picks up on all of the different things that you have within your request all that we need to do is if there are any tools we're going to Loop through all of the different tools once we've looped through them what we're going to do is we're going to have a simple TR catch and then we're going to say okay if that tool call name exists of the different tools that we have we're going to go and call the function and also parse the results from that payload that we initially got from the llm then once we have that we're going to have all of our different tools invoked and that's essentially it what you can definitely build on this if you'd like a common pattern is once looped through and got the results from all of the different function calls the pattern that you'll often see is actually sending these results back into the llm and then having the llm respond in natural language back to you maybe you have an application and you just want to get the particular results and you're not necessarily concerned with always sending that context back directly at least maybe not right away within the context of the llm response to the user but otherwise that's it for this video kudos to the team at for their good work as always otherwise if you found this video useful please comment share and subscribe otherwise until the next one