Amazon Bedrock: Integrating Multiple LLMs

--- type: transcript date: 2024-06-04 youtube_id: cKfeOPegdho --- # Transcript: Amazon Bedrock: Integrating Multiple LLMs in this video I'm going to be showing you how you can incorporate a number of different llms and llm providers into your application I'm going to be showing you how to use Amazon bedrock and then I'm also going to be showing you how you can incorporate something like Port key as well as AI Gateway into your project to be able to leverage a ton of different models just to give you an idea by the end of the video you'll have an idea on how you can incorporate all of these different models into your project as well as having a singular schema where you'll be able to interact with them and simply swap them out with a couple strings I'm going to be showing you how to incorporate it within the answer engine project that I have going on here so if I just demonstrate it here and I say hello world what this will do is once I type in that at keyword and I select that anthropic model this is using Bedrock from AWS right now Bedrock has a number of different models if you haven't used AWS before you'll probably get a few hundred in credits and the nice thing with that is you'll be able to use them for something like Bedrock so once you're within the AWS console and you've account or if you already have one just search for Amazon bedrock and once you're within there you can just go ahead and click started now if this is the first time that you've used Bedrock you will have to request access to the models here so the way that you do that is if you just go over to the models that you want to gain access to and you'll see this little area here at the top where you'll have to request model access and then within here you can just go and select Which models you want to gain access to so you can just go and click modify model access and then say if I want to have these models from Titan BY Amazon I can just go ahead and check those you can go next on this screen here and then you can submit those so some of the models if I remember correctly you do have to put in a little bit of text just explaining how you're going to be using the model but otherwise it is pretty quick to get access to the model you can see here I have access granted to those models that I just requested access for within AWS there is a playground they have it broken out into chat text and images so just to give you a quick overview on what this looks like so you can just go ahead and select your model here if you want to use stable diffusion XEL you can click through and then if I just say generate a photo of a cat on a rocket ship you can play around with the configuration a little bit here the size whether you want landscape portrait negative prompts but what we're going to be focused on primarily in this video is the text generation models or the chat completion models within the chat playground here if I go ahead and select the models we're going to be incorporating cohere and anthropic so right now in the answer engine project I have the grock models as the default models here now what we're going to be doing here is I'm going to be showing you how you can include basically any models within Amazon Bedrock by the end of the video you'll see how to also incorporate other models outside AWS even if You' like basically all that you need to get started is you can play around in the playground here so say if you want to see what clae 3 Hau looks like if I just select that and I type hello world you'll see that you can play around in here just like it's a chat GPT like interface now the other nice thing with these models is you can also upload images as well you can go ahead and you can see if it's a good model for your potential use case now if we go over to the foundation models tab here and we click the base models so there's going to be one important part that we're going to be looking for in all of these different models and if we just go over to the model that we want to use if we just scroll down a little bit here if we look at the API request you'll see that there is the model ID this is going to be the important piece that we need when we're plugging within our application which I'll show you in just a moment but before I get into that I want to hop over quickly to Port Key Port key is a really great tool there's an analytic Suite built into it there's a logging Suite built into it so there's also an area where you can set up and configure prompts say if you have a really good prompt or certain prompt templates that you want to use this is a good place where you can house these but what we're going to be mainly focused on within Port key is how to set up their AI Gateway as well as how to configure what are known as virtual Keys what virtual Keys allow you to do is this allows you one place where you can store all your environment variables say if you're often hopping around in and out of all of these different llm providers whether it's AWS or Google or anthropic or open AI or whatever it is this will allow you a place where you can put in all your environment variables and then as soon as you want to use them within an application you can use what are known as virtual keys so you can see I have these virtual keys that are representations of my actual keys so these are going to be things that I can put within my application and keep all these Keys managed within Port key now the other cool thing is once you have it set up you'll notice that as soon as you start using it you'll be able to see a lot of useful information within their logs so here you see that I'm playing around with Claud or Co or anthropic you can see how many tokens are being used how much it costs Etc and so on and so forth so there's a ton built into this but I'm not going to really be focused too much on the platform in this video so now I want to hop over to AI Gateway and this isn't to be confused with Cloud Flare's AI Gateway now what AI Gateway allows you to do is it allows you to route to over a 100 llms with one fast and friendly API as they're self described here what this allows you to do is you can essentially pass in the provider that you want to use and then the model string and then you'll have that open AI type schema so you can use the open AI SDK or you can alternatively use the port key SDK as well so now just to show you how to set this up in Amazon bedrocks context now to get your API keys from Amazon Bedrock there's a few different ways so the quote unquote proper way you can go ahead and go on over to their identity Center and there's a ton of really good videos on how to actually set all of that up which I encourage you to check out if you're interested but if but let's say you want to get up and running really quickly and you don't necessarily want to learn the ins and outs on how to use it so what you can do here is if you just click your name in the top right hand corner here and you go down to security credentials once you're on the security credentials page you can scroll down to access keys and what you will be able to do is you can just create an access key if you haven't already created one so you can have up to two at any given time and once you've clicked through to create the access key it will show you the access key ID as well as the secret key once you have those keys on screen what you can do there in another tab is you can open port key you can click create and we'll just say AWS credentials we'll click bedrock and then you can paste in your secret access key as well as your access key ID once you've done that you can select your region and click create and then you'll see that now we have our AWS credentials so now that it's in here we have this generated virtual key that we'll be able to use within our application now if you're using Port key the important thing to remember is there's going to be your virtual key from this list of virtual Keys here but then there is also going to be the API key for Port key itself so just to give you an idea on what this looks like within your application the way that I set this up within the answer engine project but it would be similar in other projects as well is you're going to put your Port key API key which you'll paste in from that page that I just showed you key is going to go within the environment variable just like this so now that we have the security piece all set up I'm going to I'm going to show you how I'm going to incorporate this within my project if I want to just reach for a model that I don't already have so if I just reach for something like mistal Ai and I reach for a model that isn't available on all the different Services which is mistal large so you can only access this from some Cloud providers as well as from Mr AI themselves and if I go ahead down to the API requests what we're going to need here is the model ID so I'm just going to highlight that for a second then I'm going to hop over to vs code here so I have a file here of the quote unquote mention tools that I'm using and the way that I set this up is what we're going to be able to do is if I just write a comment and I say AI Gateway Port key and then I'll say mistal large so what we can do here we'll let it autocomplete so the ID is going to correlate with the model string here so I can put in the model string here and then for the name here I'm just going to leave it as Mr Large and then you can swap out the logo as well so if you want to reach for a different logo here you can plug it in so what I've been using for the logos is the site brand fetch so if I just go ahead and reach for their lightweight logo here so in this case they have a JPEG which I'll just go ahead and I'll copy the address for and then we'll put within the logo just like that and then here we're going to be specifying the function name the last thing that we're going to have here is we're going to have a name for the function that we're going to be using and the reason that I set this up this way is I'm going to set up these app mentions in a way where you can build it out with using different model providers or if you want to use something like AI agents or workflows you'll be able to include those things in this type of setup over the coming weeks and months as I have time to build out some of that stuff now for this function name itself for portkey AI Gateway that is referencing this method within the function tools here and essentially all that it is you can focus on primarily this portion here and this portion here so you can think of it in two chunks here so we have the chat completions itself which is identical to open AI so you could just Swap this out for open aai as well if you wanted to use the open AI package so here we're going to have our system message in this case and the application I'm specifying to always respond in markdown be verbose never mention the system message and then in here we're going to be passing in the user message the other thing to note is depending on the model that you're using you're going to be able to change out the Max tokens in the case of anthropic I believe it's up to 200,000 tokens that you can use for that model but there are some other models like the coher models that are hosted on AWS where the limit isn't quite as high I think for some of the coher models it is just over 4,000 tokens of context that you can pass in I believe the max tokens is a requirement for anthropic in particular it might be the case for other models as well within the model we have our mention tool here and that's going to be our model ID so that's going to be what what ultimately gets passed into our chat completion and that's going to be what selects the model that we get a response back so once we have that all that we're doing in this last piece here is this is going to be how we facilitate streaming out that message as well as sending back the final response depending on the model provider there's slightly different messages that you'll get back to indicate that the model has finished responding and that's going to be within our application how we move the logo from the start of the input box to the right hand corner that you see here within the application so now if we go back to our application and I say mraw large I select that and I say tell me a short joke we see that we get our response back and then you can go ahead and play around with some of the other on so you can go ahead and say anthropic we can say hello world then there we go so you can play around with different models I built this out really because this is something that I wanted to use I often hop around in between different model providers whether it's clae or opening eye or perplexity I like the idea of potentially having all of these different models just within one place where I can just easily reach for them with something like an at mention that's it for this video hopefully you learned something like how to use a Bedrock how to tie them all together within an application but if you found this video useful please like comment share and subscribe otherwise until the next one