AnythingLLM: A Private ChatGPT To Chat With Anything

--- type: transcript date: 2023-12-22 youtube_id: or8XrZApfgE --- # Transcript: AnythingLLM: A Private ChatGPT To Chat With Anything in this video I'm going to be showing you anything llm which is a new open- source full stack application that enables you to turn any documents resources or pieces of content into context that you can then feed into an llm and reference during a conversation so it is very much like a chat GPT Al like feel but over your own documents as well as in an environment that you can control and tweak if you like so there is both an open source version as well as a hosted platform I'll be touching on both of them and touching on how you can get started with with either of them so if you want to get started with the open source repository you can just head over to the repo which I'll put in the description of the video and then from there I just wanted to quickly touch on the text stack so I love when I see a project that's built within JavaScript and the thing with this text stack is the front end the server everything is built in an environment that I am very familiar with and I think a lot of people on this channel would be glad to see so it's using vit and react on the front end and then on the back and it's using no. JS Express servers so really great and familiar for the JavaScript developers out there now in terms of the llms there's a ton of support that's already baked into the project so you can obviously use something like open AI it has the ability to integrate Cloud as well and then the other cool thing is it gives you the ability to choose from something like LM Studios or local AI so say if you just want to pull this down and use local models and play around with it you can totally do that so it will give you a nice chatbot interface where you can embed documents and do all of that locally if you're looking to do something like that so say if you want to use something like mistol or llama and just have everything running locally this is a great option so it also gives you the ability to break out the embeddings model as well as the vectors database so just to quickly show you that so this is the local version running on my machine right now so once you get set up it will look something like this now once you go within the settings tab here when you first set this up what you'll have to do is you'll first have to set your llm preference so in this case I just have it set up for open AI but you can set up like I mentioned Claud or a local model if you have either local AI or um Studio set up and then from there you can choose your embedding preference so off the bat they give you this anything llm embed which just runs locally so you don't need to incur additional cost for embeddings we can also select something like open AI if you'd like or a different model from something like local AI so similar uh just like on their embeddings within their Vector database it gives you the option to choose a bunch of different things so uh Lance DB this is the one that will run within the instance locally but you can also select your pine cone instance if you have one or if you want to use an external uh Vector database like uh chroma or we8 or what have you so you'll just have to go in and plug in your endpoints and API keys to get that all set up so there's a couple other nice things within here so they also have some data con connectors right off the bat so they have the GitHub repo data connector where you can just go in plug in a GitHub repo so you can imagine say if you're getting started with a new GitHub project and you want to know what a particular thing is doing you could in theory go in pull down that repo and have a conversation with that GitHub repository so very easy very intuitive and everything is really just built into this gooey so especially if you're going the hosted tier is you don't really need uh coding experience so some of these things obviously you want to set up GitHub it'd be good if you sort ort of have the general Sense on what things are where to find things like API Keys obviously you'll have to set up some pieces but you don't necessarily need to be a fullon developer to set this up so the other cool thing with the project is it also gives you the ability to set it up within a workpace that you can share with other people so say if you're an administrator of sorts and you want to set it up so you have your team all have access to this thing you can set it up in a way where there's the different uh permissions where say you as the administrator uh manage particular things and then you can go ahead and say limit the number of messages uh someone's querying within the chat bot each day so there's a ton of flexibility within this and it's really a great project especially that it's open source that people can contribute to this and just play around with it tweak it so say if you don't like the look of it or if you like the functionality but you want to sort of uh build it to whatever your your company's design is or implement it within a product it really gives you a nice place uh a jumping off point right so if you were to go ahead and sign up for the free tier so they have a 3-day free tier and what it will do is you can just go in you don't need a credit card put in your information and once you've done all of that it will spin up this instance for you so you can choose your custom URL for the subdomain here and then you can just go ahead and uh set it up just similar to what I showed you so it's going to look just like the local version there's no difference between the two and you'll be able to uh swap in things uh just as you would uh if you were to set this up in your own environment so if I just hop back to the GitHub repository here is there are also some uh deploy options here that are built right into the GitHub page so I can't speak to if all of these work or how well they work or if there's any bugs that you might run into but presumably these things are great right so it gives you the options if you're used to using uh Docker or AWS you can just go ahead and deploy it that way uh it will also show you on their documentation page the specifications that are recommended to run this on something like say an aws's case on like an ec2 instance and and all the resources that you'll need so in terms of the pricing so if you want to continue on with your self-hosted tier these are the pricing tiers right now so it starts at $25 a month uh it gives you 4 gigs of storage if you want to go up from there it's $100 a month now also obviously if you want to just play around with this locally or deploy it on your own instance you'll be able to control cost that way so in addition to the guy is they is also an API so say if you actually want to just build out the whole front end portion and just leverage the back end of this because it's broken out that way you could just take that Express server and then have your chatbot query these endpoints so you it's pretty full feature there's quite a bit in here I haven't actually tested out the API quite yet but it does look like there is a lot of control that you have through the API here so now in terms of actually getting started with it so you can make a new workspace so if I just say let say test what you'll be able to do is you can either switch it from query mode or chat mode but within here you can actually go ahead and drag different documents so let's say I want to put in a few new documents I'll just grab in a few PDFs that I have on my other screen here and these are regulatory filings from Apple so I think I have their annual report I have an Insider filing and I might have something else so you can submit documents uh there's a handful of documents of types you can submit you can also submit a link if you'd like it to scrape the website and then as soon as you upload them it's not going to go ahead and automatically embed them you'll actually have to go ahead and select them uh here and then you can embed them from there so let's say if I just want to go ahead and move these over to my workspace you'll be able to go ahead and save and embed from there so the other thing with this that's really nice I'll just click save and embed but if you were to have selected something like the open AI embedding zempo it would give you an approximate cost adjacent to that save and Ed embed button so you'll get a sense on whether this you know if it costs like a few cents or how much it would cost if say you have like a really big host of documents that you're plugging in there it will give you sort of a general Sense on how much it's going to cost to just embed them so once you have that all set up you can just go ahead and ask a question you can say how we apples financials I'm not we'll just get rid of that okay and let's say describe in detail how well Apple performed financially so the nice thing with this is it gives you a really easy way where you can go ahead and swap out different embedding models those different llms uh and really get a sense on what works best for your use case so it's one of those things where sometimes it is difficult to judge how well an llm is performing and honestly some of the best uh testing is just practically trying it out so imagining uh what your users would be putting in what your uh workplace would be putting in all of those types of things this gives you an environment where you can just with a couple clicks and fetch an API key you can swap out like how does anthropic work how does open AI work how does this embeddings endpoint work or how does this Vector database handle things so it gives you a lot of flexibility uh in terms of experimenting now in terms of the uh another feature that I thought was really great is it also gives citations so when it returns responses to you like you saw on the screen here what it's doing is it's fetching text chunks from the documents that you had uploaded so this will give you those text chunks and what it's referencing to actually generate the response uh back so here we saw that 25 billion number and then we see that 25 billion number within the answer here so really nice implementation kudos to the mplex team as well as the open source contributors here I encourage you to check out the project give it a star a fork it uh add to it I believe they're open to contributions so that's pretty much it for this one so if you found this video useful please like comment share and subscribe and otherwise until the next one