Jan: Bring AI to your Desktop With 100% Offline AI

--- type: transcript date: 2024-01-03 youtube_id: QpMQgJL4AZA --- # Transcript: Jan: Bring AI to your Desktop With 100% Offline AI in this video I'm going to be showing you Jan which is an open source chat jpt alternative that runs 100% offline on your computer now the nice thing with Jan is there's support for Windows Mac OS and Linux and to get started all you have to do is simply download the installer so the nice thing with this is say if you're not as comfortable with pasting in scripts within your terminal and potentially having to debug certain things that you might be not comfortable doing especially if you're not a programmer you don't really need to worry about that with this type of implementation so if you want to get started with these new open source models like the Mist models or llama models but you're not a coder this is definitely a really good approach for you so once you have it all downloaded and installed you'll have an interface like this so it's built within electron so say if you're a typescript developer or a JavaScript developer and you're familiar with web technology you might feel comfortable to contribute to this project so the nice thing with this is right within here there are a number of models that are built right within the gooey here so all you have to do is really just scroll down find the model that you want to download and click the button and you'll be up and running in no time so I have a relatively fast internet connection so my models will download relatively quickly to my machine but nevertheless once you have them downloaded you only need to do that once until you actually uninstall them so I have two models installed I have the tiny llama chat model as well as at the top there you saw the mistal instruct model so once you have at least one model in insted you can go ahead and go to the thread tab on the left hand side here and you can select the model from the right hand side here so here you see the two models that I have downloaded locally and the other nice thing that is implemented within the project is there is the ability to interact with the open AI API directly if you'd like to do that so say if you're experimenting with some of these open source models and you're trying to see if a particular use case can be solved by some of them and you run into some barriers and you just want to maybe compare it to something like gp4 you can go ahead and just select gp4 run your query and you can run the comparison right within the same thread there so that's the other nice thing with this project is you can have a conversation with one model and then you can swap out to another model and that thread will be all within uh the interface here while you're toggling back and forth between the models so there is the ability to add in your assistant message or your system message here for the models so if you want to put a message that is weighted a bit higher now how well this works on some of these open source models is sort of TBD I haven't done extensive testing on this but just as something to note that it's there so it has a very chat gbt like feel so if I just say write a short story I'll show you what it sort of looks like so about a person who has to stop time so this is a very small model I haven't actually use this one too much but let's say I want to use the mystal model let's say write a short story let's see how well this performs now you'll notice that when you actually swap out the models it does take a moment to actually start that inference server so well it is pretty quick to swap in between them when you're actually swapping between models it might just take a little bit to actually get that all instantiated and running when you're hopping between one and another so the other thing to note is I'm on a bit of an older machine so I'm on an Intel base Mac so I'm on sort of the last generation of Intel base mags before they started to really just run with their M series chips so if you're on at least an M1 chip you are going to get considerably faster results than you see on my screen here so you see my token speed response is only 2.5 seconds from this model but mind you I'm screen recording right now and like I mentioned it is an older mag so generally speaking if you're on a machine within the past few years you are going to get considerably better results than what you're seeing right now on my machine so the other nice thing here is you do see the CPU usage as well as the memory usage in the right hand side here and you can also select that from the system monitor here if you want to see it a bit more clearly and you'll be able to see that the CPU is running almost at 100% I can hear in background here that the fans on my computer are starting so it is really trying to use all the resources to get the responses back uh from my query here so other things with the this Jan project that are pretty neat is you can also import models manually from hugging face so I'm not going to be running through that in this video but just know that that is an option so there are a lot of different uh compatible models on hugging face that you can just go ahead and explore and plug in and use if you like and the other thing with uh Jen that I thought was great is if you go over to jan. you can check out their road map here and there's also some interesting plans coming up so they're planning a mobile app which I think is a really great use case I've actually looked at this a couple months ago to see what options are out there for local models running on something like an iPhone 15 and I didn't really see that many options I think I found one and it wasn't really that great of an implementation so it will be great to have other players come into this space where you can have these local models directly on something like say an iPhone and the other nice thing with the Jan project is there is going to be support for being able to use this as an endpoint API so say you have a local application that you want to use and you want to use some of these local models and you want to expose that inference API there is a road map here and you can sort of see the different statuses uh on their homepage here on how these things have been implemented so uh eventually it will be like a full-fledged um project where you'll be able to interact with it from the guey or interact from it uh with a API you'll have a lot of different options by the looks of it so if you really want to get into their road map here you can check it out on GitHub and you can see what's going to be coming out when what they're planning on actually implementing Within the project within this epic here that's pretty much it for this video if you found this video useful please like comment share and subscribe and otherwise until the next one