Groq: Accelerating LLM Processing with Unrivaled Speed

--- type: transcript date: 2024-02-20 youtube_id: j8Vm2KLPc9U --- # Transcript: Groq: Accelerating LLM Processing with Unrivaled Speed there's yet another thing to be excited about within the world of AI grock has been making waves on social media with its impressive performance of over 500 tokens per second challenging the status quo of AI processing with its large Processing Unit or lpu what's different with Gro is this is not just a software company they are actually building this Hardware there was a recent interview with the CEO Jonathan Ross where he talked about how this will also push the satisfaction rate of applications forward if compare this to the web there's a ton of different information about the load times of websites if it's faster they're going to have a higher satisfaction rate the same is going to apply for llms and apps within the llm space if you want fast inference there is no one even close to grock grock's Hardware is a custom design application specific integrated circuit or Asic that specializes in running large language models the lpu inference engine is capable of generating around 500 tokens per second which is a significant leap from even some of the fastest providers one thing to not with the models that it's running on such as mixol and llama 270b these are decently sized language models to get an output like this Jonathan Ross is the CEO of Gro he is a rich background in Ai and processing technology having initiated Google's tensor Processing Unit or TPU project his experience really has been instrumental both at Google and now at grock grock is looking to set a new era in computational speed and efficiency when it comes to llms So currently grock is a B2B business so they're going to be selling these CHS to businesses but right now you can head over to their website gro.com grocki AQ and try out their inference you can also sign up for a trial of their API if you're looking to trial the API sign up on a wait list to try this out grock's lpu is obviously a Marvel it has not been without criticism some have expressed concerns about the potential higher costs associated with acquiring and implementing this new architecture the one of the unique features of grock's innovative chip design is it allows for multiple tsp to be linked together without traditional bottlenecks found in GPU clusters this will make scaling applications much simpler now if we just look at the chart here of all of these different providers and this is not to knock any of these providers just to show you a comparison straight to Gro these platforms that really focus on these open source models are at The Cutting Edge of inference now it goes without saying it's going to be very expensive for companies to acquire these lpus from Gro especially right now but the thing with technology is now that people know that this inference speed is possible is they're going to start to this from the different applications that the apps and websites that integrate these lpus from Gro are going to have a significantly better user experience the companies that are willing to invest in these new chips and this new inference API could potentially create a bit of a flywheel of a user effect where users see how fast their application is by using this new architecture and they're going to begin to demand this across other providers one thing that's yet to be seen is how other chip providers like will respond to something like this if you think about a company like Nvidia what are their plans for something like this and then also if you think about where this will be in just a number of years with this advancement of the tokens per second as well as Google gemis announcement you're going to be able to pass in up to a million tokens of contexts with accuracy into their Gemini model with the combination of the faster inference speed and the larger context window of these models it's going to be really interesting to see the types of applications that come out of this we're going to have a lot more real-time applications things are going to be faster they're going to be more accurate and over time the cost of all of these things are going to be continually driven down so right now if you want API access you have to sign up for a form on their website this is the world's fastest INF speed for open source llms they have the Llama 270b model and 7B model as well as the Mixr model from mistol AI with more models coming soon they mentioned that there's a 10-day free trial with a million free tokens once you gain access the API is fully compatible for simple switching from the opening API now there are some terms in terms of rate limit and you're not able to do load testing groc is providing General early access to the alpha release of the API free of charge for a limited time for only research and development purposes based on demand they mentioned that at their discretion they may limit access to 7,000 tokens a minute or 350,000 tokens a day they're really throwing down the gauntlet and saying hey if you want up to 750 tokens per second with the Llama 27b model you're going to be able to get that for 10 cents per million token or for these larger models you're going to be able to get it for as low as 27 cents at the 480 tokens per second so this guarantee ensures that things are going to be interesting over the coming weeks and months as they start to roll out access to this if you want more information they have a robust site they have a number of different blog posts on their website they also have a YouTube channel with content on this I just really wanted to point you in the direction of grock it's really interesting to think about all the different use cases and applications that can be built when you have such a high throughput for the output tokens that are generated so I'm curious your thoughts on the type of applications where you think something like this would be useful leave a comment in the description below if you found this video useful please comment share and subscribe otherwise until the next one