Model Cascade

In depth

A cost-saving setup where a request goes to a cheap model first and moves up to stronger, more expensive models only when the answer fails a quality or confidence check. The FrugalGPT paper (Chen, Zaharia and Zou, 2023) formalized the LLM cascade and reported matching GPT-4 performance at up to 98% lower cost in its experiments. A router picks one model up front, while a cascade tries models in order and pays extra only on the hard cases.

Primary source: arxiv.org