Cohere API 400 Bad Request (too many tokens): what it means and how to fix it

What it means

A 400 Bad Request whose documented example messages begin with "too many tokens", sent when the prompt or requested output exceeds the model's token limits.

Causes

• Prompt exceeds the model's token limit ("too many tokens: total number of tokens in the prompt cannot exceed 4081 - received 4292.")
• max tokens above the model's maximum output ("too many tokens: max tokens must be less than or equal to 4096, the maximum output for this model - received 8192.")
• Inputs exceed the size limit ("too many tokens: size limit exceeded by 11326 tokens.")

How to fix it

1. 1
   Fix 1.

   Try using a shorter prompt, or enabling prompt truncating (as stated in the documented error message).

2. 2
   Fix 2.

   Try using shorter or fewer inputs, or setting prompt truncation to AUTO (as stated in the documented error message).

   TOML

   Copy

   prompt_truncation='AUTO'

3. 3
   Fix 3.

   Lower max tokens to at most the maximum output for the model.