Vertex AI 429 RESOURCE_EXHAUSTED: what it means and how to fix it

What it means

The number of requests exceeds the capacity allocated to process them. Depending on the error message, it is caused by API quota, shared server capacity, or the daily limit for logprobs requests.

Causes

• API quota over the limit, for example exceeding the requests per minute limit
• Server overload due to shared server capacity
• Reaching the daily limit for requests using logprobs
• Acceleration limits after a sharp increase in usage

How to fix it

1. 1
   Fix 1.

   Check the generative AI quota limits and apply for a higher quota (Quota Increase Request) if needed.

2. 2
   Fix 2.

   Retry after a few seconds using truncated exponential backoff; the API errors page recommends retrying no more than two times with a minimum delay of one second. Contact support if the error persists for hours.

3. 3
   Fix 3.

   Use the global endpoint instead of a regional endpoint whenever possible.

4. 4
   Fix 4.

   Avoid traffic spikes and ramp up traffic gradually; on Standard pay-as-you-go, smoothing traffic helps.

5. 5
   Fix 5.

   Consider purchasing Provisioned Throughput for a more consistent level of service.

6. 6
   Fix 6.

   The Gen AI SDK retries 429 and 5xx automatically; retry behavior can be configured with HttpRetryOptions:

   Snippet

   Copy

   from google import genai
   from google.genai import types
   
   client = genai.Client(
       vertexai=True,
       project=PROJECT_ID,
       location="global",
       http_options=types.HttpOptions(
           retry_options=types.HttpRetryOptions(
               initial_delay=1.0,
               attempts=5,
               http_status_codes=[408, 429, 500, 502, 503, 504],
           ),
           timeout=120 * 1000,
       ),
   )