Azure OpenAI 429 Too Many Requests (Rate limit is exceeded): what it means and how to fix it

What it means

Requests exceeded the TPM or RPM rate limit for the deployment's allocated quota. Messages include "Requests to ... have been limited" or "Rate limit is exceeded". TPM is based on an estimate that includes prompt text, max_tokens and best_of; RPM is evaluated over short windows of 1 or 10 seconds.

Causes

• TPM or RPM limit of the deployment reached
• Bursts of requests within a 1 to 10 second window even when the per-minute total is within limits
• Quota approved at subscription level but not allocated to the deployment receiving traffic

How to fix it

1. 1
   Fix 1.

   Follow the retry-after-ms header when present; otherwise retry with exponential backoff and random jitter and set a maximum number of retries.

2. 2
   Fix 2.

   Increase the deployment's TPM allocation, rebalance quota across deployments, or request a quota increase.

3. 3
   Fix 3.

   Use the SDK's built-in retry, which defaults to two retries.

   Snippet

   Copy

   client = AzureOpenAI(
       azure_endpoint="https://<your-resource>.openai.azure.com/",
       api_key="<your-api-key>",
       api_version="2024-10-21",
       max_retries=5  # up to 5 retries with automatic exponential backoff
   )