Skip to main content

Rate limits

Rate limits are restrictions that the Radium API imposes on the number of times a user or client can access our services within a specified period of time. There are two types of limits:
  1. Spend limits set a maximum monthly cost an organization can incur for API usage.
  2. Rate limits set the maximum number of API requests an organization can make over a defined period of time.
The API enforces these limits at the organization level.

Why do we have rate limits?

Rate limits are a common practice for APIs, and they’re put in place for a few different reasons:
  • They help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, Radium can prevent this kind of activity.
  • Rate limits help ensure that everyone has fair access to the API. If one organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single organization can make, Radium ensures that every customer can use the API without experiencing slowdowns.
  • Rate limits can help Radium manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, Radium can help maintain a smooth and consistent experience for all users.
Please work through this document in its entirety to better understand how Radium’s rate limit system works. We include code examples and possible solutions to handle common issues.

About rate limits

  • Limits are designed to prevent API abuse, while minimizing impact on common customer usage patterns.
  • Limits are set at the organization level. All of an organization’s projects and API keys share the same limits.
  • Every organization has the same standard limits. If you need higher limits, see Requesting higher limits.
  • You might hit rate limits over shorter time intervals. For instance, a rate of 1,000 requests per minute (RPM) is enforced as one request every 60 milliseconds, with bursts of up to 33 requests. Bursts larger than that trigger rate limit errors.
  • The API uses the token bucket algorithm to do rate limiting. This means that your capacity is continuously replenished up to your maximum limit, rather than being reset at fixed intervals.
  • All limits described here represent maximum allowed usage, not guaranteed minimums. These limits are intended to reduce unintentional overspend and ensure fair distribution of resources among users.

How do these rate limits work?

Rate limits are measured in RPM (requests per minute). To keep capacity fair between organizations, Radium also limits how many inference requests a single organization can have running at the same time. A request counts as running from when it starts until its response is complete; for a streamed response, that’s when the stream ends. Rate limits can be hit in either way depending on what occurs first. For example, if your requests take a long time to complete, you might receive a 429 error with code org_concurrency_exceeded while you are still well within your RPM limit. Other important things worth noting:
  • Rate limits are defined at the organization level, not at the project, API key or user level.
  • The number of requests you can have running at the same time is also limited to ensure fair capacity between organizations.
  • RPM counts every API request, including chat completions, messages, and listing models with an API key.
  • Requests rejected because too many are running at once, by an acceleration limit, or because the platform is busy are not counted against your RPM.

Spend limits

Every request is paid for from your organization’s credit balance. You can also set your own monthly spend limit, which is the maximum your organization can spend on the API each calendar month. You can view your organization’s credit balance and monthly spend limit on the Billing page in the Radium console.

Reaching your spend limit

Once a request would exceed your credit balance or your monthly spend limit, API requests return HTTP 402:
  • The error type is insufficient_credit when your credit balance can’t cover the request, and monthly_spend_limit_exceeded when the request would exceed your monthly spend limit.
  • The response has no retry-after header. Retrying, including the SDK’s automatic retries, fails until you add credit, raise or remove your monthly spend limit, or the next calendar month begins.
  • On the Messages API (/v1/messages), the error uses the Anthropic format: {"type": "error", "error": {"type": "...", "message": "..."}}.
If your remaining credit is temporarily held by requests that are still running, the API returns 429 with error type credit_reserved_in_flight (rate_limit_error on the Messages API) and a retry-after header. This clears as soon as those requests finish.

Setting your own spend limit

You can set a monthly spend limit to control costs:
  1. Navigate to the Billing page. In the Radium console, go to the Billing page (/manage/billing).
  2. Open the spend limit editor. In the Spend limits section, click Adjust limit (or Set limit if no limit is currently set).
  3. Adjust your spend limit. Enter a new value for Monthly spending limit and click Update limit (or Set limit). To remove the limit, click Remove limit.
The Spend limits section is shown once your organization has a payment method, to members who can manage billing.

Rate limits

The rate limits for the inference API are measured in requests per minute (RPM). If you exceed any of the rate limits you will get a 429 error describing which rate limit was exceeded. When the response has a retry-after header, wait that many seconds before retrying.
You might also encounter 429 errors because of acceleration limits on the API if your organization has a sharp increase in usage. To avoid hitting acceleration limits, ramp up your traffic gradually and maintain consistent usage patterns.
The following limits apply to your entire organization across all models and endpoints. Contact Radium support to request higher limits.

Requesting higher limits

To request higher rate limits, contact Radium support. Include the models you need raised, your expected peak requests per minute, and how many requests you expect to have running at once.

Response headers

The API response includes headers that show you the rate limit enforced, current usage, and when the limit will be reset. The following headers are returned: The x-ratelimit-* headers display the values for your RPM limit only. There are no headers for the number of requests running at once; the error code on a 429 error tells you which limit was exceeded. retry-after may be present on 429 responses caused by a temporary rate limit and 503 responses caused by temporary platform overload. It does not mean that spend limits or other errors that require user action can be resolved by retrying.

Error mitigation

Handle rate limits, rapid traffic increases and platform overload

The API returns 429 when your organization exceeds one of its rate limits or increases its request rate too quickly, and 503 when the platform or the requested model is temporarily overloaded. Check the HTTP status and error.code to tell these conditions apart: If retry-after is missing, increase the delay between retries and add a small random delay. A ramp_limit_exceeded error can occur even when your traffic is within its 1,000 RPM limit. It reflects how quickly traffic increased, not whether you exhausted that limit. The acceleration limit uses a ratio of 3.0 over 10-second windows. As a rule of thumb, keep the number of requests in any 10-second period below three times your average over the previous minute. Other 503 error codes with error type overloaded_error (replica_backstop_saturated, server_busy, routing_unavailable and service_unavailable) also mean the platform is temporarily overloaded, and should be handled the same way as pool_at_capacity. Handle both 429 and 503 in your SDK error handlers. For example, the OpenAI Python, TypeScript, and Ruby SDKs use RateLimitError for 429 and InternalServerError for 503. Other errors can use the same HTTP statuses, so inspect the error body before choosing a recovery action. For streaming requests, these HTTP error responses apply before the stream starts.

What are some steps I can take to mitigate this?

Retrying with exponential backoff

When a request exceeds a temporary rate limit, the API returns a 429 error. The response includes a retry-after header that tells you how many seconds to wait before trying again. Treat this value as a minimum: wait at least that long and add a small random delay so multiple clients don’t retry at the same time. The official OpenAI and Anthropic SDKs work with the Radium API, and each automatically retries eligible 429 and 503 responses, subject to its retry settings. Check the retry behavior of your installed version rather than assuming every server delay is supported. If you’re using your own HTTP client, follow retry-after when the header is present and contains a valid value. If it’s missing or invalid, fall back to exponential backoff with jitter. Limit both the number of attempts and the total time spent retrying. If you manage retries in your application, disable SDK retries or account for them in those limits so nested retry loops don’t multiply requests. Don’t retry spend limit or other errors that require you to take action. Exponential backoff means waiting briefly after an unsuccessful request, then increasing the delay after each unsuccessful retry. This continues until the request succeeds or reaches a configured retry limit. This approach has many benefits:
  • Automatic retries means you can recover from rate limit errors without crashes or missing data
  • Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail
  • Adding random jitter to the delay helps retries from all hitting at the same time.
Note that requests rejected with a 429 or 503 error don’t count toward your requests per minute, but retrying before retry-after has passed still fails.
Example: Manual backoff implementation
If you don’t want to use third-party libraries, you can implement your own backoff logic following this example. It follows retry-after when present, and disables SDK retries so they don’t multiply its own:

Limit the number of requests you run at once

The limit on requests running at the same time counts requests that are still running, so sending requests faster doesn’t increase throughput once you reach it. If you send many requests in parallel, cap how many run at once in your client. Start with a small number and increase it while you don’t receive org_concurrency_exceeded errors:
If you need more throughput for long-running requests, contact Radium support; see Requesting higher limits.