Build a Multi-Tenant LLM Quota Layer with FastAPI and Redis (2026)

A single enthusiastic customer, a retry loop, or a background job can consume an LLM API allowance surprisingly quickly. A per-process counter may look fine in development, then fail as soon as your application runs behind multiple workers or containers. Meanwhile, a request-per-minute limit alone does not protect you from one tenant sending enormous prompts.

This guide builds a small, distributed quota layer for an LLM-backed API using FastAPI and Redis. It applies two limits together: a request bucket and an estimated-token bucket. The goal is not to replace your model provider's limits; it is to protect your own service, allocate capacity fairly across tenants, and return useful errors before an expensive request reaches the provider.

Last checked: 24 October 2026. Provider limits, SDK retry behavior, and product features change, so confirm the current documentation for the provider and library versions you deploy.

Who this is for

• Developers exposing an LLM feature to multiple customers or teams.
• Teams using FastAPI with multiple application workers or replicas.
• Products that need per-tenant request pacing and token-throughput controls.
• Engineers who want to distinguish temporary throttling from a real spend or billing cap.

What you will build

The request path will look like this:

1. Authenticate the caller and resolve a tenant ID from trusted server-side identity data.
2. Validate the request and estimate its token demand.
3. Ask Redis to atomically check both the tenant's request bucket and token bucket.
4. Return HTTP 429 with a Retry-After hint if either bucket is empty.
5. Only then call the model provider, while still handling provider-side throttling and recording actual usage.

Redis is shared by every API worker, so the quota decision is not reset when a request lands on a different process. Redis documents this use case for per-user, per-API, and per-tenant quotas, and explains why atomic Lua scripts avoid races between reading and updating a bucket.

1. Understand the limits you need

Do not treat all limits as interchangeable. A production LLM application usually needs several independent controls:

For example, a tenant might be allowed 30 requests per minute but only 60,000 estimated tokens per minute. Either limit can be exhausted first. OpenAI's official rate-limit guide describes separate request and token metrics and notes that the applicable limit can depend on the model and organization or project. Anthropic also documents request and token-based limits. Always inspect the current limits for your own account rather than copying numbers from an article.

A token bucket is useful because it supports a controlled average rate while allowing a bounded burst. It is not a monthly billing ledger, a concurrency semaphore, or a guarantee that the upstream provider will accept the request.

2. Install the dependencies and configure Redis

Use Python 3.11 or another version supported by your chosen FastAPI and Redis client releases. Create a virtual environment, then install the packages:

python -m venv .venv
source .venv/bin/activate
python -m pip install fastapi "uvicorn[standard]" redis pydantic

On Windows PowerShell, activate the environment with .venv\Scripts\Activate.ps1. Start Redis locally for development or point the application at a managed Redis instance. Set the connection string through an environment variable:

export REDIS_URL="redis://localhost:6379/0"

Use a private network, authentication, and TLS for a remote Redis service. Do not expose an unauthenticated Redis port to the public internet.

3. Implement an atomic two-bucket check

A naive implementation might read a counter, check it in Python, then increment it. Two concurrent requests can both read the same value and both pass. The script below performs the refill, decision, and update in one Redis-side operation.

It uses Redis server time to reduce clock differences between application instances. The two keys include the same Redis Cluster hash tag so they can be evaluated together in a clustered deployment. The tenant key should be derived from authenticated identity data, not copied blindly from a client-provided header.

RATE_LIMIT_LUA = """
local now_parts = redis.call('TIME')
local now_ms = tonumber(now_parts[1]) * 1000
    + math.floor(tonumber(now_parts[2]) / 1000)

local req_capacity = tonumber(ARGV[1])
local req_rate = tonumber(ARGV[2])
local req_cost = tonumber(ARGV[3])
local tok_capacity = tonumber(ARGV[4])
local tok_rate = tonumber(ARGV[5])
local tok_cost = tonumber(ARGV[6])
local ttl = tonumber(ARGV[7])

local req_state = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local req_tokens = tonumber(req_state[1]) or req_capacity
local req_ts = tonumber(req_state[2]) or now_ms
local req_available = math.min(req_capacity,
    req_tokens + math.max(0, now_ms - req_ts) * req_rate / 1000)

local tok_state = redis.call('HMGET', KEYS[2], 'tokens', 'ts')
local tok_tokens = tonumber(tok_state[1]) or tok_capacity
local tok_ts = tonumber(tok_state[2]) or now_ms
local tok_available = math.min(tok_capacity,
    tok_tokens + math.max(0, now_ms - tok_ts) * tok_rate / 1000)

local req_wait = 0
local tok_wait = 0
if req_available < req_cost then
    req_wait = (req_cost - req_available) / req_rate
end
if tok_available < tok_cost then
    tok_wait = (tok_cost - tok_available) / tok_rate
end

local function save_bucket(key, tokens)
    redis.call('HSET', key, 'tokens', tokens, 'ts', now_ms)
    redis.call('EXPIRE', key, ttl)
end

if req_wait > 0 or tok_wait > 0 then
    save_bucket(KEYS[1], req_available)
    save_bucket(KEYS[2], tok_available)
    return {0, math.ceil(math.max(req_wait, tok_wait)),
        math.floor(req_available), math.floor(tok_available)}
end

req_available = req_available - req_cost
tok_available = tok_available - tok_cost
save_bucket(KEYS[1], req_available)
save_bucket(KEYS[2], tok_available)
return {1, 0, math.floor(req_available), math.floor(tok_available)}
"""

The script accepts capacities, refill rates per second, the cost of this request, and a key TTL. When a request is denied, it stores the refilled balances but does not deduct the request's costs. When allowed, it deducts one request token and the estimated token demand. Keeping the decision atomic prevents concurrent workers from spending the same available capacity twice.

Keep scripts short and parameterized. Redis documents that Lua scripts execute atomically, but they also block other server activity while running; a rate-limit script should do only bounded arithmetic and a few Redis operations.

4. Wire the limiter into FastAPI

The following is a focused integration example. It assumes authentication middleware has already validated a session or token and placed a trusted principal on request.state.principal. Replace that seam with your real authentication system; the example intentionally does not accept a tenant ID from the request body.

import hashlib
import math
import os

from fastapi import Depends, FastAPI, HTTPException, Request
from pydantic import BaseModel, Field
from redis.asyncio import Redis

app = FastAPI()
redis = Redis.from_url(
    os.getenv("REDIS_URL", "redis://localhost:6379/0"),
    decode_responses=True,
)
rate_limit = redis.register_script(RATE_LIMIT_LUA)

# Illustrative application policy, not a provider's published limit.
REQUEST_CAPACITY = 30
REQUEST_REFILL_PER_SECOND = 0.5
TOKEN_CAPACITY = 60000
TOKEN_REFILL_PER_SECOND = 1000
BUCKET_TTL_SECONDS = 7200


class GenerateRequest(BaseModel):
    prompt: str = Field(min_length=1, max_length=100000)
    max_output_tokens: int = Field(default=512, ge=1, le=4096)


async def require_principal(request: Request):
    principal = getattr(request.state, "principal", None)
    if principal is None or not getattr(principal, "tenant_id", None):
        raise HTTPException(status_code=401, detail="Authentication required")
    return principal


def estimate_tokens(text: str) -> int:
    # Rough fallback only. Use the model/provider tokenizer when available.
    return max(1, math.ceil(len(text.encode("utf-8")) / 4))


async def enforce_tenant_quota(tenant_id: str, estimated_tokens: int):
    if estimated_tokens > TOKEN_CAPACITY:
        raise HTTPException(
            status_code=413,
            detail="Estimated request exceeds the per-request token allowance",
        )

    # Hash the trusted ID to avoid putting raw tenant identifiers in Redis keys.
    tenant_key = hashlib.sha256(tenant_id.encode()).hexdigest()[:32]
    prefix = f"llmrl:{{{tenant_key}}}"
    result = await rate_limit(
        keys=[f"{prefix}:requests", f"{prefix}:tokens"],
        args=[
            REQUEST_CAPACITY,
            REQUEST_REFILL_PER_SECOND,
            1,
            TOKEN_CAPACITY,
            TOKEN_REFILL_PER_SECOND,
            estimated_tokens,
            BUCKET_TTL_SECONDS,
        ],
    )
    allowed, retry_after, requests_left, tokens_left = map(int, result)

    if not allowed:
        wait = max(1, retry_after)
        raise HTTPException(
            status_code=429,
            detail={
                "error": "tenant_rate_limit_exceeded",
                "retry_after_seconds": wait,
            },
            headers={"Retry-After": str(wait)},
        )

    return {
        "requests_remaining_estimate": requests_left,
        "tokens_remaining_estimate": tokens_left,
    }


@app.post("/v1/generate")
async def generate(
    payload: GenerateRequest,
    principal=Depends(require_principal),
):
    estimated_tokens = (
        estimate_tokens(payload.prompt) + payload.max_output_tokens
    )
    quota = await enforce_tenant_quota(
        principal.tenant_id, estimated_tokens
    )

    # Call your provider adapter here. Keep provider credentials server-side.
    # Record the provider's actual usage and cost after a successful response.
    result = await your_llm_service.generate(
        prompt=payload.prompt,
        max_output_tokens=payload.max_output_tokens,
    )
    return {"result": result, "quota": quota}

Important: your_llm_service.generate is an integration point, not a built-in FastAPI method. Connect it to your provider SDK or existing service layer. The quota values are examples for demonstrating the mechanics, not recommended defaults for every product.

5. Make token estimation honest

The simple byte-length estimator is deliberately labeled a fallback. It can be wrong for code, non-English text, JSON, and different tokenizers. For tighter control, use the tokenizer appropriate to the selected model or the provider's documented usage estimates. Include a conservative allowance for output tokens when you reserve capacity before generation.

Track both estimated and actual usage. After the response, persist the provider-reported input tokens, output tokens, model, tenant, request ID, latency, and cost where available. Do not log raw prompts by default: they may contain personal data, secrets, or customer documents.

The example consumes estimated tokens from the throughput bucket and does not refund unused output allowance. This is intentionally simple and conservative. A more advanced design can reserve an estimate, then reconcile the reservation with actual usage using an atomic adjustment. Design that adjustment carefully so concurrent requests cannot over-credit a tenant.

6. Handle provider-side 429s and retries separately

Your tenant quota may allow a request while the provider rejects it. That is expected: the provider has independent limits and may apply request, token, model, project, or organization constraints. Inspect the provider's error code and response headers before deciding what to do.

• For a temporary rate-limit response, honor a valid Retry-After value when present.
• If no usable delay is present, use exponential backoff with jitter and a finite attempt limit.
• Do not retry billing failures, exhausted credits, invalid requests, or authorization errors as if they were temporary throttling.
• Account for retries already performed by the SDK; nested retry loops can multiply traffic.
• Use idempotency or job IDs for operations that might be submitted more than once.

OpenAI's official documentation distinguishes temporary rate limiting from quota or billing problems and describes retry guidance. Read the equivalent documentation for your actual provider and SDK version. A 429 response alone is not enough information to decide that another retry will help.

7. Keep money budgets separate from token buckets

A token bucket is a short-window throughput control. It cannot guarantee a monthly dollar cap because model prices differ, input and output tokens may have different rates, and provider usage can include other billable operations.

For a hard spend budget, maintain a durable usage ledger. A robust flow is:

1. Authenticate the tenant and select an allowed model.
2. Estimate the maximum plausible cost and reserve against the tenant's budget atomically.
3. Call the provider with a bounded output allowance and deadline.
4. Record actual provider usage and calculate cost from the applicable price table.
5. Settle the reservation against actual cost and release any unused reservation.
6. Reject or queue new work when the budget is exhausted, according to product policy.

Do not rely on a delayed analytics dashboard alone for a hard cap; two concurrent requests could both spend against the same apparent balance. Use a database transaction, conditional update, or other atomic reservation mechanism. If you already operate a gateway, compare the custom implementation with a managed option before maintaining another billing system. LiteLLM documents customer-level usage tracking, budgets, and rate limits in its proxy features.

8. Add concurrency control and observability

Rate limiting does not prevent a tenant from starting many long-running requests at once if the request bucket refills quickly. Add a separate semaphore or expiring lease for in-flight generations. Ensure leases are released in a finally block and can recover after a worker crashes. For streaming requests, keep the concurrency slot until the stream closes or times out.

Emit low-cardinality metrics such as allowed and denied requests, provider 429 counts, retry counts, estimated-versus-actual token differences, latency, timeout counts, and spend by model. Avoid using raw tenant IDs, prompts, or request IDs as metric labels; they can explode cardinality or expose private information. Keep detailed tenant-level accounting in logs or a database with appropriate access controls.

9. Troubleshooting

10. Security and operational trade-offs

• Tenant identity: derive the tenant from a verified identity or API key record. Never trust a user-controlled tenant header by itself.
• Key isolation: keep provider keys on the server, rotate them, and use least-privilege credentials.
• Redis access: require authentication and TLS for remote connections, restrict network access, and set resource limits.
• Denial-of-service resistance: validate request size before tokenization, cap output tokens, and apply connection and concurrency limits as well as rate limits.
• Fail-open versus fail-closed: failing closed protects cost but makes the feature unavailable during a Redis outage; failing open improves availability but removes a key abuse control. Choose intentionally per endpoint and risk.
• Fairness: per-tenant buckets prevent one customer from using another customer's allowance, but global provider limits still require a shared application-level budget or queue.
• Privacy: store the minimum usage metadata required for accounting. Apply retention rules and restrict access to detailed usage records.

FAQ

Why use Redis instead of an in-memory dictionary?

An in-memory dictionary is local to one process. Multiple workers or replicas can each grant their own allowance. Redis gives the application a shared state store, and an atomic script keeps the check-and-update operation consistent across concurrent requests.

Does this prevent every provider 429?

No. Your policy is only one layer. The provider can enforce different model, project, organization, burst, or account limits. Keep provider error handling and retry logic even when the tenant limiter is working correctly.

Can I use a fixed-window counter instead?

Yes, if the product can tolerate the burst behavior at window boundaries. A token bucket offers a controlled average rate with a configurable burst capacity. Pick the simplest algorithm that matches the policy you need.

Should a rejected request consume tokens?

This example updates refill timestamps but does not deduct request or token costs when a bucket denies the request. Other policies are possible, but document the semantics so clients and support teams know what a retry will do.

Is this enough to cap monthly spending?

No. Use durable actual-usage accounting and an atomic reservation or settlement process for money budgets. The Redis buckets shown here are short-window rate controls.

Further reading on GyanAangan

• Build a Local LLM Gateway with LiteLLM — for routing requests across local model runtimes.
• Regression-Test Local LLMs with Promptfoo and Ollama — for repeatable checks before changing model or routing behavior.
• Cloud GPU vs LLM API: Pricing and Break-Even — for comparing deployment and inference cost assumptions.
• Production RAG Engineering in 2026 — for the retrieval and observability side of LLM-backed systems.

Official references

• Redis rate limiter guide — algorithms, distributed quotas, and atomic Redis-backed rate limiting.
• Redis Lua scripting documentation — script atomicity, parameterization, and script-cache behavior.
• FastAPI dependency injection — a documented pattern for shared authentication and request policy dependencies.
• OpenAI API rate-limit guide — provider request/token limits, response headers, and retry guidance.
• OpenAI API error codes — interpreting 429s and other API errors.
• Anthropic API rate-limit overview — an example of provider-specific request and token limits.
• LiteLLM customer usage — an alternative when you need customer-level usage tracking, budgets, and rate limits in an existing gateway.

Final checklist

• Use authenticated tenant identity, not a client-supplied tenant ID.
• Enforce request and estimated-token limits atomically in shared storage.
• Keep tenant limits separate from provider limits, concurrency limits, and spend caps.
• Honor valid retry hints and bound all retry loops.
• Record actual usage, reconcile estimates, and monitor denials and cost.
• Test concurrency, Redis failure behavior, and provider 429 handling before production rollout.

The central design principle is simple: put a cheap, atomic policy check before expensive model work, but never mistake that check for a complete billing, security, or provider-capacity system.