Performance

Radium is optimized for low latency and high throughput. Understand what to expect and how to measure it.

Time to first token (TTFT)

TTFT is the time from sending a request to receiving the first streamed token. Radium measures TTFT internally and will publish live P50/P99 numbers on the Radium status page in a future release. Longer prompts increase TTFT linearly. Cache hits (repeated prompts) reduce TTFT by approximately 50%.

Measuring latency in your code

import time
from openai import OpenAI

client = OpenAI(api_key="YOUR_RADIUM_API_KEY", base_url="https://api.radium.cloud/v1")

start = time.time()
response = client.chat.completions.create(
    model="clarke-1.0",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)

first_token_time = None
for chunk in response:
    if first_token_time is None and chunk.choices[0].delta.content:
        first_token_time = time.time() - start
        print(f"TTFT: {first_token_time * 1000:.0f}ms")

Temperature-0 determinism

At temperature=0 and seed fixed, Radium produces deterministic output for a given model version. Minor variance (< 1% token difference) may occur across model hot-swaps or infrastructure updates. For maximum reproducibility, pin the model version in your request headers and pass a fixed seed.

Streaming performance

Always use streaming for latency-sensitive applications. It reduces perceived TTFT to near-zero and allows partial processing of long outputs.

# Non-streaming: wait for entire response
response = client.chat.completions.create(model="hal-1.0", messages=messages)
text = response.choices[0].message.content

# Streaming: first token arrives quickly, rest follows
for chunk in client.chat.completions.create(model="hal-1.0", messages=messages, stream=True):
    print(chunk.choices[0].delta.content, end="")

Batch processing

For offline workloads, use batch to amortize overhead:

import asyncio

async def batch_requests(prompts, model="tycho-1.0"):
    tasks = [
        asyncio.to_thread(client.chat.completions.create, model=model, messages=[{"role": "user", "content": p}])
        for p in prompts
    ]
    return await asyncio.gather(*tasks)

results = asyncio.run(batch_requests(prompts))

Performance tuning checklist

Use streaming for interactive UIs
Set temperature=0 for deterministic tasks
Use hal-1.0 for fastest TTFT
Use tycho-1.0 for lower cost on simpler tasks
Enable prompt caching for repeated queries
Batch offline workloads
Keep prompts within each model’s context window (Tycho 125K, Hal 250K, Clarke 1M)

Getting started

Models

Switching

Core concepts

Tools

Guides

Enterprise

Performance

Performance

Time to first token (TTFT)

Measuring latency in your code

Temperature-0 determinism

Streaming performance

Batch processing

Performance tuning checklist

​Performance

​Time to first token (TTFT)

​Measuring latency in your code

​Temperature-0 determinism

​Streaming performance

​Batch processing

​Performance tuning checklist

Performance

Time to first token (TTFT)

Measuring latency in your code

Temperature-0 determinism

Streaming performance

Batch processing

Performance tuning checklist