Small language models
SLMs — Phi, Qwen small, Llama 3.2, Gemma
Production inference for models that don't need a cluster.
Launch start here
Pay by the hour · no contracts · terminate anytime from the dashboard
What this is
SLMs (Small Language Models) are where latency and cost-per-million-tokens matter most. Phi-4 14B, Qwen-2.5 7B, Llama 3.2 1B/3B, Gemma 2B — these run fast on a single mid-range GPU, and you don't pay H100 prices for them. The instance boots on a stock PyTorch + CUDA 12.1 base — vLLM and TGI aren't preinstalled, but a single `pip install vllm` gets you a private OpenAI-compatible endpoint in minutes.
Our value-add
We don't push you to the biggest GPU. If a $2/hr Ada is right for your model, we'll say so.
Environment & recommended stack
- PyTorch 2.2 + CUDA 12.1 (stock base image)boots clean — install the serving stack below
- vLLMrecommended — OpenAI-compatible server, continuous batching, FP8 on Ada/Hopper
- Text Generation Inference (TGI)recommended HuggingFace alternative with speculative decoding
- llama.cpp + GGUFrecommended for CPU offload or quantized serving
- OpenAI-compatible FastAPI shimroll your own to point any OpenAI SDK at your instance
You stay in control
- Pick the quantization that matches your SLA — AWQ, GPTQ, FP8, FP16
- Protect your endpoint with your own API key (vLLM's --api-key flag)
- Auto-destroy deadline for dev endpoints
- SSH to inspect vLLM metrics, tune batching, and swap weights
Example workflow
Stand up an OpenAI-compatible Phi-4 endpoint
pip install vllm
python -m vllm.entrypoints.openai.api_server \
--model microsoft/Phi-4 \
--dtype float16 \
--port 8000 \
--api-key "$(openssl rand -hex 24)"Run inside the instance after `ssh root@<instance-ip>` or via the web terminal.
Launch options
Pick a GPU for small language models
Recommended start
Start here — single-GPU production SLM
48 GB Ada runs Phi-4 14B FP8 or Qwen-2.5 7B FP16 with real throughput
$2.04/ hour
Launch with this configScale option
Larger SLM / longer context
Headroom for 32K+ context and larger batches on 14B models
$4.41/ hour
Launch with this configHourly rates shown are current target rates — the authoritative price is surfaced on the /launch page at the moment you provision. Billing is per 5 minutes of runtime.