vLLM on AWS - Your Own Inference Endpoint, Billed Like Compute Instead of Tokens
Every hosted inference API works the same way underneath: you send tokens, they run the model on hardware you'll never see, and the bill grows in exact proportion to how much your product gets used. That's fine when usage is low. It stops being fine the month your app actually takes off and the inference line item grows faster than the revenue that's supposed to be paying for it.
Meetrix packaged vLLM into an AWS Marketplace CloudFormation stack that deploys to a single GPU EC2 instance in your own AWS account. NVIDIA drivers and CUDA pre-installed, the container pre-pulled so first boot doesn't stall downloading it, nginx tuned for streaming tokens, SSL handled automatically. Ready to try it? Launch the Meetrix vLLM stack on AWS Marketplace.
What Is vLLM?
vLLM is an open-source inference and serving engine for large language models, released under the Apache 2.0 license. The throughput it's known for comes from two specific mechanisms, not vague optimization. PagedAttention manages the KV cache the way an operating system manages memory pages, so GPU memory doesn't get wasted on padding or fragmentation. Continuous batching keeps new requests flowing into a batch that's already running, instead of waiting for a batch to finish before starting the next one. Together, that's what lets one GPU serve far more concurrent requests than a naive implementation would.
It also exposes an OpenAI-compatible HTTP API at /v1, which matters more than it sounds like it should. Code written against the OpenAI SDK, LangChain, or any tool that expects that API shape works against your self-hosted model with a base URL change and nothing else. You're not rewriting integration code to switch from a hosted provider to your own instance.
What Pay-Per-Token Actually Costs at Real Volume
How Deployment Works
Setting this up by hand means picking a GPU instance, installing the right NVIDIA driver and CUDA version, wiring up the container GPU runtime, configuring nginx to not buffer streamed tokens, and downloading the model weights before serving a single request. Through the Marketplace, it's four steps:
- Subscribe and Launch from AWS Marketplace Open the Meetrix vLLM listing, subscribe, and launch the CloudFormation stack on a G-family GPU instance. g5.2xlarge is the sensible default for 7B to 8B models.
- GPU Drivers and the Container Are Already Set Up NVIDIA drivers, CUDA, and the container GPU runtime are pre-installed and matched to the instance's GPU, with the container image pre-pulled so first boot doesn't spend its first minutes downloading it.
- Pick Your Model Set the ModelName stack parameter to a Hugging Face repository ID, Qwen/Qwen2.5-7B-Instruct for example. Add a Hugging Face token only if the model is gated or private.
- Wait for Boot, Then Call the API Give the instance a few minutes to finish starting, then point any OpenAI-compatible client at your domain's /v1 path. Restrict access before you send it real traffic, it's open by default.
What Meetrix Brings to This Deployment
- No GPU Toolchain to Assemble - NVIDIA drivers, CUDA, and the container GPU runtime come pre-installed and version-matched, not a driver compatibility puzzle to solve before vLLM will even start.
- A Pre-Pulled Container, Not a Cold Download - The vLLM container image is already pulled onto the instance, so first boot is spent starting the service, not fetching gigabytes of image layers.
- SSL and a Streaming-Ready Proxy From the Start - Automatic Let's Encrypt certificates via Route53, with nginx already tuned so token-by-token streaming responses don't get buffered into one slow chunk.
- Interactive API Docs Included - Swagger UI at /docs and ReDoc at /redoc ship with the instance, so you can see and test every route without hunting through vLLM's own documentation first.
- Hardened by Default - ufw, unattended-upgrades, and the SSM agent are active out of the box, plus IP-restricted SSH, not a hardening pass you do after the fact.
- People Who Actually Run This Stack - Support goes to aws@meetrix.io with a response inside 12 hours, from engineers who deploy GPU AMIs regularly, not a queue reading from a script.
Who Is vLLM on AWS Right For?
This fits teams that want inference costs to behave like infrastructure, not like a metered utility that tracks their growth. More specifically, it's a strong match if you're:
- A team that priced out Together AI or Bedrock against real projected token volume and didn't like watching the number scale with success
- A developer who wants to run Llama, Qwen, Mistral, Gemma, or Phi without rewriting OpenAI SDK integration code to talk to it
- An engineering org that needs model weights and inference requests to stay inside its own AWS account, not a shared third-party cluster
- A team serving high, steady request volume where a flat GPU-hour cost beats a per-token fee that never stops accumulating
- Someone who wants to swap the served model by changing a stack parameter, rather than migrating to a different hosted provider entirely
- A builder who wants Swagger docs and an OpenAI-compatible endpoint on day one, not a bare inference script to wrap in an API themselves
vLLM on AWS by Meetrix vs Alternatives
| Feature | vLLM on AWS by Meetrix | Together AI | Amazon Bedrock | Self-Managed Install |
|---|---|---|---|---|
| Hosting | Your own AWS account and VPC, fully self-hosted | Together's cloud, no self-host option for serverless inference | AWS-managed infrastructure, not an instance in your own VPC | Your own GPU server, set up by you |
| License | Apache 2.0, fully open source | Proprietary platform | Proprietary AWS service | Open source, but you own every step of running it |
| Pricing Model | Flat GPU instance rate from $0.019/hr, no per-token fee | Pay-per-token, roughly $0.10-$9 per million tokens by model | Pay-per-token, roughly $0.035-$10+ per million tokens by model | GPU server cost, no per-token fee, but you build the stack |
| Model Choice | Any Hugging Face model that fits your GPU's memory | Together's curated model catalog | Bedrock's curated model catalog | Whatever you set up yourself |
| Deployment Time | Minutes via CloudFormation | Instant API signup | Instant, already inside AWS | Hours to days, drivers, CUDA, container runtime, SSL |
| Data Location | Your own AWS account | Together's infrastructure | AWS's shared infrastructure | Your server |
| Support | Meetrix engineers, response inside 12 hours | Support tiers, gated by plan | AWS support plans, gated by tier | Community forums only |
Resources
Under the Hood
Here's what's actually running once the stack finishes deploying.
vLLM in a Pre-Pulled Container
Runs as a container on Ubuntu 26.04 LTS, with the image already pulled onto the instance so first boot starts the service instead of downloading it.
Drivers Matched to the Instance
NVIDIA drivers, CUDA, and the container GPU runtime come pre-installed and matched to whichever G-family GPU you chose, not a version you have to reconcile by hand.
nginx Tuned for Streaming
Configured for token-by-token streaming responses rather than buffering, so a chat completion streams the way it's supposed to instead of arriving as one delayed block.
SSL via Let's Encrypt
Certificates issue automatically when a domain is supplied and its DNS points at the instance, no manual certbot step required.
Swagger and ReDoc Included
Interactive documentation ships at /docs and /redoc, listing every route the OpenAI-compatible API exposes.
ufw, Unattended Upgrades, and SSM
The base image ships with a firewall active, automatic security patching enabled, and the SSM agent running, plus IP-restricted SSH access.
One Thing vLLM Won't Do For You
It won't lock the door behind you. The OpenAI-compatible endpoint is open to anyone who can reach it the moment the instance boots, no API key, no auth, nothing. That's a reasonable default for a quick test against your own IP, and a genuinely bad one to leave in place before pointing a public app at it. Restrict the security group, put it behind a VPN or a reverse proxy with authentication, or turn on an API key before real traffic touches it. This is the one step in the whole deployment that's on you, not the AMI.
Frequently Asked Questions
What is vLLM, in plain terms?
An open-source engine for serving large language models efficiently. It uses PagedAttention to manage GPU memory for the attention cache without wasting it, and continuous batching to keep the GPU busy across many requests at once, which is where the throughput actually comes from. It exposes an OpenAI-compatible HTTP API, so code written for the OpenAI SDK works against it with just a base URL change.
Is vLLM free?
The software is, it's open source under the Apache 2.0 license, no license fee from Meetrix either. What you pay for is the GPU instance itself, since vLLM needs an NVIDIA GPU to run. There's no per-token fee layered on top the way there is with a hosted inference API.
What does this cost to run on AWS?
It scales with the GPU instance you pick: $0.019/hour on a g4dn.xlarge, g5.xlarge, or g6.xlarge, up to $0.048/hour on a g5.12xlarge for bigger models or heavier concurrency. There's no free trial on this listing and no refunds, but you can cancel the subscription any time.
Which instance should I actually pick?
g5.2xlarge is the sensible default, one A10G GPU with 24 GB of memory, enough for most 7B to 8B models comfortably. Move up to g5.12xlarge or a g6e size if you're serving a larger model or need more concurrent throughput. CPU-only instances won't work, vLLM needs the GPU.
Which models can I actually run on this?
Anything on Hugging Face that fits in your GPU's memory, set through the ModelName stack parameter as a repo ID like Qwen/Qwen2.5-7B-Instruct. Llama, Qwen, Mistral, Gemma, and Phi are all supported out of the box. A gated or private model just needs a Hugging Face access token added at launch.
Is the API secured by default?
No, and this is worth knowing before you deploy, not after. The OpenAI-compatible endpoint is open to anyone who can reach it out of the box. Lock down the SSHLocation and security group settings, put it behind a VPN or a reverse proxy with auth, or run vLLM with an API key. Don't leave an unauthenticated inference endpoint facing the public internet.
How do I call it from my existing code?
Point any OpenAI-compatible client at https://your-domain/v1 and use the chat completions or completions route with the model name you deployed. The interactive Swagger UI at /docs and the ReDoc page at /redoc list every route if you want to poke at it directly first.
Where does the model actually run?
On the GPU EC2 instance inside your own AWS account and VPC. The weights, the requests, and anything you send through the API stay on infrastructure you control, not on a third party's shared inference cluster.
Run Your Own Inference Server on AWS
Stop paying per token for inference you could run on your own GPU. Deploy vLLM on AWS in minutes, set up by a team that runs this stack every day.
Deploy on AWS Marketplace