Choosing an Inference Stack: A Dependency Checklist
Runtime, model format, hardware support, observability and upgrade path: the dependencies to check before committing to an LLM inference stack.
An inference stack is the set of software that turns a model file into an API your applications can call. Picking one feels like a technical preference, a choice between tools with different strengths. It is better treated as a dependency decision, because the stack sits between your applications and your hardware, and it will be there long after the demo is forgotten.
This checklist is not a comparison of products. It is a list of the five places where a choice quietly becomes a commitment.
1. The runtime: what runs the model, and who controls it?
Start with governance. Is the runtime open source under a licence you have read? Who maintains it: a foundation, a company, a handful of volunteers? How are releases made, how often, and how are security fixes communicated?
Then look at the interface it exposes. Many servers present an API shaped like a popular commercial one, which helps portability at the application layer. Check what it supports in practice: streaming, tool calling, structured output, batching, multiple models. A compatible endpoint that lacks the one feature your application uses is not compatible for you.
Questions to put to the project or vendor:
- Which API behaviours are guaranteed, and which are best effort?
- What is the deprecation policy for flags and endpoints?
- Can we pin a version and keep receiving security fixes for it?
2. The model format: how portable are your weights?
Models are distributed in several file formats, and quantised variants add another layer. Some formats are tied to a particular runtime or hardware target. Others are designed to move between runtimes. The format you choose today determines which runtimes can load your models tomorrow.
Check three things:
- Can you convert between formats without losing quality you care about, and who verifies it?
- If you quantise a model yourself, is the process reproducible and scripted, or did someone do it once on a laptop?
- Where do the original, unquantised weights live in your own storage, with their licence text?
Keep the originals. A quantised file is a derivative, and quantisation methods improve; being able to start again from the source is cheap insurance.
3. Hardware support: what runs where?
Hardware support is not a yes or no. It is a table. The vLLM documentation, for example, has a page dedicated to which quantisation methods are supported on which hardware, which tells you the answer depends on the combination of method and device, not on the runtime alone.
Make the table yourself, for your own candidates:
| Accelerator family A | Accelerator family B | CPU only | |
|---|---|---|---|
| Your model, full precision | check | check | check |
| Your model, the quantisation you plan to use | check | check | check |
| Features you rely on (batching, long context, tool calling) | check | check | check |
Fill every cell from documentation and from a test on a machine you control, not from a slide. An empty cell is information: it says your portability is narrower than you thought.
Also ask about the lower layers. Which driver and toolkit versions does the runtime require? Are they available for the operating system you standardise on? What happens when the driver you need is not compatible with the kernel your security team requires?
4. Observability: can you see what it is doing?
A serving stack that you cannot measure cannot be operated. At minimum, expect the runtime to expose, in a standard format your monitoring already understands:
- request rates, errors and queue depth
- latency broken down in a way that separates waiting from generating
- memory use on the accelerator, and when it is under pressure
- per-model and per-tenant breakdowns if you share the service
Then check the less obvious part: logs. Do prompts and outputs end up in logs by default? Can you disable that, or redact it? In a business context, the log is a data store, and it falls under the same rules as any other.
Ask whether metrics are stable across versions. Dashboards that break at every upgrade do not get maintained for long.
5. The upgrade path: what happens at the next release?
Inference software moves quickly, and so do models. A stack you cannot upgrade is a stack that ages in place, with unfixed vulnerabilities and no support for the new model you want.
Test the path before you rely on it:
- Is there a documented upgrade procedure, with release notes that name breaking changes?
- Can you run two versions side by side and shift traffic gradually?
- Do you have a set of your own test prompts that detects a change in output after an upgrade? Without one, you will find regressions from user complaints.
- How is rollback done, and has anyone done it?
Add to this a model-change test. When a new model version arrives, can you evaluate it against the old one on your own cases in a day, or does it take a month of ad hoc effort?
How the five add up
Draw one line per layer and mark, in a different colour, the dependencies that are shared with something else in your stack. A single vendor appearing under runtime, hardware and monitoring is a pattern you want to see on paper.
Then write two sentences:
- If this runtime disappeared tomorrow, how would we serve our models in a month?
- If this hardware family were unavailable, which cells of our table are filled for another?
If you cannot answer the first, your application depends on a stack you do not control. If you cannot answer the second, your hardware choice has become your software choice.
Neither answer means you should not proceed. It means you should proceed knowing the price, which is what a dependency checklist is for.