Private LLMs: Six Questions to Settle Before Buying GPUs

The purchase order for GPUs is the easiest part of a private LLM project. It is also the part that gets signed first, usually before anyone has written down what the system is supposed to do. Hardware then becomes the project, and the questions that decide whether it pays off get answered by accident.

Here are the six that should come first. None of them needs a benchmark. All of them need a named owner.

1. Which data cannot leave, and why?

"We want it private" is a feeling, not a requirement. Ask for the specific data and the specific constraint behind it: a contract clause, a regulator, a customer audit, a trade secret that would be ruinous if leaked.

List the data classes the model will see. Support tickets with personal data are not the same as internal wiki pages, which are not the same as source code. Some of those classes may be perfectly fine in a hosted service under a proper data processing agreement. If only one of them truly has to stay inside, you may need a small private deployment for that one use case and nothing more.

If nobody can name the constraint, you are not buying privacy. You are buying reassurance, and that is a poor reason to commit to a rack.

2. What response time does the user actually feel?

Latency is a user story before it is a technical number. A chat assistant that streams its answer feels fast if the first words appear quickly, even when the full answer takes a while. A batch job that summarises last night's tickets does not care at all. A code completion inside an editor cares about almost nothing else.

Write the interaction down. Who waits, for what, and what do they do if it is slow? Interactive and batch workloads want different machines and different schedules, and a design that tries to serve both from one pool usually serves neither well.

3. Have you measured your real workload?

This is where most plans are weakest. Teams size hardware from the model's parameter count and a vendor slide, then discover that their prompts are far longer than assumed because retrieval fills the context window.

Before ordering anything, collect from your own usage, or from a pilot against a hosted API with synthetic data:

• the length distribution of prompts and of answers, not the average
• the number of concurrent users at the busiest hour, not across the day
• how much of each prompt repeats from request to request (system prompts, retrieved documents)
• which tasks need the largest model and which would be fine with a much smaller one

Context length and concurrency drive memory consumption at least as much as the model itself. A plan that ignores them is a guess with a purchase order attached.

Then ask the awkward question: could a much smaller model handle most of this work? Answer it with a test on your own tasks, not with a vendor claim.

4. Can the building take it?

Accelerator servers are dense. They pull a lot of power, turn it into heat, and expect the room to remove it. Your electrical supply, your rack power distribution, your cooling and even the floor loading are questions for the facilities team, and they should be asked before the order, not after delivery.

If you plan to host in a colocation facility, ask what power density per rack the provider will actually contract, in writing. If you plan to keep it on premises, ask whether the server room was designed for this class of equipment at all. Many were not.

Add the cost of electricity to the model. Hardware that runs all day has a running cost that does not appear on the quote, and it belongs next to the purchase price in the business case.

5. Who runs it at 3 a.m.?

A private LLM is a service with users. It has to be patched, monitored, upgraded, capacity-planned and restored. Model files get replaced when better ones appear, and inference software changes quickly. Someone has to test each upgrade before it reaches users, and someone has to carry the pager.

Be honest about the team. Do you have people who have run GPU workloads in production, or people who are willing to learn on the job while the business depends on the result? Both are acceptable answers. Pretending the question does not exist is not.

Ask also what happens when that person leaves. A system understood by one engineer is a dependency, and dependencies are exactly what this publication tells you to map.

6. What new attack surface do you now own?

Moving a model inside your perimeter removes one risk (data going to a third party) and creates others. The inference endpoint is a new network service. The model files and the software around them come from outside and need provenance checks. Prompts and outputs may contain sensitive data that now sits in your logs. If the assistant can call tools or read internal documents, access control has to be enforced on the data side, because the model will happily summarise anything it is allowed to see.

Questions for your security lead:

• Who can query the model, and how is that identity checked?
• Does retrieval respect the permissions of the person asking, or of the service account?
• Where are prompts and answers stored, for how long, and who can read them?
• How are model files and serving software sourced and verified before they run?

A decision rule

If you can answer the first three questions with evidence from your own data, and the last three with names and dates, buy. If you cannot, the cheapest move is a time-boxed pilot on rented capacity or a hosted service with synthetic data, followed by a written sizing document.

The pilot will not be glamorous. It will tell you whether the project is an AI problem, a data-governance problem or an operations problem. Only the first of those is solved by GPUs.