Running a local LLM on a VPS: what actually fits in your RAM

Why run it on your own box at all

Three reasons, in the order people usually arrive at them. Cost, because a model you host has no per-token bill. Privacy, because your server's logs and configuration are exactly the kind of thing you should think twice about pasting into someone else's API. And availability, because a local model keeps working when an upstream provider is rate-limiting you, changing a model id, or simply down.

The reason not to is equally plain: a model running on CPU is slower than a hosted frontier model, and smaller models are less capable. The question is not which is better in the abstract. It is whether the smaller, slower, private one is good enough for the job in front of it — and for reading logs, writing config and answering "what is using this disk", it very often is.

Where the memory actually goes

Three things sit in RAM at once, and people usually only count the first.

The weights. This is the number on the download. A 7–8B parameter model at 4-bit quantisation lands somewhere around 4–5 GB; the same model at 8-bit is roughly double that.
The KV cache. This grows with how much context you actually use, and it is the part that surprises people. A long conversation or a large pasted log can add a gigabyte or more on top of the weights.
Everything else on the server. Your sites, your database, the page cache that is making them fast. A model that technically fits into free memory but forces the database out of cache has made the whole machine slower to answer one question.

So the practical rule is not "weights < free RAM". It is "weights + the context you will really use + headroom you were already relying on".

Quantisation, in one paragraph

Quantisation stores each weight in fewer bits. 4-bit roughly quarters the memory against 16-bit at a quality cost that, for a well-made quantisation of a modern model, is small enough that most people cannot pick it out on everyday tasks. Below 4-bit the losses stop being subtle. The usual sweet spot on a server with no GPU is a 4-bit or 5-bit quantisation of the largest model that still leaves the machine room to be a server.

Rough sizing, assuming no GPU

4 GB of RAM total. A 1–3B model, and only if the server is not doing much else. Useful for short, structured tasks; it will disappoint you on long reasoning.
8 GB. A 7–8B model at 4-bit is the standard answer and the first size that feels genuinely useful for administration work. Keep the context modest.
16 GB. The same 7–8B class with comfortable context, or a 13–14B at 4-bit if the box is otherwise quiet.
32 GB and up. Larger models become reasonable, and a mixture-of-experts model can be a better use of the memory than a dense one of similar size because only part of it is active per token.

Speed on CPU is bounded by memory bandwidth far more than by core count, which is why a model can feel slower on a many-core VPS than the core count suggests. Expect first-token latency while the model loads, and plan for the model to stay resident if you are going to ask it more than one thing an hour.

Making it useful rather than a toy

A model on its own answers questions about the world. A model that is useful on a server needs three more things:

Access to the machine's own facts — the nginx configuration, the service states, the last few hundred lines of the relevant log. Most of the value is retrieval, not reasoning.
A narrow set of actions it may take, each of which you can see and undo. "Write this vhost and reload nginx after testing the config" is a good action. "Run this shell command" is not, and should not be on the menu at all.
Something in front of it that knows what question is being asked. "Why is the server slow" is a procedure — load, then I/O, then the busiest process, then the busiest endpoint — and running that procedure reliably matters more than the model's eloquence.

How ServerPanel does it

The panel ships a model store that reads the RAM you actually have and tells you which models fit it before you download anything, so the first thing you learn is not an out-of-memory error. The models run on your hardware through a local runtime; the assistant is given the panel's own read access to logs, metrics and configuration, and a fixed set of reviewable actions rather than a shell. Nothing is sent anywhere: there is no API key to buy, because there is no third party in the loop.

You can see both screens, with sample data, in the live demo — the AI runtime page and the model store are the ones to look at.