Local Models

Running a model on your own hardware used to be a research exercise. It is now a practical choice, and for a specific set of problems it is the better one. The tools in this subcategory exist to make that choice easy: model runners, quantised builds, desktop front-ends, and applications that quietly ship inference inside themselves so you never think about it.

The case for local is straightforward. Nothing you type leaves the machine, which resolves the privacy question completely rather than through a promise in a policy. It works with no network. And the marginal cost per request is zero, which matters enormously for anything you run in a loop — bulk classification, transcription of a long archive, iterating on generated images, or an agent that makes many calls per task.

The case against is equally straightforward. Smaller models are less capable, particularly at long reasoning chains and obscure knowledge. Speed depends on your hardware, and on a laptop without a decent GPU or a modern unified-memory chip, the experience ranges from tolerable to unusable. And there is setup: model files are large, quantisation formats are confusing the first time, and the good defaults are not always obvious.

The most useful thing to know when comparing tools here is which of those three problems each one solves. Runners handle the plumbing and leave model choice to you. Desktop apps bundle sensible defaults and a chat interface, and hide the format question entirely. Embedded implementations — a transcription app or a note tool that just happens to run locally — are the ones most people should actually use, because the local part is an implementation detail rather than a project.

Hardware guidance in short: memory is the binding constraint more often than raw speed, because a model that does not fit runs at disk speed rather than chip speed. Roughly, a model needs about as many gigabytes as it has billions of parameters at eight-bit quantisation, and half that at four-bit. Fitting comfortably in memory matters more than the benchmark difference between two models of similar size.

Quality expectations should be calibrated by task. For summarising, extraction, classification, grammar, transcription and structured output, a small local model is frequently indistinguishable from a frontier one. For hard reasoning, current events and long multi-step work, it is not close. Choosing local well means knowing which of those you are doing.