Local AI: when not to send the data to the cloud

Running the model on your own machine solves exactly one thing, and solves it completely. When it is worth it, what still costs when the licence is free, and the decision rule in one table.

“Running the model locally” sounds like an infrastructure decision and is really a data decision. The question is not what machine you have: it is whether the information you are working with should leave your company at all.

We do it for part of our internal work and document it publicly — how to set the model up and how to use it as a working subagent are written out step by step (in Spanish). This article is what comes before those guides: when it is worth it.

What it solves, and what it does not

It solves exactly one thing, and it solves it completely:

• The data does not move. No third party, no international transfer to disclose, no terms of service to re-read every time they change, no copy sitting in a system you do not control.

And it solves none of these, though they get attributed to it constantly:

• It does not make the output correct. A local model gets things wrong too, sometimes more often. Human review stays mandatory where it matters.
• It does not give you a lawful basis. Processing personal data still needs a ground, wherever the model runs.
• It does not excuse you from disclosing it. If you process customer data with AI, that belongs in your privacy policy even if the server is under your desk.

When the data should not leave

For most tasks the right answer is not “local”: it is send less. If you can strip the identities and the model still does the job, the problem is gone and you built nothing.

The case for local shows up when that is not enough. Typically:

1. The task is reading the full record. Reviewing contracts, analysing histories, processing documentation about people. Reducing the data reduces the task.
2. Volume turns the exceptional into the systematic. Sending one document is a decision; sending all of them, every day, is an architecture.
3. The client put it in the contract. Increasingly common in health, legal and finance: the data does not leave the infrastructure, full stop.
4. It is your own material and you would rather not hand it over. Proprietary code, strategy, unpublished numbers. No legal obligation here — just judgement.

The real cost is not the licence

Open models are free, which makes the arithmetic look obvious. It is not: the cost simply moved.

• Memory. This is the binding constraint. A model has to fit in memory to run comfortably, and that — more than processor speed — decides what you can actually use.
• Time. It generates more slowly than a commercial service. For a batch running in the background that is irrelevant; for something a person is waiting on, it is not.
• Maintenance. Updates, configuration, things that break. Without someone keeping it alive, the local model becomes a tool nobody uses.
• A capability ceiling. On the hardest reasoning a local model sits below the best commercial one of the moment. Denying that leads to avoidable disappointment.

The useful conclusion is not “it is not worth it”, it is it is not worth it for everything. And since your own compute is not billed per use, there is an asymmetry in your favour: trying a more capable model before concluding none can do the job costs only time.

Which tasks actually run well locally

The pattern we see: it performs where the task is bounded and the judgement is mechanical.

• Transforming text from one format into another.
• Summarising long documents against a fixed structure.
• Classifying and tagging into categories you defined.
• Extracting fields from documents with a known shape.
• Long drafts that a person edits afterwards.

And it performs badly where many things have to be tracked at once, or where a subtle mistake passes unnoticed. Aggregate figures are the canonical example: a model can reason well about a dataset and still get the sum wrong. Aggregates are computed with code, not with language — and that holds for any model, local or not.

A rule you can apply in ten seconds

The row people skip is the second, and it settles most cases: the data is identifiable, but the task does not need the identities. There is nothing to set up there.

FAQ

• Do I need a special machine? You need enough memory to hold the model, which is the real constraint — more than raw processor speed. A current laptop with generous RAM runs a mid-sized model; the very large ones need dedicated hardware. Checking whether it fits is the first step, not an afterthought.
• Is a local model as good as a commercial one? Generally not at the hardest reasoning. It is often good enough at bounded, mechanical work — which is most of the volume. Treating the two as interchangeable is the mistake; they are different tools.
• Does this remove the need for a privacy policy? No. Running the model locally removes the transfer to a third party. It does not remove the need for a lawful basis, for purpose limitation, or for telling people what you do with their data.
• Can it be combined with a commercial model? That is usually the right answer. Sensitive or high-volume work runs locally; the hard reasoning goes to the best available model with the data already reduced to what it needs.