Offline AI: can you run your own AI on a laptop with no internet?

Most people meet AI through an app. You type a question into ChatGPT, Gemini or Claude and the answer comes back a few seconds later. It feels like it lives on your phone. It doesn't, and that one detail decides two things: whether you get an answer at all, and who keeps your question.

The AI you use lives on someone else's server

When you use a cloud AI, your question travels over the internet to the company's computers, gets answered there, and travels back. Three plain consequences follow.

• It needs a connection. No internet, no answer.
• It can go down. OpenAI publishes a public status history for ChatGPT and its other services. When we read it on 1 November 2026, it listed several incidents in the last week of October alone, from elevated error rates to degraded model performance. Each entry links to its own incident report. Every large online service has days like that.
• Your questions are kept on their side. OpenAI's own policy page says that for services for individuals, such as ChatGPT, "we may use your content to train our models", unless you opt out in your settings. That's a clear, published policy. It's just not the same as the question staying with you.

None of this makes cloud AI bad. It's a very good service. It's just a service, run on someone else's computers and on their terms.

A private AI can now run on your own laptop

A few years ago, running a capable language model at home meant a big graphics card and a lot of patience. Two things changed.

The models got small. Qwen3.5-4B, published by the Qwen team in February 2026, has about 4 billion parameters and is released under the Apache-2.0 licence, which allows anyone to use it. Its model card lists 201 languages and dialects. Squeezed down to 4-bit precision, the file is 2.74 GB.

The software got lean. llama.cpp is a free, MIT-licensed project whose stated aim is running these models with minimal setup on a wide range of hardware. It runs on ordinary Intel and AMD processors, on Apple silicon through Apple's Metal framework, and on built-in laptop graphics through Vulkan. It also includes a local server with an OpenAI-style interface, so other programs on the same laptop can talk to the model.

Put those together and a model that would have needed a server now runs on the laptop you already own, with the wifi switched off.

How fast is it, honestly?

The tech site It's FOSS tested small models on an older Intel i5 laptop with about 12 GB of RAM, using only the processor (published 15 May 2026). Their results give a fair picture:

• Phi 4 Mini, a 3.8 billion parameter model close in size to Qwen3.5-4B, wrote at about 6.9 tokens a second.
• The smallest models, around 1 billion parameters, ran at roughly 18 to 36 tokens a second.
• The bigger 7 to 8 billion parameter models dropped to about 3 to 4 tokens a second.

A token is a chunk of text, often part of a word. Around 7 a second is roughly reading pace, so you read along as the answer appears. The tester's own conclusion was that the idea local AI needs a graphics card is out of date, and that the 3 to 4 billion range is where the trade between speed and quality gets interesting.

Newer laptops, especially Apple silicon Macs and recent Windows machines with good built-in graphics, will usually be quicker. Plan for the older laptop anyway. It's the one that ends up in the cupboard for a bad day.

Where a small offline AI falls short

Be clear-eyed about this. A 4B model on a laptop is not ChatGPT, and it shouldn't pretend to be.

• It will get some facts wrong, especially numbers, dates, names and doses, and it can sound sure while doing it.
• It only knows what it learned in training, so recent events are missing.
• It's weaker at maths and long chains of reasoning.
• It can mix up New Zealand and US details, like emergency numbers or laws.

The fix is to give it something to check against. A local AI is far more useful with a local library beside it. With an offline copy of Wikipedia on the same laptop (our post on Kiwix and offline Wikipedia covers how), the AI can point you to the article, and the article is the source you trust.

Doing it yourself

You can build this for free. The rough steps:

1. Download llama.cpp for your system from its GitHub releases.
2. Download a model file in GGUF format from Hugging Face, sized to your RAM. A 4-bit 4B model is a sensible start on 8 GB.
3. Start llama.cpp's local server and connect a chat window to it.
4. Install Kiwix and download a Wikipedia file for reference, separately.

Expect an evening of it: choosing the right build for your graphics, picking a model that fits your memory, and getting the parts to work together. If you enjoy that kind of thing, it's a good evening.

The short version

• Cloud AI runs on someone else's server. It needs the internet, it can go down, and your questions are kept there.
• Small open models now run on an ordinary laptop. Slower, but your questions never leave your laptop.
• Pair the AI with an offline reference library so you can check what it says.

If you'd rather skip the setup, the Offline Brain download is that evening done for you: a small 4B model that runs on 8 GB of RAM, Wikipedia and WikiMed beside it, and a map pack, installed on Windows 10 or 11 (Mac version coming). It's US$29 (NZ$49), one payment, with 30 days to change your mind.