Offline-First AI: Building an LLM That Works Offline

The power's been out for nine hours, the cell network is overloaded, and the freezer in your garage is full of meat you bought on Saturday. Is it still safe to eat? Normally you'd ask a chatbot. Right now, the internet is the thing that broke first. That scenario is the design brief for a growing class of offline-first AI tools: an assistant and a reference library that live entirely on your laptop. The model, the corpus, the search index — everything runs locally, so asking "how do I purify this water" works identically with the ethernet cable yanked out. I find this more interesting as an architecture exercise than as prepper gear. Stripping the network away forces a set of engineering decisions that cloud-hosted AI gets to ignore, and those decisions turn out to be useful far beyond emergency kits. If you've thought about running large language models on your own hardware, the constraints of an offline assistant are the sharp end of that problem.

What Offline-First AI Actually Has to Ship

A cloud assistant is thin: your client sends text, a data center sends back tokens. An offline assistant has to replicate that entire stack on consumer hardware, which means shipping three big things and keeping them in harmony. First, the inference engine. In practice this means llama.cpp or one of its derivatives, because it runs on Apple Silicon, plain x86 CPUs, and consumer GPUs with a single dependency tree. The Mac app bundles it; the Windows build pulls it during setup. Either way, you can't assume it exists on the user's machine — you install it yourself, quietly, during first run. Second, the model. This is where it gets interesting, and I'll come back to it, because picking the right model is the central design problem. Third, the library. One offline assistant in this space ships roughly 930,000 Wikipedia articles (the most-read subset), medical references like MedlinePlus and WikiMed, FEMA and CDC emergency guidance, military survival manuals, wikiHow, iFixit repair guides, and Stack Exchange Q&A dumps on topics like home repair and ham radio. All plain text, all with licenses preserved and shown per-article. The full download is about 9 GB and balloons to 19 GB installed; a slim version without Wikipedia and iFixit is 1.3 GB. That's a curatorial act as much as an engineering one — somebody decided which sources survive the cut. The whole thing has to fit a three-step setup that takes five minutes of clicking and up to two and a half hours of downloading. That last number matters more than it looks: the download window is your only chance to fetch anything, ever. It's the same discipline you'd apply to software you can't reboot or reach, except the unreachable environment is a storm-darkened kitchen.

The Model Tiering Problem: One App, Many Brains

Here's the decision cloud software never faces: you don't know what hardware your user has, you can't probe their RAM and route them to a bigger cluster, and the answer changes both the download size and the quality of every answer forever after. The solution in this category of tools is hardware-probed tiering. On first launch, the app checks system memory and picks a model that fits: - 128 GB or more: a large flagship-class model (one build uses NVIDIA Nemotron 3.5 Super, ~80 GB on disk), scoring 21 of 22 on the project's blind-graded test of blackout and first-aid questions. - 32–96 GB: a mid-size model around 27B parameters (~18 GB), also 21 of 22. - 32 GB on Windows: a speed-tuned ~25 GB model, 19 of 22. - 16–31 GB: an 8B-class model (~6 GB), 14 of 22. Read that last row again. The machine most people actually own — a 16 GB laptop — gets a brain that misses the main point of official guidance roughly a third of the time, on questions where the main point is things like "don't eat the chicken that's been at room temperature for six hours." This is the uncomfortable arithmetic of offline-first AI. Quantization and small models have gotten genuinely good — a well-tuned 8B model is a fine writing assistant and a decent search interface. But the quality curve doesn't bend to fit your deployment constraints. When a cloud app wants a smarter model it flips a config flag. When an offline app wants one, it needs the user to have bought a bigger laptop three years ago. You ship the curve, and the curve ships you. The honest mitigation is the one these tools adopt: treat the small model as an interface to the corpus, not an authority on it. Every answer lists exactly which articles were read, with the passage highlighted, because the designers know the 8B tier will sometimes be wrong.

Retrieval as the Trust Layer, Not the Feature

Every retrieval-augmented generation system shows sources now; in the cloud it's a checkbox feature, a bit of transparency theater. Offline, retrieval is load-bearing. It's the only thing standing between a plausible-sounding hallucination and a user acting on it in an actual emergency. The design that emerges looks like this: the model searches the plain-text library, reads the top passages, and writes an answer that cites what it read. If the user asks it to look something up, it searches even if it had already answered from memory — and the UI says which mode each answer came from: memory, articles it opened, or search results it never opened. That last distinction is subtle and important. "I found matching snippets" and "I read the article" are different claims with different reliability, and the interface treats them as such. There's also a failure-mode detail I love: when the model went off and searched but never wrote a final answer — a real thing small models do — the app now force-feeds it the best articles it found and makes it try once more, instead of showing the user nothing. Offline, you can't fall back to a bigger model or a retry queue. You build the retry into the agent loop. It's a compact example of what people mean when they say LLM agents behave like distributed systems: partial failure everywhere, and the orchestration layer's job is to make the failure survivable. One more architectural choice worth stealing: the reasoning budget. The app exposes how many tokens the model may think before answering — off, 1,000, 4,000, 16,000, or unlimited — and at the limit the thinking is cut and the model answers from what it has. On battery, a model that thinks in circles isn't just wrong, it's a power drain. A hard cap turns an unbounded compute problem into a bounded one, which is exactly the instinct you want when the wall socket is theoretical.

The Skeptics Have a Point (About the Wrong Half)

The sharpest criticism of this whole genre: why not skip the LLM and ship a searchable archive? Kiwix has done exactly that for years — Wikipedia and medical references, compressed, indexed, searchable offline, no hallucination risk, runs on a potato. And for a dataset of curated text, a search engine is arguably the more reliable tool. One tester asked the offline assistant how to build a bow drill and got the parts right but the motion wrong — the model described rolling the spindle between the hands, ignoring the bow entirely, and mentioned "fanning the sparks," which a friction fire does not produce. A search engine can't make that mistake because it doesn't generate anything. But I think this critique misses what the LLM is actually for in these systems, and the strongest defense of the design is to reposition it: use the model as search intelligence, not as oracle. Keyword search fails when you don't know the vocabulary — when what you have is symptoms, not a diagnosis; a broken thing, not a part name. Semantic retrieval helps with "near in meaning" but not with "connected by mechanism or causation," which is exactly the gap a generative model bridges: it can take "crops yellowing at the leaf edges" and produce the five reading paths a keyword never would have found. The model's job is to get you to the right pages. The pages' job is to be right. The other critique — "if you need an LLM during the apocalypse, just learn the skills" — is fair as life advice and useless as software criticism. Nobody's first-aid knowledge covers pediatric burns, water purification chemistry, and generator repair. Books cover all of it; the argument is only about the interface. Where the skeptics land a clean hit is the 14-of-22 tier. If the interface layer is wrong a third of the time and every answer carries a "check the sources" disclaimer, at some point the honest product is the archive with a search box. The LLM earns its place only insofar as it demonstrably gets non-experts to the right passage faster than Ctrl+F does. That's an empirical question about your users, not a vibe.

A Minimal Offline-First AI Stack

If you wanted to build your own version of this — and you should, it's a genuinely instructive weekend project — the skeleton is smaller than you'd expect. The interesting parts are the choices, not the code. A retrieval-first local assistant in Python looks roughly like this:

import subprocess
from pathlib import Path
CORPUS_DIR = Path("./library")          # plain-text articles, one file each
MODEL = "./models/granite-8b-q4.gguf"   # quantized to fit 16 GB RAM
def search_corpus(query: str, k: int = 5) -> list[dict]:
"""Dumb but honest: rank articles by keyword overlap.
Swap in BM25 or embeddings later; the interface is the point."""
terms = set(query.lower().split())
hits = []
for f in CORPUS_DIR.rglob("*.txt"):
text = f.read_text(errors="ignore")
score = sum(text.lower().count(t) for t in terms)
if score:
hits.append({"path": f.name, "score": score, "text": text[:3000]})
return sorted(hits, key=lambda h: -h["score"])[:k]
def answer(question: str) -> str:
sources = search_corpus(question)
context = "\n\n".join(
f"[SOURCE {i}: {s['path']}]\n{s['text']}"
for i, s in enumerate(sources, 1)
)
prompt = (
"Answer ONLY from the sources below. Cite them as [SOURCE n]. "
"If they don't cover it, say so.\n\n"
f"{context}\n\nQuestion: {question}\nAnswer:"
)
out = subprocess.run(
["llama-cli", "-m", MODEL, "-p", prompt,
"-n", "512", "--ctx-size", "8192"],
capture_output=True, text=True,
)
citations = "\n".join(f"[{i}] {s['path']}" for i, s in enumerate(sources, 1))
return out.stdout + "\n\n--- Sources ---\n" + citations
print(answer("how long is food safe in a fridge without power"))

Everything that makes this feel like a product rather than a toy is in the constraints baked into those forty lines: the prompt forbids answering outside the sources, citations are appended mechanically (never trusting the model to format them — one real-world bug involved models inventing a "Sources:" line listing their own search commands), and the generation length is capped. Missing here but required for production: an embedding index for semantic search, a reasoning-token budget, memory probing to pick the model file, and per-file license metadata so the UI can display it. Note what isn't there: no network calls, no API keys, no telemetry. Once the model file and corpus exist on disk, this program is physically incapable of leaking a question. Privacy stops being a policy and becomes a consequence of the dependency graph.

The Constraints That Follow You Home

Build offline-first and a handful of non-obvious constraints become permanent residents of your architecture. **Storage and install windows are one-shot.** The full library needs about 27 GB of free space during installation before settling to 19 GB, and everything must arrive during the initial download. There's no lazy-loading a missing medical reference during a hurricane. This pushes you toward aggressive curation — 930,000 articles sounds like a lot until you remember Wikipedia has millions; someone ranked by readership and cut the tail. **Power is a first-class resource.** Measured on a MacBook Pro, answering draws about 100 W for 5 to 40 seconds per question — a full battery buys roughly 100 to 300 answers. Suddenly long chain-of-thought isn't just slow, it's a fuel decision. Reasoning budgets and small-model tiers stop being optimizations and become the product. **Your update channel is your lifeline.** The app has to update models, library, and binary in one click without breaking the local state, because the next time the user has bandwidth might be months away. And background resource discipline matters in a new way: a fix in one release made Cmd+Q actually stop the inference engine, which had kept a multi-gigabyte model resident in memory after the window closed. On battery, that's not a memory leak, it's a heat-and-hours leak. **User files join the trust boundary.** Letting users drop in their own PDFs — a generator manual, a family emergency plan — means your retrieval pipeline now handles adversarially messy input: scanned PDFs, weird encodings, huge files. Offline, you can't punt any of it to a cloud parser.

Why This Pattern Matters Beyond Emergencies

Privacy stops being a policy promise and becomes a consequence of the dependency graph. A program with no network calls can't leak anything, ever.

The emergency framing gets the headlines, but the architecture is the story. The same stack — local inference, curated corpus, retrieval-with-citations, hard compute budgets — is what you want for a clinic with patient data that legally can't leave the building, a factory floor with air-gapped machines, a journalist in a country where a query log is a liability, or a research ship with satellite internet billed by the megabyte. One practitioner in the discussion described exactly this: a manually curated library plus local models on property with no cell reception, refreshed before each trip, gradually replacing most of their day-to-day cloud AI use even when fiber is available. That's the trajectory I'd bet on. The gap between a quantized 8B local model and a frontier API model is real but narrow enough that for retrieval-anchored tasks — "find me the right passage and summarize it faithfully" — the local option is often sufficient, and sufficiency plus zero marginal cost plus guaranteed privacy is a compelling package. The frontier models keep getting better, but so does quantization, and small models ride the same scaling wave with a lag. If you're tracking the efficiency end of that curve, flash-tier models are where the real inference efficiency story is. My advice if you're designing anything in this space: steal the pattern, not the prepper aesthetic. Curate ruthlessly — a library of 200 documents you trust beats 930,000 you haven't read. Make citations mechanical, never model-generated. Give the model a hard budget for thinking. And decide, explicitly, whether your model is an oracle or a search interface. The tools that survive scrutiny are the ones honest enough to be the second one.

The Takeaway

Offline-first AI isn't about surviving the apocalypse. It's about discovering what your architecture looks like when every escape hatch — a bigger model, a cloud fallback, an always-on network, unlimited power — is welded shut. What remains is a curated corpus, a quantized model sized to hardware you don't control, retrieval that carries the trust, and an agent loop built to fail gracefully. Those turn out to be good disciplines online, too. The storm just makes you learn them faster.