Lessons From Apple's On-Device Knowledge Graph
Apple's on-device knowledge graph shows how to ship private, local-first ML. Five mistakes practitioners make — and the fixes that actually work.

Tell your Mac to "text my brother" and it just does it. No spinner, no visible lookup, no network round-trip — it already knows which of the hundreds of people in your contacts is your brother. That works because every Mac since macOS Ventura has been quietly running an on-device knowledge graph: a pair of daemons that read your Contacts, Photos face clusters, Calendar attendees, Mail and Messages metadata, location patterns, and Screen Time data, then fuse all of it into a queryable store of who you are, who you know, and how those people relate to you and each other.
Security researchers recently pulled the curtain back on how deep this goes. The two daemons — knowledgeconstructiond and intelligenceplatformd — maintain a directory of 51 SQLite databases under ~/Library/IntelligencePlatform. The centerpiece, graph.db, is a triple store written in a vocabulary of 181 entity classes and 278 predicates, including inferred kinship relations that nobody ever typed into a contacts card. The loudest debate around this discovery is about consent, and it's a fair debate. But I spent years shipping ML features for a living, and when I read the teardown my first reaction was different: this is one of the most instructive pieces of local-first ML engineering ever shipped at scale. So instead of relitigating the ethics, I want to walk through the five mistakes practitioners make when building systems like this — and what Apple's engineers demonstrably got right and wrong, so you don't have to learn it the hard way.
Mistake 1: Reaching for One Big Model Instead of Small Classifiers
The first instinct most teams have when they hear "the OS understands your family" is to assume there's a large language model in the loop. There isn't, and that's the lesson. The kinship inference that made headlines — the system deciding that this contact is your mother, that one is your boss — is done by narrow, specialized classifiers chewing on structured signals: Family Sharing membership and ages, the related-names fields in Contacts, calendar co-attendance patterns, communication frequency and directionality, face co-occurrence in Photos. Each classifier answers one small question well, and the graph stitches the answers together.
This matters for a brutally practical reason: budget. A daemon that runs continuously, on battery, alongside everything else the OS is doing, cannot afford a transformer forward pass per event. Small classifiers run in microseconds on the Neural Engine, they're deterministic enough to unit-test, and when one misfires you can retrain that one without touching the rest. I've watched teams burn quarters trying to make a general model do what a gradient-boosted tree over thirty well-chosen features does better. We covered the same phenomenon in vision: specialist CNNs routinely beat frontier LLMs on image classification at a fraction of the cost. The pattern holds everywhere. Personalization features almost never need a general model — they need a good schema and a pile of boring, excellent classifiers.
The other underappreciated benefit is graceful degradation. If the face-recognition pipeline never runs because the user doesn't use Photos, the kinship classifier still has Family Sharing and Contacts to work with. Monolithic models degrade catastrophically when their input distribution shifts; an ensemble of signal-specific classifiers just loses a vote. When you design your next on-device feature, enumerate the independent signals first, then ask what the smallest model per signal looks like. Save the LLM for the last mile of natural-language interaction, where it actually belongs.

Mistake 2: Storing Facts Without Provenance or Retraction
Here's a detail in the teardown that most coverage skipped past, and it's the part I find most elegant. An edge in this graph — "this person works at this organization" — is not one row. It's two rows sharing a relationshipId: one row names the relationship type, the other points at the related entity. There are 46 relationship-type classes that exist purely to carry this indirection. That's called reification, and the reason you do it is so the edge itself can carry properties — a confidence score, a source, a timestamp — and so it can be retracted later without pretending it never existed.
This is the mistake I see most often in homegrown personalization systems: facts stored as flat columns. user.employer = "Acme". It works until someone changes jobs, gets divorced, or the inference was just wrong, and now you're either overwriting history you needed for debugging or leaving stale facts in place because deleting them breaks something downstream. Personal facts decay fast. People move, relationships end, employers change. A store that can't express "we believed this with confidence 0.7, sourced from Calendar attendees, retracted on March 3rd" will quietly fill up with ghosts. The schema that handles it looks roughly like this:
-- A reified edge: two rows share one relationship id,
-- so the edge itself can carry confidence and provenance.
CREATE TABLE triples (
subject TEXT NOT NULL, -- an entity id
predicate TEXT NOT NULL, -- e.g. 'relationshipType', 'relatedEntity'
object TEXT NOT NULL, -- a value or another entity id
rel_id TEXT, -- groups the rows forming one edge
confidence REAL, -- how sure are we?
source TEXT, -- which signal produced this fact
created INTEGER NOT NULL,
retracted INTEGER -- soft delete: facts decay, history stays
);
-- One edge, two rows:
-- ('person:8f2a', 'relationshipType', 'employerOf', 'rel:91', 0.82, 'calendar', ...)
-- ('person:8f2a', 'relatedEntity', 'org:acme', 'rel:91', 0.82, 'calendar', ...)
Note what the implementation choice also tells you: this whole thing lives in plain SQLite. No exotic graph database, no custom storage engine — just tables you can query with the most battle-tested embedded database on earth. If you're tempted to reach for a dedicated graph store for an edge count in the low millions on a single device, don't. We've written about where embeddable graph databases earn their keep beyond SQLite, and the honest answer is that most applications never hit that point. Triples in SQLite, with disciplined indexing on subject and predicate, will take you astonishingly far. Spend your complexity budget on provenance and retraction semantics instead — that's where these systems actually break.
Mistake 3: Treating Entity Resolution as a Solved Detail
Every design doc I've ever read for a system like this has a bullet point that says "then we unify the entities," as if it's an afternoon's work. It's the hardest part of the whole pipeline by a wide margin. Your Contacts say "James Whitfield." Photos has a face cluster with no name. Calendar has an attendee "j.whitfield@company.com." Messages has a handle "+1-415-555-0142." The graph's value depends entirely on whether those four observations become one person — and on whether they correctly don't merge with the other James, the one from your running club.
The pragmatic architecture is incremental clustering with asymmetric error costs. A false merge — deciding two different people are one — poisons every downstream inference and is miserable to unwind, because facts from both real people are now tangled on one node. A false split just means the graph knows a little less until more evidence arrives. So you set your merge threshold high and let shared hard identifiers (a phone number appearing in two sources) do most of the work, with soft signals like name similarity and co-occurrence only breaking ties. The core loop is boring on purpose:
# Incremental entity resolution: precision over recall.
# A false merge corrupts everything downstream; a false split
# just means you learn slower. So merge only on strong evidence.
HARD_SIGNALS = {"phone", "email", "contact_id"} # near-certain identity
MERGE_THRESHOLD = 0.95 # soft-signal bar, set high
def ingest(observation, graph):
candidate = find_candidate(observation, graph)
if candidate is None:
return graph.new_entity(observation)
if shares_hard_signal(observation, candidate, HARD_SIGNALS):
return graph.merge(observation, candidate, confidence=1.0)
score = soft_match_score(observation, candidate) # name sim, co-occurrence
if score >= MERGE_THRESHOLD:
return graph.merge(observation, candidate, confidence=score)
return graph.new_entity(observation) # when in doubt, split
Two follow-on rules that save you later. First, every merge decision is itself a fact with provenance — record which signals justified it, because when a user says "that's not my sister, that's my colleague," you need to unmerge surgically, not delete the node. Second, never let a single classifier's output merge entities on its own; require corroboration from an independent signal. One of the quietly impressive things in Apple's design is exactly this: kinship, employment, and communication edges all hang off the resolved entity, so a resolution error is contained instead of amplified — provided you got the resolution right in the first place. Budget half your timeline for this. You'll need it.
Mistake 4: Assuming "On-Device" Means the Privacy Work Is Done
Now the part where I stop praising the engineering. The standard defense of this architecture — and it dominated the public reaction — is "it never leaves your device, so it's fine." That's half an argument. Local-first ML genuinely solves the data-residency problem: there's no server-side copy to breach en masse, no ad-tech pipeline slurping up your social graph, no subpoena target of a hundred million users at once. That is a real, meaningful win over how every other platform company does personalization, and I'll defend it.
But aggregation is its own risk multiplier, and local storage does nothing about it. Any single source — your contacts, your calendar, your messages metadata — is sensitive in a bounded way. The fused graph, which knows your employer, your family tree with ages, your communication cadence with each person, your movement patterns, and your daily activities, is categorically more valuable than any of its inputs. And this particular store sits in plaintext SQLite, protected only by Full Disk Access, undocumented except for a two-sentence man page, with no per-feature toggle and no obvious way to opt out. Forensics tooling already reads the iOS copy. The researchers behind the teardown ran the obvious thought experiment: hand that store to malware, or to an abusive partner with the device password, and you have a turnkey dossier — who you talk to, how often, where you go, who your children are.
Local-first solves the data-residency problem. It does nothing for the aggregation problem — a fused dossier on disk is still a dossier.

The fixes are not exotic, which is why their absence stings. Encrypt the store at rest with a key tied to user presence. Ship a visible toggle — not "Siri Intelligence" as a monolith, but granular control over which signals feed the graph. Document the schema. And treat inferred facts as more sensitive than raw ones, not less: nobody consented to "this system deduced your daughter's age," even though they technically granted access to the inputs that enabled it. If you build local-first ML, say this sentence to yourself in design review: the fact that we can't see the data doesn't mean nobody ever will. Plan for the day the device is compromised, seized, or sold, because one of those will happen to some fraction of your fleet.
Mistake 5: Shipping Inference Without an Inspection Path
Consider how we know any of this exists. Not from documentation — the official description is that terse man page. We know because independent researchers reverse-engineered the databases and built an open-source dashboard so users can read their own graph, and a forensics maintainer taught his tool to parse the iOS version. Users discovered inferred kinship labels about their own families by running third-party code against their own machines. That's a failure mode, and it's one engineering teams walk into with their eyes open: the system produces conclusions about a person, and the person has no first-party way to see, correct, or delete them.
This isn't just an ethics checkbox; it's an operational one. If you can't see the graph, you can't debug it. When a user reports "your assistant thinks my ex is my wife," the only way to answer why is to trace the edge back through provenance to the signal that produced it. Teams that build the read path early — an internal viewer, an export button, an "explain this" affordance on every inferred fact — ship better models, because wrong inferences become visible bug reports instead of silent resentment. Teams that don't end up with users screenshotting reverse-engineered dashboards on social media, which is a significantly worse look than any settings page would have been.
My rule of thumb, learned the hard way: for any feature that draws conclusions about a user, the inspection UI is not a nice-to-have that follows in a later release. It ships in the same release or the feature doesn't. The cost is a settings pane and a JSON export. The payoff is trust you don't have to beg for in a press release. And practically speaking, regulators are circling exactly this gap — systems that infer sensitive attributes about people owe those people visibility, and "it's on-device" will not be a recognized defense for invisibility.
What to Steal for Your Next Local-First ML Feature
Strip away the controversy and this is the architecture I'd hand to any team building personalization that can't phone home — and honestly, most personalization shouldn't phone home. The blueprint, condensed:
- Many small classifiers over one big model. One per signal, each independently testable, retrainable, and cheap enough to run on battery forever.
- A triple store with reified edges. Every fact carries confidence, source, and timestamps; every fact can be retracted without erasing history. SQLite is enough.
- Entity resolution tuned for precision. False merges are the catastrophic error; false splits just slow you down. Corroborate before you unify.
- Privacy engineering that goes past residency. Encryption at rest, granular consent, retention limits — because aggregation creates a target no single input ever was.
- An inspection path from day one. If the system concludes things about the user, the user gets to see, correct, and delete those conclusions.
If you want a concrete starting point for the serving side, we've written about building LLM features that work fully offline, and the same discipline applies here — assume the network is gone, the power budget is small, and the user owes you nothing. The uncomfortable coda is that Apple's engineers built something genuinely admirable and then undermined it with silence. The classifiers, the schema, the restraint to do inference locally instead of shipping everything to a data center: that's the part worth copying. The undocumented, always-on, opt-out-by-terminal-command posture is the part worth avoiding. Build the graph. Tell the user it's there. Both halves are engineering, and the second one is not optional.


