Your AI Is Only as Smart as the Data Architecture Behind It
Every company adding AI right now has RAG on the roadmap. Most of them will build it wrong. Not because the retrieval is bad or the model is weak, but because they never asked the question that actually determines whether RAG works: what is the AI retrieving from?
The RAG problem nobody is solving
The standard approach: take your documents, chunk them, embed each chunk into a vector, and search by similarity when a question comes in. The closest chunks get stuffed into the prompt. The model generates an answer.
This works for demos. It does not work for building intelligence a business can depend on.
A flat vector store has no concept of recency. A document about a client's requirements from eight months ago carries the same weight as a conversation from yesterday. There is no mechanism to say "this fact replaced that fact," so the AI retrieves outdated information and presents it with equal confidence.
A flat vector store has no relationships. It cannot traverse from a client to their projects to the contacts who manage the accounts. It can only find text that looks similar to your question. If the answer requires connecting information that was never stated together in one document, flat RAG cannot find it.
And a flat vector store does not self-correct. Duplicate entities proliferate. Contradictory facts accumulate. There is no merge mechanism, no audit trail. The data gets messier over time, not cleaner.
The model is not the bottleneck. The data architecture is. In previous posts I covered agentic workflows, owning your software, and the engineering discipline behind safe AI-assisted development. This post is about the layer underneath all of it.
Ontology vs. knowledge graph, and why you need both
These terms get used interchangeably, and they should not be.
An ontology is the schema. It defines what types of things can exist in your domain and what relationships are permitted between them. A client can have contacts. A project belongs to a client. A proposal references skills. The ontology contains no data. It contains structure.
A knowledge graph is the populated instance: the actual entities and relationships that conform to the ontology. "Acme Corp" is a client entity. "Jane WORKS_AT Acme" is a directed edge with metadata for her role.
An ontology is to a knowledge graph what a database schema is to the data in the tables. Without a clear ontology, you end up with inconsistent types, duplicate entities, and edges that mean different things in different contexts.
In the system I built for my own business operations, the ontology makes a deliberate distinction between two categories:
Structured entities are backed by rows in a source table. A client exists because someone created a client record. These entities have authoritative source data.
Concept entities are extracted by AI from unstructured text: an organization mentioned in a job description, a person referenced in meeting notes, a skill discussed in a cover letter. They have no source row. They exist because the system inferred them from natural language.
Structured entities are authoritative. Concept entities are probabilistic. The system treats them differently in search, merging, and confidence scoring, and that distinction is baked into the schema, not handled ad hoc in application code.
The production ontology: eleven structured types (client, contact, project, proposal, job post, conversation, message, campaign, prospect, user, blog post), five concept types (organization, person, location, skill, topic), and fourteen predicates (WORKS_AT, PRIMARY_CONTACT_OF, LOCATED_IN, FOR, LINKED_TO_CLIENT, HAS_CONTACT, SUBMITTED_BY, DISCUSSES, USES_SKILL, ABOUT_TOPIC, PUBLISHED_BY, COVERS_TOPIC, CREATED_BY, REFERS_TO). Everything else builds on that foundation.
The five tables that run everything
I built this knowledge graph as the spine of my own operations platform. The CRM, the sales pipeline, the chat interface, the content management: all of it reads from and writes to the same graph. Here is the actual schema:
kg_entities
id, entity_type, canonical_name, display_name
summary, metadata (JSONB), embedding (vector 1024-dim)
merged_into (self-referencing FK, tombstone pattern)
source_table, source_id (NULL for concept entities)
HNSW index on embedding; trigram index on canonical_name
kg_edges
from_entity_id, to_entity_id, predicate
metadata (JSONB, e.g. {"role": "VP Engineering"})
Unique on (from, to, predicate)
kg_facts
entity_id, predicate, value_text / value_numeric / value_json
confidence (0.0 to 1.0), source_type, source_id
observed_at, superseded_at
Current facts: WHERE superseded_at IS NULL
kg_events
event_type, entity_id, payload (JSONB), processed_at
Immutable append-only log; doubles as enrichment work queue
kg_entity_merges
loser_id, winner_id, reason, merged_at
Audit trail for every mergeFive tables. That is the entire knowledge graph. Three design decisions matter more than the others.
First: embeddings live on the entity table itself. Every entity, regardless of type, occupies the same 1024-dimensional embedding space, generated by Amazon Titan v2 and stored in PostgreSQL via the pgvector extension with HNSW indexing. A client, a skill, a topic, and a past conversation all exist in the same vector space, so one query searches across every entity type. The system does not need to know in advance whether the answer involves a client, a project, or a skill.
Second: facts are append-only with supersedure. When a contact gets a new title or a project status changes, the old fact is never deleted. A new fact is inserted and the old one's superseded_at timestamp is set. Retrieval filters to current facts, but the full timeline is always available. This is what lets the AI reason about what is true now versus what was true six months ago, and it makes every fact traceable to a source, a timestamp, and a confidence score.
Third: the tombstone pattern for merges. When two entities turn out to represent the same real-world thing, one merges into the other. The loser's merged_into field points to the winner, all edges and facts migrate, and the merge is recorded for audit. Nothing is deleted. The graph heals itself while preserving a complete history of how it got there.
Structure plus meaning
A knowledge graph without embeddings answers structured questions: "Give me all projects for Client X." Embeddings answer a different kind entirely: "What was that thing we discussed about the migration timeline?" Structure gives you precision. Embeddings give you recall. Together they give you an AI that navigates business data the way a senior team member does: following explicit connections when they exist, making associative leaps when they do not.
Two separate layers do this work, and you can swap either independently. The embedding model converts text to vectors. I use Titan v2 because it runs inside AWS, the same account that hosts everything else, so client and financial data never leave the security perimeter. The vector database stores and searches those vectors. I use pgvector inside the same PostgreSQL database where the entities and facts already live: no separate vector store, no synchronization, everything queryable in one transaction.
The similarity thresholds were tuned empirically against real queries: 0.55 for entities (broad recall across diverse types), 0.65 for past messages (conversations need to be genuinely relevant, not tangential). And because entities and chat messages share the same embedding space, a single search finds both the relevant business objects and the relevant past conversations about them. Conversational memory comes for free.
The pipeline behind every question
Here is what happens every time someone asks a question in the internal chat:
1. Embed the message (Titan v2, 1024-dim)
2. Vector search kg_entities: top 10, similarity > 0.55,
skipping tombstoned entities
3. For each match: load current facts, name, type, summary
4. Vector search past messages: top 5, similarity > 0.65
5. Focus boosting: if the conversation was opened from a
specific client or proposal, always include that entity
and its 1-hop neighbors regardless of similarity score
6. Format the context block into the system prompt
7. LLM generates a response grounded in business contextThe focus boosting in step 5 ensures that a conversation about a specific client never loses that thread even when the latest message drifts. And here is roughly what the model sees after retrieval:
## Relevant Knowledge
- **Acme Fintech** (client)
Status: active | HQ: San Francisco | Rate: $200/hr
Edges: Jane Doe (WORKS_AT, VP Engineering),
Platform Migration (FOR, project)
- **Jane Doe** (contact)
Role: VP Engineering | Confidence: 1.0 (structured)
## Relevant Past Conversations
- [Acme Fintech: Scoping Call] (2026-01-22):
"They need RAG for their underwriting pipeline..."This is not a wall of chunked text. It is typed knowledge with relationships, confidence scores, and provenance. When the AI states a client's rate, that fact exists in the graph with confidence 1.0, sourced from the client record, and not superseded by anything newer.
One production principle is critical: the whole pipeline fails open. If the vector search fails or the embedding service is slow, the chat still works, just with less context. The graph enhances every interaction and blocks none of them.
How the graph learns
Data enters at the application layer. When a handler creates a client or submits a proposal, it also upserts the entity, asserts the edges that are deterministic from source data, asserts facts, and emits an event. Every one of those calls is wrapped in try/catch: if the graph write fails, the user action still succeeds. Fail-open on the write path too, which means you can instrument an existing application one entity type at a time without risky deployments.
The enrichment worker is where the graph becomes more than a mirror of your forms. Every two minutes, a scheduled trigger processes up to twenty pending events. For events carrying unstructured text (a job description, meeting notes, a chat message), Claude extracts organizations, people, locations, skills, topics, and facts with confidence scores. Each becomes a concept entity connected to its source by typed edges. Paste in a job description mentioning "a Series B fintech in San Francisco needing React and AWS Bedrock experience," and the graph now knows things nobody explicitly entered: the skills are searchable, the location traversable, the topic connected to every other fintech conversation.
Then comes concept promotion, the self-healing piece. Say "Acme Fintech" was extracted as a concept from that job description. Three weeks later the deal closes and a user creates a proper client record. The system detects the match with dual-signal verification: trigram name similarity of at least 0.85 and cosine embedding similarity of at least 0.92, both required. That conservatism is deliberate. False merges corrupt the graph, and you cannot cleanly un-merge; duplicates can always be merged later. On promotion, every edge and fact migrates to the client record, the concept is tombstoned, and the merge is logged. All the context absorbed from the original description and conversations now lives on the authoritative record.
The subsystem is covered by 87 tests across 12 test files: every merge path, promotion threshold, supersedure chain, and the fail-open guarantee itself. You cannot afford to discover merge bugs in production when your business intelligence depends on the graph.
What this looks like in practice
A prospect emails: a VP of Engineering at a SaaS company who needs AI features. You talk, take notes, move on. Without a graph, that context fragments across your inbox, a notes app, and memory. Six months later you are reconstructing it from scratch.
With the graph, the enrichment worker extracts the person, their company, the skills discussed, and the topics, and links them to the conversation. When you later create a prospect record, the concept promotes into it. Six months on, you ask the chat, "What do I know about this prospect?" and get back something like:
"Sarah Chen is the VP of Engineering at Meridian Health, a Series B healthtech company in Boston. You spoke in October about a clinical document pipeline using RAG. They run on AWS and need HIPAA-compliant infrastructure. No proposal was submitted. She was evaluating two other vendors."
Nobody typed any of that into a CRM field, and every claim traces back to a source entity with a confidence score and a timestamp. That is the difference between a system that stores information and a system that understands it.
Why flat RAG degrades and graphs compound
The common objection: "Why can't I just embed my documents and search by similarity?" You can, and it will work for about six months.
Month one: you embed everything and the answers are surprisingly good. Month three: the store holds old and new versions of the same information, and the AI retrieves the wrong one often enough to erode trust. Month six: three versions of the pricing document exist as chunks, former employees surface as current, and a cancelled project shows up as active because the cancellation was discussed in Slack rather than in the embedded doc. Data without structure becomes noise, and the quality ceiling drops as volume grows.
A knowledge graph inverts that trajectory. Every new entity adds connections that make everything it touches more findable and more contextual. Superseded facts drop out of retrieval but stay auditable. Duplicates merge and the graph heals. A graph with two years of business context is not just bigger than a six-month-old one. It is qualitatively better.
Building this for your own operations
The specifics vary by domain and stack, but the principles hold.
Start with the ontology. Before writing code, name your entity types and predicates explicitly. Get this right and everything downstream is cleaner. Skip it and you will clean up inconsistent data for the life of the system.
Choose embedding model and vector storage by where your data lives. Business data should not cross your security boundary for vectorization. On AWS, Titan keeps it inside the perimeter; open-source models like BGE or E5 run on your own infrastructure. For storage, if you run PostgreSQL, pgvector is the path of least resistance; MongoDB has Atlas Vector Search. Dedicated vector databases work, but keeping vectors beside your structured data eliminates an entire category of consistency bugs.
Build the event pipeline early. The enrichment worker is what turns a static graph into a living one. Start with entity creation and mutation events; add unstructured extraction once the foundation is stable.
Set merge thresholds conservatively. Require multiple signals. Loosen only when you have data to justify it.
Make it fail-open. The graph should enhance everything and block nothing. Intelligence is additive, never a bottleneck.
The data layer is the competitive moat
Companies adding AI today are focused on the model: which LLM, which API, which prompting technique. That focus is misplaced, because models are commoditizing. Switching models is a configuration change. The model is a replaceable component.
The data layer is not. A knowledge graph with years of enriched context, refined merges, and tuned confidence across thousands of facts is not something a competitor replicates by signing up for the same API. It is the compound asset. The graph feeds the agentic workflows with context, the platform you own gives you freedom to build the graph your business needs, and engineering discipline keeps it reliable. Remove the graph and the AI is guessing.
Every engagement, project, and conversation adds to the graph, so the intelligence compounds because the data compounds. Starting a year from now means starting a year behind.
Start with the ontology. Build the graph. Layer intelligence on top. Your AI is only as smart as the data behind it.