Your RAG System Won't Know What Your Org Cares About (Unless You Tell It)

Enterprise RAG deployments are hitting a wall: demos pass, production fails silently. Teams blame retrieval quality on embeddings, rerankers, or context windows. They tune prompts. They swap models. They add guardrails.

Nothing helps—because the problem isn't downstream. It's upstream.

Why This Problem Is Showing Up Now

Two years ago, "RAG" meant demos with Wikipedia. Today, enterprises are feeding internal docs—architecture standards, compliance policies, onboarding guides, runbooks—into production RAG systems.

Here's the mismatch: most documentation is normative, not descriptive. It doesn't just say what something is; it says what you should do, what's required, and what exceptions exist.

• Descriptive: "Python is a programming language."
• Normative: "All new services MUST use Python 3.11+ unless approved by the platform team."

Generic chunkers have no concept of the difference. They split text by token count or paragraph breaks. They don't know that "MUST" is a constraint, that "unless approved" is an exception path, or that the entire paragraph is a policy—not a suggestion.

The Hidden Assumption Every Chunker Makes

Every chunker makes an implicit assumption: any chunk, in isolation, is retrievable and useful.

That assumption is false for normative content.

Example: A security policy

"All API endpoints must require authentication.
Exception: /health and /metrics endpoints may be unauthenticated
for monitoring purposes. Exception approval: ops-security@."

If chunking splits this into:
  Chunk 1: "All API endpoints must require authentication."
  Chunk 2: "Exception: /health and /metrics endpoints may be..."

Retrieval for "Does /health need auth?" returns Chunk 1.
Answer: "Yes, authentication is required."
Correct answer: "No, /health is an approved exception."

The chunker destroyed the meaning by separating the rule from its exception. No embedding model can recover from this. No reranker. No prompt.

What Embeddings Actually Capture (And What They Can't)

Embeddings are powerful at semantic similarity. They answer: "Does this chunk sound like the query?"

They do not answer:

• Is this chunk a rule or an example?
• Is this chunk current or deprecated?
• Does this chunk represent mandatory policy or optional guidance?
• Does this chunk require context from adjacent chunks to be correct?
• Is this chunk complete or a fragment?

These are organizational opinions. They don't exist in the text. They exist in the heads of the people who wrote the text—and they need to be encoded explicitly.

Treating Chunking as a First-Class System

Most teams treat chunking as a preprocessing step: run once, never revisit. That's backwards. Chunking should be:

1. Version controlled

Your chunking logic is code. It should live in source control, be reviewed, and evolve with your documentation. When someone updates a policy doc, the chunking should be re-evaluated.

2. Document-type aware

A runbook and a compliance policy have different structures. A config file and an architecture decision record have different semantics. One chunking strategy doesn't fit all.

Document Type       Chunking Strategy
─────────────────────────────────────────────────
Policy docs         Rule + exceptions as atomic unit
Config files        Entire file or logical section
Runbooks            Step sequences preserved
ADRs                Decision + rationale together
API docs            Endpoint + params + examples

3. Metadata-enriched

Every chunk needs organizational metadata that embeddings can't infer:

{
  "chunk_id": "sec-policy-auth-001",
  "doc_type": "security_policy",
  "content_type": "rule_with_exceptions",  // not just "text"
  "status": "approved",
  "effective_date": "2024-01-15",
  "owner": "security-team",
  "requires_whole_doc": false,
  "supersedes": ["sec-policy-auth-000"]
}

4. Tested

If you have a test suite for your API, you should have a test suite for your chunking. Given this document, does chunking produce the expected units? Given this query, does retrieval return the right chunk?

Patterns That Encode Organizational Knowledge

Semantic chunking with structure awareness

Don't chunk by token count. Parse document structure (headings, lists, code blocks) and chunk by semantic units. A policy section stays together. A config block stays together.

Parent-child relationships

When you must split, preserve relationships:

{
  "chunk_id": "exception-health-endpoint",
  "parent_chunk": "rule-api-auth-required",
  "relationship": "exception_to",
  "retrieve_with_parent": true
}

Content-type tagging

Tag chunks with what they represent: rule, example, exception, deprecated, reference. Use these tags in retrieval filtering.

Whole-document markers

Some content only makes sense as a whole: config files, terraform modules, example implementations. Mark these as "retrieve entire document" and don't chunk them at all.

Temporal validity

Policies have effective dates. Standards have versions. Mark chunks with temporal metadata and filter by recency during retrieval.

A Practical Chunking Checklist

Before you deploy (or redeploy) your RAG system:

• Audit your document types. What kinds of content are you indexing? What structure does each type have?
• Identify normative content. Which documents contain rules, constraints, or policies (vs. descriptions or references)?
• Map exceptions to rules. For every constraint, is the exception co-located in the same chunk?
• Test retrieval, not just generation. Given 20 representative queries, does retrieval return the right chunks? (Not: does the LLM produce a plausible answer.)
• Version your chunking logic. When docs change, does the chunking get re-evaluated?
• Add organizational metadata. Status, owner, doc_type, content_type, effective_date—at minimum.
• Define "retrieve together" relationships. Which chunks must be retrieved as a unit?

The Bigger Picture

The pattern repeats across AI systems: teams focus on model selection, prompt engineering, and output quality—while data modeling problems silently corrupt everything downstream.

RAG isn't different. The "R" in RAG is retrieval, and retrieval quality is bounded by chunking quality. If the right chunk doesn't exist, retrieval can't find it. If the chunk is incomplete, the LLM will fill in the gaps with hallucination.

What's one document type in your system where chunking is definitely wrong? Start there.