← Back to all posts

Harnessing Agentic AI Systems: Semantic Memory Router Pattern

Problem 6 of 15: choosing what context to inject. The Semantic Memory Router pattern vs the RAG Firehose anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesmemoryretrievalrag

Problem 6 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Rolling Window Compression Pattern · Next: State Snapshot & Rollback Pattern.

The Problem — Choosing what context to inject

Not every retrieved fact belongs in every prompt. Retrieval without judgment drowns the instruction — and every injected chunk is untrusted input. The system must decide, and the decision cannot be the model's.

Field P6 — Semantic Memory Router (pattern) A6 — The RAG Firehose (anti-pattern)
Forces / Smell Grounding vs focus; freshness vs stable prefix; utility vs untrusted input. Top-K chunks by raw keyword match; the instruction buried; retrieval dominating the prompt.
Solution / Anti-solution Intercept ongoing tasks, query vector stores, and inject context fragments just-in-time into the agent's prompt. "More chunks equals better grounding."
Consequences / Failure Grounded, lean prompts; attention curated by the system rather than dumped by default. Injected context is the highest-leverage observation — and the highest-leverage attack. Drowns the instruction where Lost in the Middle predicts; every injected chunk is untrusted input — an injection vector (OWASP LLM08).
Tradeoffs / Refactoring The router is an injection surface; just-in-time injection fights prefix-cache discipline; retrieval quality is decided by chunking, metadata filtering, and reranking, not top-K volume. P6 with chunking, metadata filtering, and reranking deciding what is injected.
Evidence Pinecone's RAG guides (learn); the always-on survey's provenance and authority axes (always-on agents). Pinecone advanced RAG (learn); OWASP LLM08.
Related Refactoring for A6; composes with P8 (the tiers are the router's stores). Is the absence of P6; retrieval form of A4.

Discussion

The router is the decision layer RAG was missing: not every retrieved fact belongs in every prompt, and the decision cannot be the model's because the model cannot see what it was not shown. The mechanism is curation — chunking, metadata filtering, reranking — and the security corollary is that retrieved content is untrusted input: the router is both the grounding layer and the injection surface.

Key Insight

The system curates what the model sees. Injected context is the highest-leverage observation and the highest-leverage attack — retrieval quality is a decision, not a volume, and the decision cannot be the model's.

References

Pinecone RAG guide (learn) and advanced RAG (learn); OWASP Top 10 — LLM08 (2025); Liu et al., Lost in the Middle (arXiv:2307.03172); archive: Agentic-First CLI, always-on agents.