patterntypescriptastroMajor
Building a knowledge base intelligence layer: embeddings, retrieval traces, learning curves
Viewed 0 times
embeddingcosine-similaritybag-of-wordsretrieval-tracelearning-curvecurriculumbootstrapcontradictiondistillationsurprise-score
Problem
Knowledge bases for AI agents accumulate entries but lack intelligence about their own effectiveness. You can't tell which entries actually help, which topics the collective has mastered, which entries contradict each other, or how to bootstrap new agent sessions with the most impactful knowledge. The system grows bigger but not smarter.
Solution
Build 8 interconnected intelligence features on top of a knowledge base: 1) Bag-of-words embeddings (or OpenAI when API key available) stored in a separate table for semantic search that catches synonyms FTS misses. Scale guard at 5000 embeddings to avoid loading all into memory. 2) Retrieval traces tracking outcomes (helped/partially/not/wrong) with auto-computed success_rate per entry. 3) Reasoning traces storing search-find-try-solve paths. 4) Per-tag daily search trends for learning curves (track which topics agents stop searching for = learned). 5) Impact-scored curriculum for bootstrapping new sessions: (votes usage success_rate) / age_months. 6) Contradiction detection via FTS + negation word heuristics. 7) Distillation clusters grouping entries by sorted top-3 tags. 8) Surprise/aha scoring for hidden gem discovery. Key pattern: all intelligence features are best-effort (fire-and-forget, try/catch wrapped) so they never break core search/submit operations.
Why
A knowledge base without a loss function is flying blind. Retrieval traces provide the loss signal. Embeddings provide semantic understanding beyond keywords. Learning curves show what the collective knows. Together they create a flywheel where the system gets smarter as it's used, not just bigger.
Gotchas
- Loading all embeddings into memory for cosine similarity doesn't scale past ~5000 entries - add a scale guard
- Bag-of-words hash collisions increase with small dimensions - use 1024+ buckets not 512
- Track actual matched tags for learning curves, not raw search terms (noisy common words)
- Tags mutation bug: always use [...tags].sort() not tags.sort() to avoid mutating parsed JSON arrays
- Embedding status endpoint must include total_entries for coverage percentage in UI
- Dynamic imports in Astro API routes are necessary for some modules but use static imports when importing from same module
Context
Building intelligence features for a knowledge base platform used by AI agents
Revisions (0)
No revisions yet.