{"id":23704,"date":"2026-09-02T09:00:29","date_gmt":"2026-09-02T09:00:29","guid":{"rendered":"https:\/\/scannn.com\/an-organizational-second-brain-building-an-ai-that-learns-from-experts\/"},"modified":"2026-09-02T09:00:29","modified_gmt":"2026-09-02T09:00:29","slug":"an-organizational-second-brain-building-an-ai-that-learns-from-experts","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/an-organizational-second-brain-building-an-ai-that-learns-from-experts\/","title":{"rendered":"An Organizational Second Brain: Building an AI That Learns From Experts"},"content":{"rendered":"\n<div>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">We\u2019ve built an AI agent that acts as a secondary expert for a given domain, making deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">This is not a typical domain-specific agent. Its novelty comes from integrating two layers:\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">A structured, auditable <\/span><b>knowledge architecture<\/b><span style=\"font-weight: 400;\"> separates what the agent knows from how it reasons.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">A <\/span><b>self-improvement loop<\/b><span style=\"font-weight: 400;\"> then compiles expert feedback into verified, regression-tested updates <\/span><b>without model retraining<\/b><span style=\"font-weight: 400;\">.\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Together, these two layers turn one-off expert corrections into permanent, compounding institutional memory, and the pattern is designed to generalize to other domains governed by retrievable text rather than model weights.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">This system is saving domain subject matter experts (SME)s at Meta substantial time, allowing them to focus more on the work where their knowledge matters most.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Many large organizations have the same problem when it comes to specialist knowledge. While some of it is written down in the form of models, playbooks, checklists, and frameworks, the most valuable specialist knowledge lives in people\u2019s heads and rarely gets captured anywhere durable. In compliance domains, for example, the same types of questions can arise across hundreds of product reviews, expert assessments take days of manual research, and inconsistency between assessments creates real organizational risk.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It\u2019s not uncommon for experts to spend more time answering routine questions than on genuinely novel and ambiguous work where their judgment matters most. We need systems that can capture how an organization\u2019s experts reason and make that knowledge available to everyone who needs it, so that expertise is easier to share, build on, and preserve.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\"><br \/><\/span><span style=\"font-weight: 400;\">We set about solving this challenge by codifying institutional intelligence into an AI agent for a specific compliance domain. The agent combines a knowledge system that acts as the organization\u2019s \u201csecond brain,\u201d a reasoning layer that mirrors how domain experts actually think, and an automated improvement pipeline that compounds expert effort permanently. The patterns generalize to any enterprise domain with deep specialist knowledge, whether that is finance, security, or engineering.<\/span><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">The Architecture at a Glance<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Off-the-shelf LLMs provide a strong foundation, but they often need deeper institutional context to be fully effective in specialist domains. Without that grounding, a general purpose model has limited value given it will not be able to distinguish between what an organization could do (a summary of general information) and what it should consider doing (based on historic positions, company direction, business context, etc.). In high-stakes domains, closing this gap requires supplying the model with the organization\u2019s own knowledge and priorities so its analysis reflects how the organization actually reasons.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The system we\u2019ve designed has four layers, each solving a distinct problem:<\/span><\/p>\n<p><\/p>\n<p><span style=\"font-weight: 400;\">These layers depend on each other. The knowledge system\u2019s file structure makes automated editing possible. The reasoning layer\u2019s explicit procedures make failure attribution tractable. The evaluation framework gates every change. And the improvement loop feeds back into both knowledge and reasoning. Remove any one layer and the others degrade.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Building the Organizational Second Brain<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Large organizations can accumulate thousands of documents as a byproduct of expert work. It is tempting to treat those documents as organizational knowledge, but the real knowledge is implicit: how experts reason, what they prioritize, and how they resolve ambiguity. An agent that retrieves document chunks at inference time has to re-derive that reasoning from raw sources on every run, which is slow, error-prone, and inconsistent.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We make that implicit knowledge explicit ahead of time. A long-running offline process reasons through source documents and distills them into structured knowledge files \u2013 curated statements of how the organization interprets its domain, with constraints, boundaries, and routing implications made machine-readable.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Most significantly, that knowledge then forms the basis of a feedback loop that allows the agent to learn from and implement feedback from human experts without the underlying model having to be retrained.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The industry has converged on a similar idea. Andrej Karpathy\u2019s<\/span><a href=\"https:\/\/l.facebook.com\/l.php?u=https%3A%2F%2Fgist.github.com%2Fkarpathy%2F442a6bf555914893e9891c11519de94f&amp;h=AUCFK9lNTF7CgPxu9GE4kppMNNk7LorBrI7LY3GzOa3kMgTySl8uAd5nSl4jVfG0COztV4O13lh9Hy12nLeS0AgFYo18TgK-eXNXGfh_dSx9aV20Z-4F06PknKgl0_C6eXuOeiEeWu48ys5p\" target=\"_blank\" rel=\"noopener\"> <span style=\"font-weight: 400;\">LLM Wiki<\/span><\/a><span style=\"font-weight: 400;\"> structures agent knowledge as a navigable graph of files, and Google\u2019s<\/span> <a href=\"https:\/\/l.facebook.com\/l.php?u=https%3A%2F%2Fcloud.google.com%2Fblog%2Fproducts%2Fdata-analytics%2Fhow-the-open-knowledge-format-can-improve-data-sharing&amp;h=AUBMty2NM7KatAvutLJZPW0e0H4k0HLX6DBV7zD5kMh89ySEjle1BsVOkCmjjJAOkNSrUtBbT6we54TEY98iGZOWnGCCcLNz_zSRFtTvKjQccJnI2sRPsP7jLkwuqCLIR5-_Hncgljn43LXH\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">Open Knowledge Format<\/span><\/a><span style=\"font-weight: 400;\"> standardizes this for cross-agent interoperability. The shared insight is that knowledge should be pre-extracted, explicitly structured, and progressively disclosed rather than re-derived on every query. We extended these principles into a system where citation fidelity and institutional consistency are non-negotiable, organizing 200+ files into a strict taxonomy:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Position files<\/b><span style=\"font-weight: 400;\"> capture authoritative organizational stances: how the organization has decided to interpret a given domain question, along with its constraints, boundary conditions, and machine-actionable routing implications that tell the reasoning layer when to apply it.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Taxonomy and vocabulary files<\/b><span style=\"font-weight: 400;\"> act as an authoritative glossary for the terms the organization uses to describe its domain, such as entity types, activity categories, and classification tiers. Each is maintained as a single source of truth so the agent and the organization use language consistently.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Routing indexes<\/b><span style=\"font-weight: 400;\"> map input characteristics to the relevant positions and procedures, determining which files apply without relying on embedding similarity alone. This makes retrieval deterministic and auditable.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Gateway files<\/b><span style=\"font-weight: 400;\"> define threshold tests the agent must pass before entering an analytical domain, preventing it from applying specialized knowledge where it does not belong.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Every file declares its dependencies (<\/span><span style=\"font-weight: 400;\">depends_on<\/span><span style=\"font-weight: 400;\">) and consumers <\/span><span style=\"font-weight: 400;\">(<\/span><span style=\"font-weight: 400;\">referenced_by<\/span><span style=\"font-weight: 400;\">) in YAML frontmatter, forming a bidirectional dependency graph. When one file changes, you can trace exactly what else might be affected, which matters when the self-improvement loop proposes automated edits.<\/span><\/p>\n<figure id=\"attachment_24426\" aria-describedby=\"caption-attachment-24426\" style=\"width: 1999px\" class=\"wp-caption alignnone\"><img fetchpriority=\"high\" decoding=\"async\" class=\"size-full wp-image-24426\" src=\"https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png\" alt=\"\" width=\"1999\" height=\"1173\" srcset=\"https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png 1999w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png?resize=916,538 916w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png?resize=768,451 768w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png?resize=1024,601 1024w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png?resize=1536,901 1536w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png?resize=96,56 96w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain-Expert-AI-knowledge-system.png?resize=192,113 192w\" sizes=\"(max-width: 992px) 100vw, 62vw\"\/><figcaption id=\"caption-attachment-24426\" class=\"wp-caption-text\">An illustration of the knowledge system: files are organized as a navigable filesystem (left), and each file\u2019s YAML frontmatter (right) declares when it applies (the triggering scenarios) plus its dependencies and consumers, forming a bidirectional dependency graph the agent can traverse and maintain easily.<\/figcaption><\/figure>\n<h2><span style=\"font-weight: 400;\">Organizing Knowledge by Density and Usage Frequency<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">A key architectural decision is how to partition knowledge between the curated wiki and supplementary retrieval (RAG). We split on <\/span><b>information density and expected usage frequency<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><b>High-density, frequently referenced sources<\/b><span style=\"font-weight: 400;\"> go into the wiki<\/span><b>:<\/b><span style=\"font-weight: 400;\"> Distilled files capturing how the organization reasons, such as positions, decision frameworks, boundary examples, and strategic interpretations. The agent consults these on nearly every turn. Because they encode the organization\u2019s evolving thinking, they need to stay current, and the wiki structure makes them easy to update, version, and validate.<\/span><\/p>\n<p><b>Sparse, situationally relevant sources<\/b><span style=\"font-weight: 400;\"> are served through semantic or lexical search (RAG): documents that matter deeply when they apply but are not needed in detail on most runs, such as detailed reference material, individual product specifications, historical decision records, and niche external knowledge. Loading all of them into the wiki would bloat the system and dilute attention.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The result is that the agent\u2019s core reasoning is always grounded in the most refined, current organizational knowledge, while it can still reach for supporting evidence when a scenario demands it. The combination produces an organizational second brain that encodes how the organization interprets and applies information, rather than only where to find it.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Expert Reasoning via Composable Recipes<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Knowledge alone is not enough. Domain experts do not simply recall facts, they follow structured methodologies: a financial analyst works through a valuation model step by step, a security engineer follows a threat modeling procedure. The challenge is capturing those methodologies in a form an LLM can execute reliably.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We solve this with <\/span><b>composable procedures we call recipes<\/b><span style=\"font-weight: 400;\">. Where knowledge files are declarative, recipes are imperative. Each one prescribes a multi-step analytical workflow, specifying what to examine first, which knowledge to load at each step, what decision procedures to follow, and what constitutes a complete analysis.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The critical design choice is <\/span><b>separating what the agent knows from how it reasons<\/b><span style=\"font-weight: 400;\">. Recipes reference knowledge files but contain no domain facts; knowledge files state positions but prescribe no procedures. This means:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Adding an organizational position means adding a knowledge file and updating a routing index. No recipe changes.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Fixing a flaw in the agent\u2019s methodology means editing a recipe. No knowledge files change.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Failures attribute cleanly to one layer. Was the knowledge wrong, or the procedure?<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Recipes compose into pipelines, much like a head chef\u2019s master recipe for a dinner service delegates to sub-recipes for each component (the sauce, the protein, the garnish) without containing those details itself. Our top-level routing recipe examines the input and selects which downstream recipes to invoke, each handling one analytical phase.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This is also what enables <\/span><b>progressive disclosure<\/b><span style=\"font-weight: 400;\">. Rather than front-loading a monolithic instruction set covering every possible scenario, each recipe step carries only the instructions and knowledge relevant to that phase. Early versions used a single flat instruction file and loaded all sources via semantic search, pulling a large volume of mixed-relevance files into the context window on every run. After restructuring into recipe-driven stages, each query touches only a small, targeted subset, cutting tokens consumed per turn by around 80%. Context windows are finite and attention degrades with volume, so delivering the right instructions at the right time directly improves reasoning quality.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Keeping Humans in Control<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Human experts stay in control of this system throughout. The agent accelerates and structures their work; it does not replace their judgment or their authority over the outcome.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We enforce this through two mechanisms:\u00a0<\/span><\/p>\n<p><b>Checkpoints<\/b><span style=\"font-weight: 400;\"> are defined points in the analysis where the agent surfaces its intermediate reasoning for expert review before proceeding, and the expert can confirm, correct, or redirect.\u00a0<\/span><\/p>\n<p><b>Escalations<\/b><span style=\"font-weight: 400;\"> trigger when the agent hits genuine ambiguity, whether from underspecified inputs or evidence that supports more than one defensible reading. Rather than forcing a resolution, it hands the question to the expert, whose choice determines the path the analysis takes.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Checkpoints and escalations serve three purposes simultaneously:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Quality and direction control:<\/b><span style=\"font-weight: 400;\"> Experts catch errors before they compound downstream, and keep the analysis on the path they consider most relevant.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Training signal:<\/b><span style=\"font-weight: 400;\"> Every correction and every escalation becomes input for the self-improvement loop.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Trust calibration:<\/b><span style=\"font-weight: 400;\"> Experts build confidence incrementally by observing the agent\u2019s reasoning rather than just its final output, and by seeing it flag uncertainty instead of masking it.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">The more consequential the decision, the more this matters, which is why we recommend keeping a human in the loop by default across domains like compliance, financial risk assessment, security review, and engineering safety.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">The Self-Improvement Flywheel<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">We consider the self-improvement flywheel to be the most distinctive part of this system. While the structured knowledge system and composable recipes produce a system that is legible by humans and agents, testable, and modular, the number of interdependent files make manual maintenance impossible to scale.<\/span> <span style=\"font-weight: 400;\">When domain experts provide feedback to an agent that feedback has to be translated into precise file edits. This process can take weeks because it requires understanding the full dependency graph, verifying nothing else breaks, and validating the fix actually works.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">There has been a large body of work \u2013 from RAG memory systems to model-weight knowledge editing \u2013 devoted to addressing how agents store, retrieve, and update knowledge. But much less attention has gone toward<\/span> <span style=\"font-weight: 400;\">keeping a <\/span><b>document-based<\/b><span style=\"font-weight: 400;\"> institutional knowledge base correct as it grows and as expert positions evolve. Agents that auto-draft their own fixes are increasingly common; but we haven\u2019t seen this level of validation rigor applied to a structured knowledge base without model retraining.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We treat that maintenance as a <\/span><b>compilation problem<\/b><span style=\"font-weight: 400;\"> and automate it. Every expert correction moves through four phases:<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Diagnose expert feedback into actionable issues with their root cause.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Compile issues into minimal verified edits.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Validate that fixes work without regressions.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Have domain experts review them.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">Once the loop completes, the regression test suite is enriched with the issue that was just fixed so that future updates preserve this behavior.<\/span><\/p>\n<figure id=\"attachment_24423\" aria-describedby=\"caption-attachment-24423\" style=\"width: 1376px\" class=\"wp-caption alignnone\"><img loading=\"lazy\" loading=\"lazy\" decoding=\"async\" class=\"wp-image-24423 size-full\" src=\"https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png\" alt=\"\" width=\"1376\" height=\"768\" srcset=\"https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png 1376w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png?resize=916,511 916w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png?resize=768,429 768w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png?resize=1024,572 1024w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png?resize=96,54 96w, https:\/\/engineering.fb.com\/wp-content\/uploads\/2026\/08\/Domain_Expert_AI_Graph1-FINAL-e1787758306337.png?resize=192,107 192w\" sizes=\"auto, (max-width: 992px) 100vw, 62vw\"\/><figcaption id=\"caption-attachment-24423\" class=\"wp-caption-text\">The self-improvement loop. Expert corrections are diagnosed to their root cause, compiled into minimal verified edits, and evaluated against replay and regression tests before they are reviewed and landed. Each fix is then folded back into the regression suite, so the gain is permanent.<\/figcaption><\/figure>\n<h3><span style=\"font-weight: 400;\">Diagnosis: Attributing Each Correction to a Root Cause<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">Raw expert feedback comes from conversation traces where domain SMEs interacted with the agent and provided corrections. The diagnosis phase extracts structured signals from these conversations.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Our first approach classified feedback by conversational form. If the expert provided information, it must be a knowledge gap; if they redirected the agent, it must be a procedure problem. This heuristic failed because conversational form is a poor proxy for root cause. An expert correcting a conclusion might be exposing a knowledge gap, a recipe flaw, or a genuine ambiguity.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The working approach separates extraction from classification. First, extract every substantive signal from the expert alongside the agent\u2019s full knowledge manifest (every file loaded, when, and how used). Second, read the actual knowledge files and apply a single attribution test: Could the agent have reached the correct conclusion from its source materials?<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">If the materials contained the right answer but the agent still erred: recipe problem.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">If the materials did not contain the right answer: knowledge gap.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">If experts themselves disagree on the right answer: ambiguity, flagged for human discussion.<\/span><\/li>\n<\/ul>\n<h3><span style=\"font-weight: 400;\">Compilation: Surgical Multi-Agent Edits<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">The compiler translates each diagnosed issue into minimal file edits. Sub-agents analyze impact in parallel, examining cross-references, conflicts with existing positions, token budget impact, test coverage, and duplication risk.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<p><span style=\"font-weight: 400;\">Two design choices make this trustworthy:<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<p><b>Independent adversarial review.<\/b><span style=\"font-weight: 400;\"> A separate agent, running in a fresh context with no knowledge of the improvement rationale, receives only the proposed diffs to the knowledge base. Its job is to find problems such as contradictions introduced, edge cases broken, or positions undermined. Because it shares no context with the proposing agents, it cannot inherit their blind spots.<\/span><\/p>\n<p><b>Deterministic structural validation.<\/b><span style=\"font-weight: 400;\"> A linter catches issues programmatically, dangling cross-references, file size budget violations, identifier collisions, and dependency cycles. This layer is not probabilistic. It passes or fails.<\/span><\/p>\n<h3><span style=\"font-weight: 400;\">Evaluation: Proving the Fix Works<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">Every proposed change goes through a two-stage validation:<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<p><b>Targeted replay<\/b><span style=\"font-weight: 400;\"> runs the agent on the original scenario that triggered the feedback. The agent does not know it is being tested. A separate judge evaluates the new output against the original expert feedback without knowing what was changed. This deliberately blind design prevents confirmation bias. If targeted replay fails, compilation is retried.<\/span><\/p>\n<p><b>Regression testing<\/b><span style=\"font-weight: 400;\"> runs multiple benchmarks for that domain, which are usually structured test suites of Q&amp;A pairs. For analytical domains whether there might be multiple correct answers, an independent LLM judge gives each test case a pass\/fail based on certain criteria. The agent is run in parallel, independent sessions against the benchmark questions, and regressions in performance are detected. If regression testing fails, compilation is retried with an updated prompt describing where the agent regressed, along with the original issue and attempted fix.<\/span><\/p>\n<h3><span style=\"font-weight: 400;\">Landing and Enrichment: Compounding Returns<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">The output of the pipeline is a pull request (diff) with a complete audit trail. A human expert reviews a proven fix rather than debugging a raw failure. Once approved and landed (the knowledge file or recipe is updated), the original failing scenario and its validated correct answer are automatically added to the regression test suite. This means every fix permanently raises the bar and future changes to the knowledge system must preserve the behavior that was just corrected.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Results<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">After three development sprints spanning six weeks, the system achieved:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Domain SMEs rated agent outputs <\/span><b>useful almost all the time<\/b><span style=\"font-weight: 400;\">, a significant improvement from early versions where outputs frequently required substantial rework.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Days to minutes<\/b><span style=\"font-weight: 400;\"> reduction in individual assessment time.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Automated self-improvement<\/b><span style=\"font-weight: 400;\"> producing validated knowledge edits at a rate that previously required full engineering sprints.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Zero regressions<\/b><span style=\"font-weight: 400;\"> across improvement cycles, with every fix automatically strengthening the regression suite.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Domain experts consistently reported the agent handles the <\/span><b>vast majority of the analytical work<\/b><span style=\"font-weight: 400;\">, allowing them to focus on the genuinely ambiguous cases that require human judgment.<\/span><\/li>\n<\/ul>\n<h2><span style=\"font-weight: 400;\">Applying This Architecture<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The specific domain we built this for required synthesizing dozens of sources (both internal positions and external material) into risk-weighted assessments. But the architecture is domain-independent. It applies wherever:<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Specialist knowledge lives as tribal knowledge in experts\u2019 heads.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Consistency across assessments matters.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The volume of work exceeds available expert capacity.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Off-the-shelf LLMs produce inadequate analysis.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Concrete domains where this pattern fits include regulatory compliance, protocol adherence, financial risk assessment, security review, engineering standards compliance, and procurement evaluation. The common thread is that organizations need AI systems with genuine institutional expertise, not just general knowledge.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The requirements for adopting this architecture are:<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A <\/span><b>structured knowledge system<\/b><span style=\"font-weight: 400;\"> with explicit file boundaries, cross-references, and a dependency graph (the organizational second brain for the domain).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A <\/span><b>procedural layer<\/b><span style=\"font-weight: 400;\"> that separates domain knowledge from analytical methodology (recipes).<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">An <\/span><b>automated evaluation suite<\/b><span style=\"font-weight: 400;\"> that grows with each improvement cycle.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Human-in-the-loop checkpoints<\/b><span style=\"font-weight: 400;\"> calibrated to the domain\u2019s risk tolerance.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">The deeper principle is straightforward: keep the complexity in text files that are readable by both humans and agents, rather than fine-tuned model weights. Every improvement is a text edit that a domain expert can review in 30 seconds. Every change is version-controlled, diffable, and reversible. The compilation pipeline is sophisticated, but its outputs are always transparent.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<p><span style=\"font-weight: 400;\">The goal is a system where expert effort compounds permanently. Every expert interaction makes the system better. Every correction persists as a verified improvement. The organization\u2019s collective knowledge stops being trapped in individuals and starts being available, consistently and at scale, to everyone who needs it.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">Acknowledgements<\/span><\/h2>\n<p><em><span style=\"font-weight: 400;\">The authors would like to express our gratitude to the contributions of the following people, who have played a crucial role in developing this system. In particular, we would like to extend special thanks to (in last name alphabetical order): <\/span><span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Cecilia Baek&quot;,&quot;per_e&quot;:&quot;cbaek@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Cecilia Baek,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Philipp Kaufold&quot;,&quot;per_e&quot;:&quot;philippkaufold@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Philipp Kaufold,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Cat Hughes&quot;,&quot;per_e&quot;:&quot;cathughes@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Cat Hughes,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Suzanne Leijten&quot;,&quot;per_e&quot;:&quot;suzanneleijten@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Suzanne Leijten,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Michael Marcusa&quot;,&quot;per_e&quot;:&quot;mgmarcusa@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Michael Marcusa,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Jordi Mola&quot;,&quot;per_e&quot;:&quot;jordim@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Jordi Mola,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Timothy Neo&quot;,&quot;per_e&quot;:&quot;neotimothy@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Timothy Neo,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Elliott Prentiss&quot;,&quot;per_e&quot;:&quot;eprentiss@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Elliott Prentiss,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Laia Reyes&quot;,&quot;per_e&quot;:&quot;laiareyes@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Laia Reyes,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;John Ross&quot;,&quot;per_e&quot;:&quot;johnrross@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">John Ross,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Julio Santil&quot;,&quot;per_e&quot;:&quot;jsantil@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Julio Santil,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Taylor Wilson Thomas&quot;,&quot;per_e&quot;:&quot;taylorthms@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Taylor Wilson Thomas,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Mansi Tripathi&quot;,&quot;per_e&quot;:&quot;mansitri@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Mansi Tripathi,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Nikhil Shanbhag&quot;,&quot;per_e&quot;:&quot;nshanbhag@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Nikhil Shanbhag,<\/span> <span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Madeleine Vos&quot;,&quot;per_e&quot;:&quot;madeleinevos@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Madeleine Vos,<\/span><span style=\"font-weight: 400;\"> and <\/span><span style=\"font-weight: 400;\" data-rich-links=\"{&quot;per_n&quot;:&quot;Jackie Zajac&quot;,&quot;per_e&quot;:&quot;jackiezajac@meta.com&quot;,&quot;type&quot;:&quot;person&quot;}\">Jackie Zajac.<\/span><\/em><\/p>\n<\/p><\/div>\n<p><script id=\"gdprconsent-js-after\">\nif (window.gdprSafeTrack) {\n  \t\t\t\t\t\twindow.gdprSafeTrack('https:\/\/www.googletagmanager.com\/gtag\/js?id=391063774');\n\t\t\t\t\t\twindow.gdprSafeTrack(function() {\n\t\t\t\t\t\t\tvar google_analytics_g4  = decodeURIComponent( '391063774' );\n\t\t\t\t\t\t\twindow.dataLayer = window.dataLayer || [];\n\t\t\t\t\t\t\tfunction gtag(){dataLayer.push(arguments);}\n\t\t\t\t\t\t\tgtag('js', new Date());\n\t\t\t\t\t\t\tgtag('config', google_analytics_g4);\n\t\t\t\t\t\t});\n\t\t\t\t\t}\nif (window.gdprSafeTrack) {\n\t\t\t\t\t\twindow.gdprSafeTrack(function() {\n\t\t\t\t\t\t\tvar google_tag_manager  = decodeURIComponent( 'GTM-KZRLGZ8' );\n\t\t\t\t\t\t\t(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':\n\t\t\t\t\t\t\tnew Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],\n\t\t\t\t\t\t\tj=d.createElement(s),dl=l!='dataLayer'?'&l=\"+l:\"';j.async=true;j.src=\"https:\/\/www.googletagmanager.com\/gtm.js?id=\"+i+dl;f.parentNode.insertBefore(j,f);\n\t\t\t\t\t\t\t})(window,document,'script','dataLayer',google_tag_manager);\n\t\t\t\t\t\t});\n\t\t\t\t\t}\nif (window.gdprSafeTrack) {\n\t\t\t\t\t\twindow.gdprSafeTrack(function() {\n\t\t\t\t\t\t\tvar facebook_pixel  = decodeURIComponent( '660742068067366' );\n\t\t\t\t\t\t\t!function(f,b,e,v,n,t,s)\n\t\t\t\t\t\t\t{if(f.fbq)return;n=f.fbq=function(){n.callMethod?\n\t\t\t\t\t\t\tn.callMethod.apply(n,arguments):n.queue.push(arguments)};\n\t\t\t\t\t\t\tif(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0';\n\t\t\t\t\t\t\tn.queue=[];t=b.createElement(e);t.async=!0;\n\t\t\t\t\t\t\tt.src=v;s=b.getElementsByTagName(e)[0];\n\t\t\t\t\t\t\ts.parentNode.insertBefore(t,s)}(window, document,'script',\n\t\t\t\t\t\t\t'https:\/\/connect.facebook.net\/en_US\/fbevents.js');\n\t\t\t\t\t\t\tfbq('init', facebook_pixel);\n\t\t\t\t\t\t\tfbq('track', 'PageView');\n\t\t\t\t\t\t});\n\t\t\t\t\t}\n\/\/# sourceURL=gdprconsent-js-after\n<\/script><br \/>\n<br \/><a href=\"https:\/\/engineering.fb.com\/2026\/09\/02\/ml-applications\/organizational-second-brain-ai-learns-from-experts\/?utm_source=tldrai\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We\u2019ve built an AI agent that acts as a secondary expert for a given domain, making deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon. This is not a typical domain-specific agent. Its novelty comes from integrating two layers:\u00a0 A structured, auditable knowledge architecture separates what [&hellip;]<\/p>\n","protected":false},"author":16,"featured_media":23705,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-23704","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23704","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=23704"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/23704\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/23705"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=23704"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=23704"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=23704"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}