How PIBot Works

What is PIBot?

PIBot is the Protocol Institute’s knowledge infrastructure — a retrieval-augmented generation (RAG) system that answers questions grounded in the PI corpus. It is not a fine-tuned model. The underlying language model is Claude Sonnet; what makes it PI-specific is the corpus material injected as retrieval context at query time and a system prompt encoding PI’s intellectual commitments, scope, and vocabulary.

PIBot is the Protocol Institute’s research assistant: it answers from the Institute’s own corpus and cites what it draws on.

Two voices, one corpus. PIBot serves two audiences with distinct response styles selected by a context field in the request body:

• Web / MCP (context absent): research-librarian voice — dense, source-specific, 3–5 paragraphs. For researchers and extended sessions.
• Discord (context: "discord"): office-manager voice — 2–3 sentences, one named resource, plain prose. A DISCORD VOICE OVERRIDE block appended to the base system prompt supersedes all length and format instructions.

The corpus

Total: ~26,300 vectors across 11 Pinecone namespaces. The corpus is live: Discord and SIG channels sync incrementally every 30 minutes via a local daemon; Substack syncs daily via GitHub Actions; the lexicon and PDFs are updated manually when the source set changes.

Ingest pipeline pattern

Every corpus source follows a three-layer pattern. Deviating requires explicit justification. This pattern is what makes retrieval work well — it is worth reproducing exactly when adding new sources.

Layer 1 — Haiku enrichment

One Claude Haiku call per document. Input: title + authors + type/tags + first ~1,500 chars of text. Output saved to sources/<type>/enriched_meta.json keyed by document ID:

{
  "summary":        "Two concrete sentences about the specific argument or contribution.",
  "categories":     ["protocol-theory", "governance"],
  "primary_author": "Author Name",
  "all_authors":    ["Author Name"]
}

Shared category vocabulary (used across all sources for cross-corpus query consistency): protocol-fiction · protocol-theory · protocol-watching · editorial · research-report · technology-ai · governance · announcement · interview · memory-archival · organizations. Idempotent; skips existing entries unless --force is passed.

Layer 2 — Body chunks

Text chunked at 512-token windows with 64-token overlap. The text sent to Voyage for embedding is prefixed with title and summary information — anchoring the chunk to its topic even when topic keywords don’t appear in the chunk itself. The stored display text is clean (no prefix). Vector IDs are SHA-256 hashes of raw chunk text, so duplicate passages across documents are automatically deduplicated.

# Embedding input (not stored):
"Title: {title}\nAuthors: {authors}\nType: {type}\nSummary: {summary}\n\n{chunk_text}"

# Stored in Pinecone metadata:
{ text: "{chunk_text}", title, authors, date, url, chunk_type, source, ... }

Layer 3 — Document summary vector

One additional vector per document for “what is this document about” queries. Vector ID: {id}__doc_summary (or {slug}__post_summary for Substack). When a summary vector ranks in the top results, the worker fires a follow-up filtered query to fetch real body chunks from the same document — giving Claude actual prose rather than an abstract. Summary hits are replaced in the result set by their body-chunk siblings before the LLM sees them.

Checklist for adding a new source

To reproduce this pattern for a new corpus source: (1) write an enrichment script that calls Claude Haiku and saves to sources/<type>/enriched_meta.json; (2) write an ingest script that chunks, prefixes, embeds via Voyage, and upserts to a named Pinecone namespace; (3) create an incremental state file keyed by document ID or timestamp watermark; (4) add a normalize<Type>() function in api/worker.js that maps raw Pinecone metadata to the standard source object shape; (5) add the namespace to mergeResults() with an appropriate tier weight; (6) register the source in config/source_registry.json and config/corpus_map.json; (7) add a daemon step in bin/daemon.py.

The Pinecone index

Single index c3po — 1,024 dimensions, cosine metric, serverless (AWS us-east-1), Protocol Institute org account. All corpus namespaces queried in parallel on every request; results merged and tier-weighted before passing to Claude.

Query pipeline (step by step)

Every POST /query, ask_pibot MCP call, and Discord @mention follows the same pipeline:

1. Embed the query — Voyage AI voyage-3 encodes the question into a 1,024-dim vector (input_type: "query").
2. Parallel namespace query — all 9 corpus namespaces queried simultaneously at top_k = TOP_K_EACH (typically 5); meta queried at top-3; transcripts queried at top-3; sig_meeting_page sub-query fired in parallel for the sig namespace.
3. Normalize — each namespace has a normalize*() function mapping raw Pinecone metadata to a standard shape: { source, type, label, title, authors, date, url, summary, excerpt, score, ... }. Namespace-specific fields (e.g. sig_display, channel_name, domain) are preserved alongside.
4. Merge and tier-weight — mergeResults() applies the tier weights above, deduplicates by URL, extracts transcript cache hits (score ≥ 0.52), and returns the top MAX_SOURCES items.
5. Secondary retrieval — summary-type vectors in the top results trigger follow-up body-chunk queries for their documents. Results merged back in.
6. Build context block — buildContextBlock() formats each source as a labeled excerpt block for the Claude prompt: [PDF — "Title" — Author(s) — YYYY]\n{excerpt}, with namespace-appropriate prefixes.
7. Claude Sonnet — system prompt cached with cache_control: ephemeral; user message is "Question: {q}\n\nRelevant archive excerpts:\n\n{context}". Discord requests use the DISCORD_SYSTEM_PROMPT variant. History array passed for multi-turn sessions.
8. Return — response includes answer, sources (with metadata for badge/link rendering), and cache_hits (high-scoring transcript matches surfaced separately).

The language model and system prompt

Claude Sonnet is used throughout — query answering, document enrichment, meeting summarization, link relevance scoring (Haiku for the latter two). The research material is dense and cross-disciplinary; strong synthesis matters more than fast extraction.

System prompt structure. The system prompt is an inline document (~1,100 tokens) in api/worker.js, maintained alongside the retrieval code. It is not loaded from a file at runtime. It was originally inspired by SOUL.md (the conceptual identity document in the repo root), but has since evolved independently and is now substantially more detailed. An agent reproducing this pattern should treat the inline prompt as the source of truth. It contains seven sections:

1. Role and mission — who PIBot is; corpus-grounded research assistant, not a general chatbot.
2. About the Protocol Institute — SoP provenance, current programs, leadership, independence. Prevents hallucinated org facts.
3. Scope declaration — 11 topic areas explicitly in scope; 4 categories explicitly out of scope. Prevents false denials (“I don’t know about that”) for topics that are in the corpus.
4. Intellectual commitments — 9 substantive PI analytical stances (e.g. “hardness is a design variable”, “context tank not think tank”). Shapes how Claude frames answers.
5. Voice, analytical moves, format — scholarly but accessible; specific about sources; 5 characteristic analytical moves (cross-domain comparison, hardness analysis, historical situating, formalization ladder, stakeholder analysis).
6. Indexed corpus map — lists the key named papers, magazine, YouTube series, and SIG groups by name so Claude never falsely denies having them.
7. Protocol lexicon — 40+ PI-coined or PI-specific terms with compact definitions injected inline. Prevents chunk-boundary definition splits and ensures PI vocabulary is used correctly even when a term doesn’t appear verbatim in retrieved chunks.

The Discord variant appends a DISCORD VOICE OVERRIDE block that supersedes the voice and length instructions. Both variants exceed the 1,024-token Anthropic prompt-cache threshold and cache independently (cache_control: ephemeral).

Rate limits: 20 queries per IP per hour via the web UI. After 8 turns, the conversation can be downloaded as Markdown and continued in Claude, or accessed without a turn limit via MCP.

Delivery interfaces

Three interfaces share the same corpus, query engine, and Cloudflare Worker. They differ in voice, turn model, and how they reach users.

Web UI — pibot.protocolized.io

Browser-based chat at the root URL. Full research-librarian voice. 8-turn session limit (download as Markdown to continue). Submitted conversations can be shared publicly or kept private; public submissions are indexed into the transcripts namespace for self-memory. The /chats route is a public transcript browser.

Discord bot — PIBot

Gateway bot (discord.py, WebSocket) running on the host machine under launchd (org.protocol-institute.c3po-bot). All bot requests pass context: "discord" to the Worker, selecting the 2–3 sentence office-manager response style.

• @mention in any channel — opens a thread, responds with answer + sources. Thread replies continue for up to 5 turns without re-mention; full history passed to the Worker. Side-conversation filtering: replies to another human (not the bot) are silently skipped unless the bot is @mentioned. The 5-turn cap notice is sent exactly once; subsequent messages are silently ignored.
• Navigation queries — “where should I post about X?” intent is detected by regex and routed to a discord_guide query instead of corpus RAG, returning the top 3 relevant channels with blurbs and meeting schedules.
• #introductions monitoring — new members (joined ≤60 days ago) receive a corpus resource recommendation + channel suggestion. Members who joined >60 days ago and post ≥80 characters receive a “welcome back” variant. Both paths use the same underlying corpus query; VGR-authored sources are filtered from intro recs to ensure diversity. New-member welcomes are queued with up to 3 retry attempts so a bot restart can recover missed welcomes.
• /ask, /search, /help slash commands — Discord Interactions webhook received at POST /interactions (Ed25519 verified); command enqueued to Cloudflare Queue; Worker queue consumer runs the RAG pipeline and posts back via Discord followup webhook.

Completed bot conversations are spooled to data/spool/bot_conversations/; the listener daemon picks them up each cycle and ingests them into the transcripts namespace.

MCP server — /mcp

PIBot is available as a Model Context Protocol server (JSON-RPC 2.0 + Streamable HTTP) at https://pibot.protocolized.io/mcp. Connect it to Claude Code or Claude Desktop to query the corpus directly inside your AI client — no turn limit, no browser required.

search_corpus is open — no key required. Good for agentic workflows that need raw retrieval without LLM cost or turn limits.

ask_pibot requires a Bearer token (each call invokes Claude Sonnet and Voyage AI at real cost). To request access email team@protocol-institute.org.

Claude Code

Search only (no key needed) — run once in your terminal:

claude mcp add pibot --transport http https://pibot.protocolized.io/mcp

Full access with Bearer token:

claude mcp add pibot --transport http https://pibot.protocolized.io/mcp \
  --header "Authorization: Bearer <your-key>"

Claude Desktop

Add to claude_desktop_config.json (on Mac: ~/Library/Application Support/Claude/):

{"mcpServers": {"pibot": {
  "type": "http",
  "url": "https://pibot.protocolized.io/mcp",
  "headers": {"Authorization": "Bearer <your-key>"}
}}}

For search-only without auth, omit the headers key.

Infrastructure