# Cogitator > Self-hosted, production-grade AI agent runtime for TypeScript: multi-provider LLMs, tools with approvals, memory and RAG, DAG workflows, multi-agent swarms, sandboxed execution, MCP and A2A. Cogitator is a set of npm packages under the `@cogitator-ai/*` scope. Start with `@cogitator-ai/core` (agents, tools, LLM backends) and add packages for memory, workflows, swarms, RAG, evals, voice, browser automation, messaging channels and server adapters as you need them. Install: ```bash npm install @cogitator-ai/core zod npx create-cogitator-app my-agents ``` Every link below points to the Markdown version of a docs page; drop the `.mdx` suffix for the HTML page. ## Overview - [Welcome to Cogitator](https://cogitator.app/docs.mdx): The Sovereign AI Agent Runtime — self-hosted, production-grade orchestration for LLM swarms and autonomous agents. ## Getting Started - [Installation](https://cogitator.app/docs/getting-started.mdx): Install Cogitator and set up your development environment. - [Quick Start](https://cogitator.app/docs/getting-started/quick-start.mdx): Build your first AI agent in 5 minutes. - [Project Scaffolding](https://cogitator.app/docs/getting-started/scaffolding.mdx): Create new Cogitator projects in seconds with create-cogitator-app. - [Project Structure](https://cogitator.app/docs/getting-started/project-structure.mdx): How Cogitator projects and the monorepo are organized. - [Configuration](https://cogitator.app/docs/getting-started/configuration.mdx): Configure Cogitator with YAML files, environment variables, and provider settings. - [CLI](https://cogitator.app/docs/cli.mdx): Command-line interface for Cogitator — scaffold assistants, run them as services, chat with agents, manage Ollama models and deploy. ## Core - [Cogitator Runtime](https://cogitator.app/docs/core/cogitator.mdx): The main runtime that orchestrates agent execution, LLM routing, memory, and observability. - [Agents](https://cogitator.app/docs/core/agents.mdx): Configure and run AI agents with tools, instructions, and model parameters. - [Tools](https://cogitator.app/docs/core/tools.mdx): Create type-safe tools that give agents the ability to interact with external systems. - [LLM Backends](https://cogitator.app/docs/core/llm-backends.mdx): Supported LLM providers, configuration, and provider-prefix routing. - [Model Registry](https://cogitator.app/docs/core/model-registry.mdx): Dynamic model registry with pricing, capabilities, and multi-provider support. - [Structured Outputs](https://cogitator.app/docs/core/structured-outputs.mdx): Get typed JSON responses from LLMs using Zod schemas and response format modes. - [Vision & Audio](https://cogitator.app/docs/core/vision-and-audio.mdx): Multi-modal capabilities including image analysis, image generation, audio transcription, and text-to-speech. ## Tools - [Built-in Tools](https://cogitator.app/docs/tools/built-in.mdx): Reference for the 26 built-in tools that ship with @cogitator-ai/core, the environment variables they read, and the tool factories next to them. - [WASM Tools](https://cogitator.app/docs/tools/wasm-tools.mdx): Run tools in isolated WebAssembly sandboxes with Extism for security, speed, and portability. - [Creating Custom Tools](https://cogitator.app/docs/tools/custom-tools.mdx): Build type-safe tools for your agents using the tool() factory, Zod schemas, and structured error handling. - [Approvals](https://cogitator.app/docs/tools/approvals.mdx): Pause an agent before it runs a sensitive tool, let a person decide — now or hours later — and continue the run where it stopped. - [Agent as Tool](https://cogitator.app/docs/tools/agent-as-tool.mdx): Wrap agents as tools with agentAsTool() to enable hierarchical delegation between agents. - [Tool Caching](https://cogitator.app/docs/tools/tool-caching.mdx): Cache tool results with TTL, LRU eviction, and semantic similarity matching using in-memory or Redis storage. ## Browser - [Browser](https://cogitator.app/docs/browser.mdx): Playwright-based browser automation for AI agents with 33 tools across navigation, interaction, extraction, vision, and network modules. - [BrowserSession](https://cogitator.app/docs/browser/session.mdx): Manage browser lifecycle, tabs, cookies, proxy, and stealth configuration with BrowserSession. - [Tools Reference](https://cogitator.app/docs/browser/tools.mdx): Complete reference for all 33 browser tools across navigation, interaction, extraction, vision, and network modules. - [Stealth](https://cogitator.app/docs/browser/stealth.mdx): Anti-detection evasions, fingerprint randomization, and human-like behavior for browser automation. - [Vision Tools](https://cogitator.app/docs/browser/vision.mdx): Screenshot capture, accessibility tree traversal, and natural language element interaction. - [Network Tools](https://cogitator.app/docs/browser/network.mdx): Intercept HTTP requests, block resources, capture HAR logs, and monitor API calls. ## Memory & RAG - [Memory Overview](https://cogitator.app/docs/memory.mdx): Understand how Cogitator agents persist context across conversations using pluggable memory adapters, embeddings, and knowledge graphs. - [Memory Adapters](https://cogitator.app/docs/memory/adapters.mdx): Choose and configure memory adapters for agent persistence — InMemory, Redis, Postgres, SQLite, MongoDB, and Qdrant. - [Embedding Providers](https://cogitator.app/docs/memory/embeddings.mdx): Configure embedding providers for semantic search — OpenAI, Ollama, and Google — and understand how embeddings power agent memory. - [Hybrid Search](https://cogitator.app/docs/memory/hybrid-search.mdx): Combine BM25 keyword search with vector similarity using Reciprocal Rank Fusion for accurate, recall-maximizing retrieval. - [Knowledge Graphs](https://cogitator.app/docs/memory/knowledge-graphs.mdx): Extract entities and relationships from conversations, store them in a graph, run inference rules, and enrich agent context with structured knowledge. - [RAG Pipeline](https://cogitator.app/docs/rag.mdx): Build retrieval-augmented generation pipelines with @cogitator-ai/rag — document loading, chunking, embedding, retrieval, and reranking in a single composable pipeline. - [Document Loaders](https://cogitator.app/docs/rag/loaders.mdx): Load documents from text files, markdown, JSON, CSV, HTML, PDFs, and web URLs into the RAG pipeline. - [Chunking Strategies](https://cogitator.app/docs/rag/chunking.mdx): Split documents into embeddable chunks using fixed-size, recursive, or semantic chunking strategies. - [Retrieval Strategies](https://cogitator.app/docs/rag/retrieval.mdx): Retrieve relevant document chunks using similarity search, MMR diversity, hybrid vector+keyword search, or multi-query expansion. - [Reranking](https://cogitator.app/docs/rag/reranking.mdx): Improve retrieval precision by re-scoring results with an LLM or Cohere's reranking API. ## Voice - [Voice](https://cogitator.app/docs/voice.mdx): Add voice and realtime audio capabilities to Cogitator agents with @cogitator-ai/voice — speech-to-text, text-to-speech, voice activity detection, and native speech-to-speech via OpenAI Realtime and Gemini Live APIs. - [STT Providers](https://cogitator.app/docs/voice/stt-providers.mdx): Speech-to-text providers for @cogitator-ai/voice — OpenAI (gpt-transcribe) for batch transcription and Deepgram (nova-3) for real-time streaming with interim results. - [TTS Providers](https://cogitator.app/docs/voice/tts-providers.mdx): Text-to-speech providers for @cogitator-ai/voice — OpenAI (gpt-4o-mini-tts with 13 voices and instructions) and ElevenLabs (eleven_flash_v2_5 with low-latency streaming). - [Voice Activity Detection](https://cogitator.app/docs/voice/vad.mdx): Detect when users start and stop speaking with @cogitator-ai/voice VAD providers — zero-dependency EnergyVAD for simple environments and neural Silero VAD for noisy conditions. - [Pipeline Mode](https://cogitator.app/docs/voice/pipeline.mdx): Process voice through STT -> Agent -> TTS with VoicePipeline for batch processing and PipelineSession for continuous streaming conversations with VAD-driven turn detection. - [Realtime Mode](https://cogitator.app/docs/voice/realtime.mdx): Native speech-to-speech with RealtimeSession — connect to OpenAI Realtime API or Gemini Live API for low-latency voice conversations with tool calling support. ## Evaluation - [Evaluation Framework](https://cogitator.app/docs/evals.mdx): Systematically evaluate LLM agents with @cogitator-ai/evals — dataset-driven testing, deterministic and LLM-as-judge metrics, assertions, A/B comparison, and CI-ready reporters. - [Datasets](https://cogitator.app/docs/evals/datasets.mdx): Load evaluation test cases from JSONL files, CSV files, or build them programmatically with filtering, sampling, and shuffling. - [Metrics](https://cogitator.app/docs/evals/metrics.mdx): Score agent responses with deterministic checks, LLM-as-judge evaluation, statistical aggregation, and custom metric functions. - [Assertions](https://cogitator.app/docs/evals/assertions.mdx): Set pass/fail criteria with threshold checks, baseline regression detection, and custom assertion logic. - [A/B Comparison](https://cogitator.app/docs/evals/comparison.mdx): Compare two agent configurations with statistical significance testing using paired t-tests and McNemar's test. - [Reporters](https://cogitator.app/docs/evals/reporters.mdx): Output evaluation results to the console, JSON files, CSV files, or CI-friendly format with automatic failure exit codes. ## Workflows - [Workflows](https://cogitator.app/docs/workflows.mdx): DAG-based orchestration for multi-step agent tasks with parallel execution, conditional branching, and error handling. - [WorkflowBuilder](https://cogitator.app/docs/workflows/builder.mdx): Fluent API for constructing DAG-based workflows with nodes, edges, conditionals, loops, and parallel branches. - [Node Types](https://cogitator.app/docs/workflows/nodes.mdx): Built-in workflow node factories for agents, tools, functions, custom logic, and human-in-the-loop interactions. - [Workflow Execution](https://cogitator.app/docs/workflows/execution.mdx): Running workflows with the WorkflowExecutor, streaming events, checkpointing, and resuming long-running workflows. - [Sagas & Compensation](https://cogitator.app/docs/workflows/sagas.mdx): Retry policies, circuit breakers, compensation rollbacks, dead letter queues, and idempotency for resilient workflows. - [Scheduling & Triggers](https://cogitator.app/docs/workflows/scheduling.mdx): Scheduled runs with WorkflowManager, cron, webhook and event triggers, and durable timers. - [Workflow Patterns](https://cogitator.app/docs/workflows/patterns.mdx): MapReduce, sub-workflows, parallel execution, conditional branching, loops, fan-out/fan-in, scatter-gather, race, and fallback patterns. ## Swarms - [Swarms](https://cogitator.app/docs/swarms.mdx): Multi-agent coordination for complex tasks that require collaboration between specialized agents. - [Coordination Strategies](https://cogitator.app/docs/swarms/strategies.mdx): All 7 built-in strategies for multi-agent coordination — hierarchical, round-robin, consensus, auction, pipeline, debate and negotiation. - [Swarm Builder](https://cogitator.app/docs/swarms/builder.mdx): Build swarms with the fluent SwarmBuilder API or the Swarm class directly. - [Agent Communication](https://cogitator.app/docs/swarms/communication.mdx): Blackboard shared state, MessageBus direct messaging, and EventEmitter coordination for inter-agent communication. - [Swarm Assessment](https://cogitator.app/docs/swarms/assessment.mdx): Automatically analyze tasks and assign optimal models to agents with SwarmAssessor, TaskAnalyzer, RoleMatcher, and ModelDiscovery. - [Distributed Swarms](https://cogitator.app/docs/swarms/distributed.mdx): Scale swarms across multiple machines with Redis-backed coordination, distributed worker nodes, and shared state. ## Channels - [Gateway](https://cogitator.app/docs/channels/gateway.mdx): Route messages from messaging platforms to AI agents with sessions, middleware, and streaming. - [Wizard Setup](https://cogitator.app/docs/channels/wizard.mdx): Interactive assistant setup with cogitator wizard. - [Smart Memory](https://cogitator.app/docs/channels/smart-memory.mdx): Three-level memory system with knowledge graph, core facts, and compaction. - [Scheduler](https://cogitator.app/docs/channels/scheduler.mdx): Schedule tasks, reminders, and recurring jobs with HeartbeatScheduler. - [Telegram](https://cogitator.app/docs/channels/telegram.mdx): Connect your AI agent to Telegram on the current Bot API, with rich Markdown, streamed drafts with a stop button, buttons, topics, albums and command menus. - [Discord](https://cogitator.app/docs/channels/discord.mdx): Connect your AI agent to Discord servers and DMs. - [Slack](https://cogitator.app/docs/channels/slack.mdx): Connect your AI agent to Slack workspaces. - [WhatsApp](https://cogitator.app/docs/channels/whatsapp.mdx): Connect your AI agent to WhatsApp via Baileys. - [WebChat](https://cogitator.app/docs/channels/webchat.mdx): Built-in WebSocket channel for custom web interfaces and CLI tools. - [Terminal](https://cogitator.app/docs/channels/terminal.mdx): Chat with your assistant from the local terminal. - [Middleware](https://cogitator.app/docs/channels/middleware.mdx): Intercept and filter messages with pairing, rate limiting, and custom middleware. - [Streaming](https://cogitator.app/docs/channels/streaming.mdx): Progressive message delivery with StreamBuffer and message editing. - [Owner Commands](https://cogitator.app/docs/channels/owner-commands.mdx): Manage your bot at runtime through slash commands in chat. ## Server Adapters - [Server Adapters](https://cogitator.app/docs/server-adapters.mdx): Expose your Cogitator agents, workflows, and swarms as REST APIs with built-in streaming, WebSocket, and OpenAPI support. - [Express Adapter](https://cogitator.app/docs/server-adapters/express.mdx): Serve your Cogitator agents over HTTP with Express.js — class-based setup with middleware, SSE streaming, and WebSocket support. - [Fastify Adapter](https://cogitator.app/docs/server-adapters/fastify.mdx): Register Cogitator as a native Fastify plugin with automatic schema validation, Swagger UI, rate limiting, and WebSocket support. - [Hono Adapter](https://cogitator.app/docs/server-adapters/hono.mdx): Deploy Cogitator agents to any JavaScript runtime — Node.js, Bun, Deno, and Cloudflare Workers — using the lightweight Hono adapter. - [Koa Adapter](https://cogitator.app/docs/server-adapters/koa.mdx): Integrate Cogitator into your Koa application using the middleware-chain pattern with @koa/router. - [Tetsu Adapter](https://cogitator.app/docs/server-adapters/tetsu.mdx): Serve Cogitator agents, threads, workflows and swarms from a Tetsu application on Bun — one controller with validated requests, SSE, WebSocket and an OpenAPI document. - [OpenAPI Generation](https://cogitator.app/docs/server-adapters/openapi.mdx): Auto-generate OpenAPI 3.0 specs and serve Swagger UI from any Cogitator server adapter. ## Integrations - [Next.js](https://cogitator.app/docs/integrations/nextjs.mdx): First-class Next.js integration with streaming API route handlers and React hooks for building chat interfaces. - [Vercel AI SDK](https://cogitator.app/docs/integrations/ai-sdk.mdx): Bidirectional bridge between Cogitator and the Vercel AI SDK 4, 5, 6 and 7 -- use Cogitator agents as AI SDK models or use AI SDK models inside Cogitator. - [OpenAI Compatibility](https://cogitator.app/docs/integrations/openai-compat.mdx): Drop-in OpenAI Assistants API compatibility layer that lets existing OpenAI SDK clients interact with Cogitator agents. - [Model Context Protocol (MCP)](https://cogitator.app/docs/integrations/mcp.mdx): Full MCP support -- connect to external MCP servers to use their tools, or expose Cogitator tools as an MCP server. - [A2A Protocol (Agent-to-Agent)](https://cogitator.app/docs/integrations/a2a.mdx): Native implementation of Google's A2A Protocol v0.3 for cross-framework agent interoperability. ## Advanced - [Reflection Engine](https://cogitator.app/docs/advanced/reflection.mdx): Enable agents to reflect on their own outputs, extract insights from past executions, and continuously improve through self-evaluation loops. - [Tree-of-Thought Reasoning](https://cogitator.app/docs/advanced/reasoning.mdx): Explore several approaches to a problem with the ThoughtTreeExecutor, score branches, prune weak ones and return the best reasoning path. - [Agent Learning & Optimization](https://cogitator.app/docs/advanced/learning.mdx): Improve agents over time with execution traces, DSPy-style compilation of instructions and demos, A/B tests, degradation monitoring and automated optimization. - [Prompt Versions](https://cogitator.app/docs/advanced/prompt-versions.mdx): Deploy new instructions without redeploying code, roll back in one call, and A/B test instructions on live traffic with the results recorded per version. - [Constitutional AI](https://cogitator.app/docs/advanced/constitutional-ai.mdx): Define safety principles, filter harmful inputs and outputs, guard tool calls, and apply critique-revise loops to ensure agents behave within defined boundaries. - [Cost-Aware Routing](https://cogitator.app/docs/advanced/cost-routing.mdx): Route tasks to the cheapest capable model, enforce spending budgets, estimate token usage before calling, and track costs across agents and runs. - [Time-Travel Debugging](https://cogitator.app/docs/advanced/time-travel.mdx): Capture execution snapshots at any step, replay from checkpoints, fork alternate timelines, and compare runs side-by-side to debug agent behavior. - [Causal Reasoning](https://cogitator.app/docs/advanced/causal-reasoning.mdx): Build causal graphs, run d-separation analysis, perform counterfactual reasoning, and predict intervention effects. - [Context Management](https://cogitator.app/docs/advanced/context-management.mdx): Automatically manage conversation length with truncation, summarization, sliding window, and smart hybrid compression strategies. - [Neuro-Symbolic AI](https://cogitator.app/docs/advanced/neuro-symbolic.mdx): Combine neural LLM reasoning with Prolog-style logic, SAT/SMT constraint solving, knowledge graph queries and verified planning. - [Self-Modifying Agents](https://cogitator.app/docs/advanced/self-modifying.mdx): Build agents that generate new tools at runtime, adapt reasoning strategies, and evolve their architecture through multi-armed bandits. - [Security](https://cogitator.app/docs/advanced/security.mdx): Protect agents from prompt injection, keep personal data from model providers, filter harmful content, gate risky tools, sandbox untrusted code and rate-limit your API. - [Multiple Users](https://cogitator.app/docs/advanced/multi-user.mdx): Serve many users with one agent — each with their own threads, memory and tool credentials, and no way to read each other's conversations. ## Deployment - [Deploy Package](https://cogitator.app/docs/deployment/deploy-package.mdx): One-command deployment for Cogitator agents — Docker and Fly.io targets with auto-detection. - [Docker Deployment](https://cogitator.app/docs/deployment/docker.mdx): Run Cogitator with Docker Compose — PostgreSQL, Redis, Ollama, and production container setup. - [Edge Runtimes](https://cogitator.app/docs/deployment/edge.mdx): Run Cogitator agents on Cloudflare Workers, Deno and Bun with the Hono adapter — what works, what to configure, and what needs a Node.js server. - [Redis Client](https://cogitator.app/docs/deployment/redis.mdx): Unified Redis client with standalone and cluster support — the foundation for memory, worker queues, and pub/sub. - [Worker Queues](https://cogitator.app/docs/deployment/worker-queues.mdx): Distributed job processing with BullMQ — background execution for agents, workflows, and swarms. - [Sandboxed Execution](https://cogitator.app/docs/deployment/sandbox.mdx): Run untrusted code safely with Docker containers and WASM isolation. - [Observability](https://cogitator.app/docs/deployment/observability.mdx): Trace agent executions with Langfuse and OpenTelemetry — cost tracking, span visualization, and production debugging. ## Testing - [Testing Overview](https://cogitator.app/docs/testing.mdx): Testing strategies for Cogitator tools, agents, workflows and swarms without calling a real LLM. - [Test Utilities](https://cogitator.app/docs/testing/test-utils.mdx): @cogitator-ai/test-utils — a scriptable mock LLM backend, a recording memory adapter, fixtures and async/stream helpers for testing Cogitator code. ## API Reference - [Core API](https://cogitator.app/docs/api-reference/core.mdx): Public exports of @cogitator-ai/core — runtime, agents, tools, LLM backends and the advanced subsystems — with their signatures. - [Swarms API](https://cogitator.app/docs/api-reference/swarms.mdx): Public exports of @cogitator-ai/swarms — the Swarm class and builder, strategies, communication, distributed execution, swarm tools, workflow nodes and the assessor — with their signatures. - [Workflows API](https://cogitator.app/docs/api-reference/workflows.mdx): Public exports of @cogitator-ai/workflows — builder, executor, nodes, checkpoints, human approvals, timers, sagas, map-reduce, subworkflows, triggers, run management and observability — with their signatures. - [Memory API](https://cogitator.app/docs/api-reference/memory.mdx): Public exports of @cogitator-ai/memory — memory adapters, embeddings, hybrid search, context building, facts, sessions and knowledge graphs — with their signatures. - [Types Reference](https://cogitator.app/docs/api-reference/types.mdx): Key TypeScript type definitions from @cogitator-ai/types. ## Architecture - [Architecture](https://cogitator.app/docs/architecture.mdx): How the Cogitator runtime and its packages fit together. ## Optional - [Full documentation](https://cogitator.app/llms-full.txt): every docs page above as Markdown in one file - [Cookbook](https://cogitator.app/cookbook): end-to-end recipes for agents, workflows and swarms - [Examples](https://github.com/cogitator-ai/Cogitator-AI/tree/main/examples): runnable TypeScript examples for every package - [Source code](https://github.com/cogitator-ai/Cogitator-AI): the monorepo on GitHub