Giving My AI Agents a Shared Brain (Self-Hosted, Third Attempt)

How my Claude Code and Antigravity sessions ended up sharing one long-term memory — a self-hosted Hindsight instance on a Proxmox LXC — after two failed prototypes.

The Problem: Agents With Amnesia

I live in AI coding agents. Claude Code for most work, Antigravity on the side, multiple machines, dozens of sessions a week. And every single session starts the same way: the agent knows nothing about my infrastructure, my past decisions, or that time we already debugged this exact issue.

Pasting context by hand is a tax. CLAUDE.md files help, but they're static — someone has to curate them, and that someone keeps forgetting.

What I wanted was simple to say and annoying to build: one long-term memory, shared by every agent on every machine, that remembers things automatically.

Attempts One and Two: The Graveyard

This took three tries over two months, which I'm told is the traditional number.

Attempt 1: DIY pgvector gateway

A custom FastAPI service in front of Postgres + pgvector. It worked in the demo sense: embed text, store vector, cosine-search it later. But "a table of embeddings" is not a memory. No fact extraction, no deduplication, no consolidation — recall quality degraded as junk accumulated. I was slowly reinventing a memory engine, badly.

Attempt 2: mem0 gateway

Next I wrapped mem0 in a FastAPI + FastMCP gateway on its own LXC container. Better — it at least had the concept of extracting memories from conversations. But I was still maintaining a custom gateway, custom auth, custom collection routing, and the retrieval quality was mediocre. It became the thing I debugged instead of the thing that helped.

Attempt 3: Hindsight, Off the Shelf

Third attempt: stop building, start deploying. I moved to Hindsight running in Docker on a dedicated Proxmox LXC:

  • One container, bundled Postgres included. ghcr.io/vectorize-io/hindsight, done.
  • Local embeddings — bge-small-en-v1.5 (384-dim) runs inside the container. No embedding API bills, no data leaving the box for retrieval.
  • Local cross-encoder rerank on top of hybrid semantic + keyword + graph search.
  • The only external call: OpenRouter gpt-4o-mini for fact extraction and consolidation when memories are written.

The retained/recall model is what I'd been failing to build by hand. You retain raw conversation content; Hindsight extracts discrete facts and entities, consolidates them against what it already knows, and preserves high-entropy strings (IPs, IDs, serial numbers) verbatim. recall does the hybrid retrieval; reflect runs an LLM over the memory bank for questions that need reasoning, not just lookup.

Wiring It Into Everything

The magic ingredient is that Hindsight speaks MCP over streamable HTTP, so every agent that supports MCP gets the same brain with one config line:

claude mcp add --transport http --scope user memory \
  http://<lxc-ip>:8888/mcp/main/ \
  --header "Authorization: Bearer $HINDSIGHT_API_KEY"

The memory bank is chosen by the URL path — I started with separate homelab and work-mac banks, then merged everything into one main bank. Separation felt organized; in practice it just meant recall missed things that were filed in the other bank.

Antigravity points at the same endpoint through its shared MCP config. Two different agent ecosystems, one memory. Ask either of them "what do you know about my SFP experiment" and they both recall the same saga — including the part where it overheated.

The Privacy U-Turn

For the first iteration I exposed the API publicly through a Cloudflare Tunnel with bearer auth, so agents could reach it from anywhere.

Then I thought about what actually lives in this database: my network topology, hostnames, container IDs, infrastructure decisions... basically a burglar's guide to my digital house, one bearer token away.

Same day, the public hostname was deleted. The service is now LAN and Tailscale only — remote machines reach it through a Tailscale subnet router advertising the home network. Slightly less convenient, dramatically less "please index my infrastructure."

Verdict After the Dust Settled

  • Buy (well, self-host) beats build for memory engines. The interesting problem is what you do with memory, not implementing consolidation from scratch.
  • Local embeddings matter more than I expected — retention and recall happen constantly, and constantly hitting a paid embedding API would add up fast.
  • One bank, not many. Your agents can't recall what's filed in a bucket they didn't look in.
  • The third attempt being "just run the good open-source thing" is a home-lab tradition at this point. See also: every monitoring stack I've ever built.

The lab now remembers. Which is more than I can say for myself.

← Back to blog