Building Cobraine: A Telegram Bot That Became My Second Brain

I wanted a place to dump everything. Notes, PDFs, random photos of whiteboards, half-baked ideas at 2am. Not another app I'd forget to open. Something that lived where I already was: Telegram.

So I built Cobraine. It's a conversational AI assistant that runs entirely inside Telegram. You send it stuff, it remembers. You ask for something back, it finds it. You say "remind me tomorrow at 9am," it does. And behind a surprisingly thin chat interface, there's a full-blown agentic system with tool calling, vector search, scheduled jobs, and encrypted multi-provider LLM routing.

This post walks through how I architected it, the trade-offs I made, and the parts I'm actually proud of.

---

The Problem I Was Solving

I kept losing things. I'd save a note in Apple Notes, a PDF in Google Drive, a bookmark in Safari, a voice memo in the recorder app. When I needed to find something three weeks later, I had no idea where it went. And none of these tools talked to each other.

I wanted one place where I could throw everything and later just ask for it in natural language. "What was that article about distributed caching?" or "Find the photo I saved of that architecture diagram." No folders, no tags, no organizational discipline required.

Telegram was the obvious surface. I already had it open all day. The bot API is solid. File handling (photos, documents, audio, voice memos) works out of the box. And with a mini-app integration, I could always bolt on a richer UI later.

---

Architecture Overview

The stack is TypeScript end to end. Here's how the pieces fit together:

Runtime layer:

• Grammy (Telegram Bot framework) handles all inbound updates
• Three composers split concerns: /start onboarding, /config for LLM setup, and the main Cobraine composer for everything else
• A TelegramReceiver class extracts and normalizes each inbound message into a typed domain object
• A TelegramResponder class owns all outbound Telegram API calls (thinking indicators, final responses, inline keyboards)

Agent layer:

• LangGraph (from LangChain) runs a ReAct-style agent loop: START → agent → tools → agent → ... → END
• 16 custom tools handle everything from saving notes to semantic search to reminder scheduling to file management
• Graph interrupts (via LangGraph's interrupt()) let the agent pause mid-workflow, ask the user a question, and resume when they answer
• A MemorySaver checkpointer persists graph state across Telegram messages for the same chat thread

Data layer:

• PostgreSQL with pgvector for structured storage and vector similarity search
• Redis for reminder caching (cache-aside pattern) and as the BullMQ transport
• Voyage AI for text embeddings (description embeddings on files, chunk embeddings on items)
• AES-256-GCM encryption for API keys at rest

Background processing:

• BullMQ for delayed job scheduling (reminders fire at the right time via delayed queue jobs)
• A separate reminder worker process consumes the queue and delivers notifications with inline action buttons

---

How a Message Flows Through the System

When someone sends a message, here's roughly what happens:

1. Grammy receives the Telegram update and routes it to the Cobraine composer.
2. TelegramReceiver.fromCtx() extracts the telegram ID, chat ID, text/caption, and any attachments (documents, photos, videos, audio, voice). It upserts the user in Postgres and runs usage gating.
3. Usage gating checks: does this user have their own LLM provider configured? If yes, use their key. If no, are they under the free message limit? If yes, use the shared key and atomically increment their counter. If neither, deny with a message telling them to add their API key.
4. The buildLLM() factory constructs a ChatOpenAI instance. It supports OpenRouter, OpenAI, DeepSeek, Together, and Groq since they all expose OpenAI-compatible endpoints. Each request builds a fresh instance so stale credentials are never cached.
5. buildMessageContext() loads conversation history from Postgres, formats any attachments as structured metadata in the prompt, and assembles the full message array.
6. The LangGraph agent runs. The system prompt, environment details (timezone handling, user name), and the message history are fed in. The agent can call any of the 16 tools across multiple turns.
7. When the agent finishes, the response is saved to chat history and sent back via TelegramResponder.

If the agent calls ask_user (a tool that triggers a graph interrupt), the whole graph pauses. The user sees a question, possibly with inline keyboard options. Their next message resumes the graph from exactly where it left off, with the answer injected.

---

The Parts That Were Tricky

Timezone-Aware Reminders

Reminders sound simple until you realize users are in different timezones and most of them just say "remind me at 9am" without specifying which 9am they mean. Here's how I handled it:

The first time a user tries to set a reminder, the agent checks their stored timezone. If it's UNSET, the agent asks them what timezone they're in, then saves it via a set_user_timezone tool. From that point on, all time parsing happens relative to their timezone.

Date and time parsing is its own utility module. It handles inputs like "25/12", "25 Dec", "9am", "21:30", "9:30pm" and combines them into a proper Date object in the user's timezone. The resulting UTC timestamp goes into Postgres and gets scheduled as a BullMQ delayed job.

When the job fires, the worker sends a Telegram notification with "Done" and "Snooze" inline buttons. Snooze options (15m, 30m, 1h, 2h) reschedule the same job. If it's a goal checkpoint reminder, completing or snoozing it triggers the agent to give motivational feedback or a nudge.

Encrypted Multi-Provider LLM Routing

I wanted users to bring their own API keys so they're not dependent on my free tier. But storing raw API keys in Postgres felt wrong, so every key is encrypted with AES-256-GCM before it hits the database. The encryption key lives in an environment variable and never in code.

The /config command walks users through a multi-step inline keyboard flow: pick a provider, paste your API key (the message is immediately deleted for security), pick a model (the bot fetches available models from the provider's API), confirm. The whole flow happens inside a single evolving Telegram message, no page refreshes.

The buildLLM() factory function maps provider names to their base URLs. Since OpenRouter, OpenAI, DeepSeek, Together, and Groq all expose OpenAI-compatible chat completion endpoints, every provider gets the same ChatOpenAI class with just the base URL swapped. This kept the integration surface tiny.

Semantic Search Across Everything

Every note, PDF text, and file description gets chunked and embedded via Voyage AI. Chunks go into a chunks table, embeddings into an embeddings table with a pgvector column. File descriptions get their own description_embedding column directly on the files table.

When a user asks "find that thing about X," the query gets embedded with the same model, and a cosine similarity search runs against pgvector. The agent gets the top matches and composes a response.

I went with Voyage over OpenAI embeddings because of the cost per token and the quality on retrieval benchmarks for the kind of mixed-format content I was indexing.

Graph Interrupts for Multi-Turn Conversations

This was probably the most interesting architectural decision. LangGraph supports interrupt(), which lets a tool throw a special signal that pauses the entire agent graph mid-execution. The checkpointer saves the complete graph state (which node was executing, what the tool stack looked like, accumulated messages).

I use this for the ask_user tool. When the agent needs clarification ("Which timezone are you in?" or "I found 3 matching files, which one do you want?"), it calls ask_user with a question and optional button labels. The graph pauses. The user sees an inline keyboard or a free-text prompt. Their response resumes the graph from the exact point it stopped, with their answer injected.

The in-process interruptedSessions map tracks which users have a paused graph. When their next message comes in, it routes to resumeAgentGraph() instead of starting a fresh invocation.

---

Goal Coaching and Checkpoint Reminders

This is a feature I added because I kept telling myself I'd start running and never did. Cobraine can act as a goal coach.

When you tell it you want to build a habit or achieve something, it doesn't immediately create a schedule. It pushes back. It asks what your actual motivation is, what might get in the way, what a realistic timeline looks like. Only after that discussion does it create a plan (stored in Postgres), break it into checkpoints, and schedule reminder jobs for each one.

When a checkpoint reminder fires and you hit "Done," the agent responds with genuine motivational feedback. If you hit "Snooze," it gets pushy. Not in an annoying way, but in a "hey, you said this mattered to you, don't stall" way.

The plan and checkpoint data lives in its own table. Each checkpoint reminder carries a checkpointId in its metadata so the system knows to trigger the coaching response instead of a plain acknowledgment.

---

File Management

Cobraine handles photos, documents, videos, audio, and voice messages. When a file comes in, the agent decides what to do based on context. If the user says "save this," the save_file tool stores the Telegram file ID, downloads and saves the file locally, extracts text from PDFs, embeds the description, and writes everything to the files table.

Retrieval uses a two-strategy approach: first try the Telegram file_id (instant, zero bandwidth), and if that's expired or invalid, fall back to streaming the local copy. This means files are almost always served instantly without re-downloading.

Search works both semantically (via description embeddings) and structurally (by type, mime type, keyword). The agent picks the right approach based on how the user asks.

---

What I'd Change

The checkpointer. Right now it's MemorySaver, which means it's in-process memory. If the bot restarts, all interrupted sessions are lost. I'd swap this for a Postgres-backed checkpointer for durability. The LangGraph team has one, I just haven't migrated yet.

Embedding cost. Every save triggers an embedding call. For a single user it's fine. For hundreds it'd get expensive fast. I'd add a queue for embedding jobs so they can be batched and rate-limited.

The mini-app. There's a Next.js mini-app scaffolded in the repo but it's not wired up yet. The idea is to give users a richer settings UI, usage dashboard, and file browser instead of doing everything through chat commands.

Testing. There's a test directory with tool-level tests, but end-to-end flow testing through the Grammy bot is something I want to build out. Mocking the Telegram API to simulate full message flows would catch a lot of integration-level bugs.

---

Stack Summary

---

Wrapping Up

Cobraine started as a weekend project to stop losing my notes and turned into a full agentic system with multi-provider LLM routing, vector search, scheduled jobs, encrypted credential storage, and goal coaching. The core insight that made it work is that Telegram is a genuinely good platform for this kind of application. The bot API handles most of the UI complexity, and for everything it can't do, inline keyboards and mini-apps fill the gap.

If you're thinking about building something similar, my biggest piece of advice: get the TelegramReceiver / TelegramResponder separation right early. Keeping the Telegram API surface completely isolated from your domain logic makes everything downstream cleaner. Your agent, your services, your tools should never import Grammy or touch ctx directly.

The code is structured so that swapping out any single layer (the LLM provider, the embedding model, the queue system, even the chat platform) doesn't ripple through the rest of the codebase. That's not accidental. That's what made the project survivable as it grew.