Build log · DocuMind AI
How DocuMind is actually built
I have no client references yet, so instead of a testimonial here is the engineering reasoning behind my own production system, taken from its system documentation. Judge the thinking, not a logo wall.
01 · SourceFile, URL or site crawlUpload is scanned before anything else touches it; crawls are bounded and resumable.
02 · ParseExtract + paragraph-aware chunking~800-token chunks with 150-token overlap, page and section provenance attached at this step.
03 · Embedtext-embedding-3-small into pgvectorVectors are written per tenant; RLS scopes every read to the owning workspace.
04 · RetrieveSimilarity search + rerankRetrieval quality is tuned before model choice. Only whitelisted context reaches the prompt.
05 · AnswerGemini 2.5 Flash, citations enforcedA fixed system template answers only from retrieved chunks and returns source links with each claim.
Every stage runs server-side. The browser holds publishable keys only, which is why the widget can be embedded on a customer's own domain without exposing anything.
- What the product does: DocuMind AI is an enterprise, multi-tenant knowledge base and conversational AI platform. Customers upload documents by file, single URL or full site crawl; the system parses, chunks and embeds them, and end users get cited answers either in the dashboard chat or through an embeddable JavaScript widget on their own site.
- The governing principle: the browser is hostile: The architecture is server-authoritative. Every business rule, quota check, security decision and AI call executes server-side; the browser holds nothing but publishable keys. Row Level Security across 60+ tables is the final safety net rather than the first line of defence. That principle is what makes the rest of the decisions below consistent instead of ad hoc.
- Stack, and why each piece is there: React 19 with TanStack Router/Start on Cloudflare Workers (nodejs_compat) for edge SSR without servers to operate. Supabase Postgres 15 with pgvector, pg_cron and pg_net so retrieval, scheduling and outbound calls all live in one managed database. Gemini 2.5 Flash for generation through a managed gateway, text-embedding-3-small for embeddings. Resend behind a queue table for mail, Sentry for errors with a PII-scrubbing logger.
- Ingestion: the boring parts are the ones that fail in production: Uploads go through /api/ingest, /api/ingest-url or /api/ingest-crawl. Files are virus-scanned and MIME/size validated before anything is parsed, PDFs and HTML are parsed into pages with page numbers preserved, and only then chunked. Preserving page numbers at parse time is non-negotiable: provenance cannot be retrofitted later without re-ingesting the whole corpus.
- Chunking: ~800 tokens with 150 overlap, paragraph-aware: Large chunks look efficient because they cut embedding count and cost. In practice they bury the answer in noise and no reranker rescues a chunk that is mostly irrelevant. Smaller paragraph-aware chunks with overlap mean an answer that straddles a boundary still survives retrieval. It costs more embeddings and it is the right trade, because retrieval precision is what a user actually experiences.
- Retrieval before model choice: Swapping models moves answer quality far less than people expect. What moves it is retrieval quality and a prompt built from cited context, streamed back with citations rendered inline. Citations are also the cheapest hallucination defence there is: when a user can click through to the source, a wrong answer becomes visible instead of persuasive.
- Multi-tenancy from the first migration, and what it costs: Everything — documents, chunks, chats, usage — is scoped to workspace_id, with membership and roles in workspace_members and 61 SQL migrations carrying schema, grants, RLS and policies together. Every public-schema table ships GRANT, ENABLE RLS and its policies in the same migration as its CREATE TABLE. This is genuinely more work up front and on a single-tenant pilot it would be over-engineering. I did it because retrofitting tenant isolation into a live system means touching every query, and the failure mode is one tenant reading another's documents.
- Auth: MFA, bot protection and leaked-password checks: Signup passes through a server-side gate that enforces per-IP and per-email rate limits, Cloudflare Turnstile verification and password validation including a HIBP leaked-password check. TOTP MFA runs on Supabase Auth factors: after password login the session is aal1, and if a verified factor exists a guard forces step-up to aal2 before protected work. Recovery codes are stored hashed and are single-use. Password reset always returns a generic success message so the endpoint cannot be used to enumerate accounts.
- The widget: HMAC tokens, because the obvious approach is unsafe: An embeddable widget carrying an API key in page source is a public API key. Instead the widget mints a short-lived HMAC token server-side, bound to the calling origin and backed by an origin allow-list and a circuit breaker. An attacker who copies the embed snippet onto their own domain gets nothing. The same public surface handles widget chat, escalation to a human, action tracking and voice.
- Quotas and rate limiting live in the database: Usage aggregates per day in workspace_usage_counters, plan_limits defines caps per plan, and every AI or embedding call checks limits before executing with denials written to usage_deny_log. Rate limiting is a token bucket in rate_limit_buckets with stricter buckets on widget endpoints. Cost is computed from actual model pricing per workspace, so an unusual bill can be traced to a tenant rather than guessed at.
- Email is queued, never fire-and-forget: Branded React templates render on demand from an auth webhook and queue into email_send_log, drained by a processor through Resend. Each email carries an idempotency key so a retry cannot double-send, unsubscribe tokens are stored, and bounces and complaints hard-suppress the address. A failed dispatch stays in the log with an attempt counter and lands in a dead-letter state after N tries instead of vanishing.
- Compliance treated as schema, not a footer link: Terms, Privacy, DPA, NDA and the security policy are versioned rows in legal_documents, with acceptance recorded per user including IP and user agent. On a version bump users must re-accept before protected server functions run at all. GDPR deletion requests can be filed by end users through the widget or by authenticated users in the dashboard, and are processed from the admin console.
- Failure modes designed for, not discovered later: AI gateway timeout opens a circuit breaker and serves a cached answer where one exists rather than hanging. A brief database outage is absorbed by query retries and an error boundary instead of a white screen. A suspend-bypass attempt is blocked at the route guard, the auth ban and the database RPC. A prompt injection embedded in an uploaded document meets a fixed system template that cites only whitelisted context.
- Honest state of the product: DocuMind is my own product, not a client engagement. There are no online payments — Pro upgrades are a request an admin approves. Stripe self-serve, SAML SSO, a public API with key management and a multi-region embedding cache are roadmap, and they are labelled roadmap here for the same reason I would label them that way in a proposal.
- What I would do differently: Build the eval harness before the second feature rather than after the tenth, because a prompt tweak that improves tone while quietly dropping citation accuracy is invisible without a gate. Keep the schema smaller until a real second tenant exists. And measure time-to-first-token from the user's browser rather than from the server, because only one of those numbers is what a person actually feels.
- How this maps to your build: The discipline is what you are buying: retrieval before model choice, provenance from ingestion, limits enforced in the database, secrets never in the browser, ADRs recording every rejected alternative, and numbers measured rather than quoted. If any decision above looks wrong to you, say so in your first message — I would rather argue about architecture before an invoice than after one.
- See it running: DocuMind is live at documind.agenticcore.tech. Open it, upload your own document and try to break it. That is a more useful reference than anything I could write here.
Questions about any of this? Email daniyal@agenticcore.tech — I reply within one business day.