This is the SAVE-worthy chapter. What you built works and teaches the whole RAG shape — but it's a demo on purpose. Six changes turn it into something you'd trust.

1. Auth and RLS — the functions are wide open

Right now both functions are public (--no-verify-jwt) and every call uses the service-role key, which bypasses RLS entirely. Anyone who finds your function URL can ingest into and query your knowledge base — there's one shared library and no notion of who's asking. For anything multi-tenant, add real per-user auth: verify a Supabase JWT at the top of each function, add a user_id (or tenant_id) column to documents, and write RLS policies keyed on auth.uid() so Postgres itself enforces who sees which chunks — instead of leaning on the service role to see everything. That's the exact fence-vs-identity trade the sibling MCP blueprints spell out: the service-role demo is the fast path; real per-user policies are the grown-up one.

2. Async ingestion — stop embedding in the request

The ingest handler embeds every chunk in sequence, inside the HTTP request — one gte-small inference per chunk, then a bulk insert. There's no network round-trip (the model runs in-process), but it's still CPU work on the function, so a short paste is fine while a 300-page PDF will blow past the function timeout and hold the browser hostage. Move ingestion off the request path: accept the upload, drop a job on a queue (Supabase has pgmq/Queues, or a background worker), and embed the chunks there — in parallel across workers, or on a beefier compute tier. Return a job id immediately and let the UI poll for "embedded 420/900 chunks."

3. Chunking — smarter than a sliding window

Fixed 1000-character windows are the simplest thing that works, and they cut sentences and tables in half. Upgrade to structure-aware chunking: split on headings, paragraphs, or sentence boundaries; keep tables and code blocks intact; attach metadata to each chunk (document title, section, page, date) so you can filter retrieval ("only chunks from the 2026 handbook") and show richer citations. Better chunks are the single highest-leverage retrieval-quality change you can make — the model can only ground on what a chunk actually contains.

4. Reranking — a second, sharper pass

Vector search is fast but blunt: cosine over the top-5 hnsw hits gets you roughly the right chunks, not the best-ordered ones. Add a reranker — retrieve a wider net (say the top 20), then run a cross-encoder or a reranking model (Cohere Rerank, a Gemini rerank pass) that scores each candidate against the actual question and keeps the best 5. You feed the model fewer, more relevant chunks, which lifts answer quality and cuts token cost at the same time. Retrieve wide, rerank, ground narrow.

5. Evals — measure before you trust

Nothing above is worth shipping until you can measure it. Build two eval sets: a retrieval eval (for a set of questions, did the right chunk make it into the top-k?) and an answer-quality eval (is the answer correct, grounded, and properly cited — LLM-as-judge or human-graded). Run them on every change to chunking, match_count, the prompt, or the reranker, so "this felt better" becomes "recall went from 0.71 to 0.88." Without evals you're tuning a RAG system blind, and RAG regressions are silent — the answer still looks confident.

6. Cost — generation, compute, and caching

Embeddings are free here — gte-small runs inside Supabase's Edge Runtime, so there's no per-vector charge, just the function's compute time. That leaves two real levers. Generation: you pay Gemini per answer; gemini-2.5-flash at temperature 0.2 is already the cheap, tight choice, and capping match_count and chunk size keeps each prompt lean. Cache identical questions so a repeated query doesn't re-bill. Rate limits: the public demo functions have no throttle — add per-IP or per-user rate limiting so a scraper (or a bug) can't run up your Gemini bill or peg your function compute. Set a spend alert in Google AI Studio / Cloud before you set anything else.

What you built

  • A deployed, one-page RAG app: upload text / PDF / URL, ask, get answers with sources.
  • A clean two-phase design — ingest (chunk → embed → store) and query (embed → retrieve → ground) — with retrieval as the tool the answer stands on.
  • Postgres + pgvector as the vector store, a match_documents() cosine function, and an hnsw index for speed.
  • Supabase's built-in gte-small for embeddings (384-dim, in-runtime, no external API) and Gemini for grounded generation — both server-side.
  • The two-keys discipline: an anon-key-only browser, service-role and Gemini keys locked in function secrets, and an RLS-fenced table nobody reaches by going around the functions.

You've got the retrieval-and-grounding core every agentic RAG system is built on. Wire that match_documents call into an agent's tool loop and the same machinery becomes agentic — but the hard part, the part you just built, is already done.