RAG Chatbot
RAG Chatbot
Architecture
- Chat Client — Web/mobile chat UI. Sends chat messages, streams answers, and uploads source documents via presigned S3 URLs.
- Cognito — User pool issuing JWTs. API Gateway validates the token on every request; the user sub scopes conversations and uploads.
- API Gateway — REST API: /conversations and /documents resources. JWT authorizer against Cognito, request validation, throttling.
- Chat Orchestrator — Handles a chat turn: loads recent history from DynamoDB, retrieves top-k chunks from the Knowledge Base, builds the augmented prompt, invokes Claude, persists the turn, returns answer + citations.
- Conversation Store — conversations, messages and documents tables. Messages keyed by (conversation_id, message_id ULID) so a Query returns the transcript in order.
- Bedrock Knowledge Base — Managed retrieval layer. Retrieve API embeds the query and runs a k-NN search over the vector store, returning chunks with scores and source metadata.
- Claude (Bedrock) — Foundation model. Receives system prompt + retrieved chunks + recent history; answers grounded in the retrieved context and cites sources.
- OpenSearch Serverless — Vector index (one collection, HNSW). Stores chunk embeddings + text + document metadata; queried by the Knowledge Base, written by the ingestion pipeline.
- Source Docs Bucket — Raw source documents (PDF, HTML, MD). Uploads arrive via presigned PUT; ObjectCreated events kick off ingestion.
- Ingestion Pipeline — Triggered per uploaded object (via SQS): extracts text, chunks (~500 tokens, 50 overlap), calls Titan for embeddings, bulk-indexes into OpenSearch, updates document status in DynamoDB.
- Titan Embeddings — amazon.titan-embed-text-v2 — embeds chunks at ingest time; the Knowledge Base uses the same model at query time so vectors are comparable.
Event flow
- Source Docs Bucket (S3) — Presigned PUT uploads land here; every ObjectCreated emits document.uploaded
- ingest-queue (SQS) — Buffers bursts of uploads; redrives to a DLQ after 3 failed receives
- Ingestion Worker (Lambda) — Extract text → chunk (~500 tokens, 50 overlap) → Titan embeddings → bulk index into OpenSearch. Publishes the outcome to doc-status
- doc-status (SNS) — Fan-out of ingestion outcomes
- Status Updater (Lambda) — Writes status, chunk_count, indexed_at / error_message to the documents table in DynamoDB
- Ops Alerts — Subscription filtered to document.failed → CloudWatch alarm / email
- document.uploaded — S3 ObjectCreated notification for a new source document
- document.indexed — Emitted after chunks are embedded and bulk-indexed into OpenSearch
- document.failed — Ingestion failed at extract, chunk, embed or index stage
About this design
- # About this design ## What this is A retrieval-augmented (RAG) chatbot on AWS serverless. Users chat with an assistant that answers **grounded in your own documents** — every answer carries citations back to the source chunks it used. The stack is: API Gateway + a single Lambda orchestrator for the chat turn, Bedrock Knowledge Base over an OpenSearch Serverless vector index for retrieval, Claude on Bedrock for generation, DynamoDB for conversation history, and an event-driven ingestion pipeline that turns S3 uploads into searchable vectors. There are no servers to run and everything scales to zero. ## How it works **Query path** (see the sequence diagram): the client POSTs to `/conversations/{id}/messages`. The orchestrator Lambda loads the last ~10 turns from DynamoDB, calls the Knowledge Base `Retrieve` API (which embeds the question and runs a k-NN search over OpenSearch), then invokes Claude with a prompt of *system instructions + retrieved chunks + history + question*. The prompt tells Claude to answer only from the provided context and to say so when nothing relevant was retrieved — that single instruction is most of your hallucination defence. Both turns are persisted with citations and token usage, and the answer returns as `{ message, citations, usage }`. **Ingestion path** (see the event flow): clients upload via presigned S3 PUT URLs issued by `POST /documents`. The `ObjectCreated` event lands on an SQS queue (bursty uploads, DLQ after 3 attempts); the ingestion worker extracts text, chunks at ~500 tokens with 50 overlap, embeds each chunk with Titan (`titan-embed-text-v2` — pinned per document so a model change is detectable), and bulk-indexes into OpenSearch. Outcomes fan out on an SNS topic: a status updater writes progress to the `documents` table, and a filtered subscription alerts on `document.failed`. The `/documents` API is how the UI shows ingestion progress. **Data model**: three DynamoDB tables. `messages` is keyed `(conversation_id, message_id)` with ULIDs, so one Query returns the transcript in order. Everything is scoped to the caller's Cognito sub. ## How to extend - **Streaming**: swap `InvokeModel` for `InvokeModelWithResponseStream` and stream tokens back via Lambda response streaming (or a WebSocket API for typing indicators). - **Better retrieval**: add hybrid search (BM25 + vector) in OpenSearch, or a reranking step over the top-20 before taking the top-5. - **Safety**: put Bedrock Guardrails in front of both the user input and the model output — it slots into the orchestrator with one API change. - **Multi-tenant**: add `tenant_id` to every table key and as a metadata filter on `Retrieve`. - **Evals**: log `(question, chunks, answer)` triples from the messages table — that is a ready-made dataset for a retrieval-quality harness before you tune chunk sizes.