RAG Chatbot
RAG Chatbot
Open & fork this on Zstem →Architecture
- AI / Retrieval: Query-time retrieval + generation: the Knowledge Base retrieves grounding chunks from the vector store and Claude answers over them.
- Ingestion Pipeline: Document ingestion path: extract, chunk, embed, and index. Runs asynchronously off S3 upload events, separate from the chat request path.
- Chat Client: Web/mobile chat UI. Sends chat messages, streams answers, and uploads source documents via presigned S3 URLs.
- Cognito: User pool issuing JWTs. API Gateway validates the token on every request; the user sub scopes conversations and uploads.
- API Gateway: REST API: /conversations and /documents resources. JWT authorizer against Cognito, request validation, throttling.
- Chat Orchestrator: Handles a chat turn: loads recent history from DynamoDB, retrieves top-k chunks from the Knowledge Base, builds the augmented prompt, invokes Claude, persists the turn, returns answer + citations.
- Conversation Store: conversations, messages and documents tables. Messages keyed by (conversation_id, message_id ULID) so a Query returns the transcript in order.
- Bedrock Knowledge Base: Managed retrieval layer. Retrieve API embeds the query and runs a k-NN search over the vector store, returning chunks with scores and source metadata.
- Claude (Bedrock): Foundation model. Receives system prompt + retrieved chunks + recent history; answers grounded in the retrieved context and cites sources.
- OpenSearch Serverless: Vector index (one collection, HNSW). Stores chunk embeddings + text + document metadata; queried by the Knowledge Base, written by the ingestion pipeline.
- Source Docs Bucket: Raw source documents (PDF, HTML, MD). Uploads arrive via presigned PUT; ObjectCreated events kick off ingestion.
- Ingestion Worker: Triggered per uploaded object (via SQS): extracts text, chunks (~500 tokens, 50 overlap), calls Titan for embeddings, bulk-indexes into OpenSearch, updates document status in DynamoDB.
- Titan Embeddings: amazon.titan-embed-text-v2 — embeds chunks at ingest time; the Knowledge Base uses the same model at query time so vectors are comparable.
Event flow
- Source Docs Bucket (S3): Presigned PUT uploads land here; every ObjectCreated emits document.uploaded
- ingest-queue (SQS): Buffers bursts of uploads; redrives to a DLQ after 3 failed receives
- Ingestion Worker (Lambda): Extract text → chunk (~500 tokens, 50 overlap) → Titan embeddings → bulk index into OpenSearch. Publishes the outcome to doc-status
- doc-status (SNS): Fan-out of ingestion outcomes
- Status Updater (Lambda): Writes status, chunk_count, indexed_at / error_message to the documents table in DynamoDB
- Ops Alerts: Subscription filtered to document.failed → CloudWatch alarm / email
- document.uploaded: S3 ObjectCreated notification for a new source document
- document.indexed: Emitted after chunks are embedded and bulk-indexed into OpenSearch
- document.failed: Ingestion failed at extract, chunk, embed or index stage
Sequence
Participants: Chat Client, API Gateway, Chat Orchestrator (Lambda), DynamoDB, Bedrock Knowledge Base, OpenSearch Serverless, Claude (Bedrock)
- Chat Client→API GatewayPOST /conversations/{id}/messages{ message } with Cognito JWT
- API Gateway→Chat Orchestrator (Lambda)invoke (proxy event)JWT already validated by the Cognito authorizer
- Chat Orchestrator (Lambda)→DynamoDBQuery last 10 turnsmessages table, key = conversation_id, ULID-ordered
- DynamoDB→Chat Orchestrator (Lambda)conversation history
- Chat Orchestrator (Lambda)→Bedrock Knowledge BaseRetrieve(question, k=5)
- Bedrock Knowledge Base→OpenSearch Serverlessembed query + k-NN searchSame Titan embedding model used at ingest time
- OpenSearch Serverless→Bedrock Knowledge Basetop-k chunks + scores
- Bedrock Knowledge Base→Chat Orchestrator (Lambda)chunks + source metadata
- Chat Orchestrator (Lambda)→Claude (Bedrock)InvokeModel: system + chunks + history + questionPrompt instructs Claude to answer only from the provided context and cite sources; if no chunk clears the score threshold it must say it does not know
- Claude (Bedrock)→Chat Orchestrator (Lambda)grounded completion
- Chat Orchestrator (Lambda)→DynamoDBPutItem user turn + assistant turn (+ citations, usage)
- Chat Orchestrator (Lambda)→API Gateway{ message, citations, usage }
- API Gateway→Chat Client200 OK
API contract
- POST
/conversationsStart a new conversation - GET
/conversationsList the caller's conversations - DELETE
/conversations/{conversationId}Delete a conversation and its messages - GET
/conversations/{conversationId}Get a conversation with its full transcript - POST
/conversations/{conversationId}/messagesSend a message and get a grounded answer - POST
/documentsRegister a document and get a presigned upload URL - GET
/documentsList source documents with ingestion status - DELETE
/documents/{documentId}Delete a document and its vectors - GET
/documents/{documentId}Get a document's ingestion status
Database
| conversation_id | UUID v4 |
|---|---|
| user_id | Cognito sub of the owner — every access is scoped to this |
| title | Auto-generated from the first user message, editable |
| message_count | |
| last_message_preview | First 120 chars of the latest turn, for list rendering |
| created_at | |
| updated_at |
| conversation_id | |
|---|---|
| message_id | ULID — lexicographic sort == chronological order |
| role | user | assistant |
| content | |
| citations | assistant turns only: [{documentId, excerpt, score}] used to ground the answer |
| model_id | e.g. anthropic.claude-sonnet-4-5, recorded per assistant turn |
| input_tokens | |
| output_tokens | |
| latency_ms | |
| created_at |
| document_id | UUID v4, also embedded in the S3 key |
|---|---|
| s3_key | uploads/{document_id}/{filename} |
| filename | |
| content_type | |
| size_bytes | |
| status | uploaded | chunking | embedding | indexed | failed |
| chunk_count | Set when indexing completes |
| embedding_model | amazon.titan-embed-text-v2 — pinned per document so reindexing is detectable |
| error_message | Populated when status = failed (stage + cause) |
| uploaded_by | Cognito sub |
| created_at | |
| indexed_at |
About this design
About this design
What this is
A retrieval-augmented (RAG) chatbot on AWS serverless. Users chat with an assistant that answers grounded in your own documents — every answer carries citations back to the source chunks it used. The stack is: API Gateway + a single Lambda orchestrator for the chat turn, Bedrock Knowledge Base over an OpenSearch Serverless vector index for retrieval, Claude on Bedrock for generation, DynamoDB for conversation history, and an event-driven ingestion pipeline that turns S3 uploads into searchable vectors. There are no servers to run and everything scales to zero.
How it works
Query path (see the sequence diagram): the client POSTs to /conversations/{id}/messages. The orchestrator Lambda loads the last ~10 turns from DynamoDB, calls the Knowledge Base Retrieve API (which embeds the question and runs a k-NN search over OpenSearch), then invokes Claude with a prompt of system instructions + retrieved chunks + history + question. The prompt tells Claude to answer only from the provided context and to say so when nothing relevant was retrieved — that single instruction is most of your hallucination defence. Both turns are persisted with citations and token usage, and the answer returns as { message, citations, usage }.
Ingestion path (see the event flow): clients upload via presigned S3 PUT URLs issued by POST /documents. The ObjectCreated event lands on an SQS queue (bursty uploads, DLQ after 3 attempts); the ingestion worker extracts text, chunks at ~500 tokens with 50 overlap, embeds each chunk with Titan (titan-embed-text-v2 — pinned per document so a model change is detectable), and bulk-indexes into OpenSearch. Outcomes fan out on an SNS topic: a status updater writes progress to the documents table, and a filtered subscription alerts on document.failed. The /documents API is how the UI shows ingestion progress.
Data model: three DynamoDB tables. messages is keyed (conversationid, messageid) with ULIDs, so one Query returns the transcript in order. Everything is scoped to the caller's Cognito sub.
How to extend
Streaming: swap InvokeModel for InvokeModelWithResponseStream and stream tokens back via Lambda response streaming (or a WebSocket API for typing indicators).
Better retrieval: add hybrid search (BM25 + vector) in OpenSearch, or a reranking step over the top-20 before taking the top-5.
Safety: put Bedrock Guardrails in front of both the user input and the model output — it slots into the orchestrator with one API change.
Multi-tenant: add tenant_id to every table key and as a metadata filter on Retrieve.
Evals: log (question, chunks, answer) triples from the messages table — that is a ready-made dataset for a retrieval-quality harness before you tune chunk sizes.