LLM Training & Context Engineering Pipeline

Architecture

• Data Ingestion & Prep
• Context Engineering
• Training & Fine-Tuning
• Inference & Serving
• Observability & Feedback
• Raw Corpus (S3): Source documents, code, transcripts, scraped data landing zone
• ETL / Clean & Dedup: Parse, clean, deduplicate, PII scrub, chunk documents
• Curated Dataset (S3): Cleaned, chunked, versioned training + retrieval corpus
• Embedding Model: Generates vector embeddings for corpus chunks and queries
• Vector Store: OpenSearch/Pinecone index of corpus embeddings for retrieval
• RAG Pipeline: Retrieve top-k chunks, rerank, assemble grounded context window
• Prompt & System Prompt Store: Versioned system prompts, templates, few-shot examples
• Context Assembler: Merges system prompt + retrieved context + user query into final payload
• Training / Fine-Tune Jobs: SFT, LoRA/PEFT and eval runs on curated dataset
• Model Registry: Versioned model + adapter artifacts, eval-gated for promotion
• Eval Harness: Automated quality, safety and regression evals before release
• API Gateway: Entry point for client inference requests
• Inference Orchestrator: Calls context assembler, invokes model, applies guardrails
• LLM Endpoint: Serves base or fine-tuned model for generation
• Guardrails: Input/output content filtering and safety policies
• Metrics & Logs: Latency, token usage, retrieval hit rate, model performance
• Feedback Stream: Captures user feedback + traces to feed retraining loop
• Client App: End-user application sending prompts

Workflow

• Incoming Request: Client sends prompt/query to the public inference API
• API Gateway: TLS termination, request routing, correlation ID assigned
• WAF / Edge Filtering: Block known bad IPs, SQLi/XSS patterns, DDoS shielding
• Threat detected?
• Authenticate: Verify API key / JWT / OAuth token against identity provider
• Identity valid?
• Authorize: Check scopes/roles and tenant entitlement for this endpoint
• Permitted?
• Rate Limit / Quota: Per-key and per-tenant token-bucket + monthly quota check
• Within limit?
• Validate Payload: Schema validation, size caps, required fields, content-type
• Well-formed?
• Input Moderation: Pre-flight safety / prompt-injection / PII screening
• Passes policy?
• Normalize & Enrich: Canonicalize prompt, attach tenant/user metadata, correlation ID
• Enqueue for Pipeline: Publish accepted request to the inference/context queue
• 403 Blocked: Rejected by WAF
• 401 Unauthorized: Authentication failed
• 403 Forbidden: Authorization failed
• 429 Too Many Requests: Rate limit or quota exceeded; Retry-After returned
• 400 Bad Request: Validation failed
• 422 Policy Violation: Blocked by input moderation
• Accepted: Request admitted to the pipeline
• Rejected: Error response returned to client

Event flow

• Inference Orchestrator: Emits traces, quality logs, and fault events for every request
• LLM Endpoint: Emits per-call latency, token counts, and failure signals
• RAG / Retrieval: Emits retrieval latency and hit-rate spans
• Client App: Emits thumbs up/down and comments on completions
• Telemetry Stream: High-volume OTel trace/metric stream
• Quality Log Topic: Prompt/completion pairs for quality scoring
• Feedback Queue: User feedback events
• Fault Event Topic: External-dependency failure events
• Observability Platform: Traces, dashboards, distributed tracing (Grafana/X-Ray/OpenSearch)
• Cost & Latency Analytics: Aggregates token spend & latency SLOs per tenant/model
• Quality Eval Service: LLM-as-judge + heuristic scoring (groundedness, relevance, toxicity), correlated with user ratings
• Drift Detector: Detects response/embedding drift vs baseline distribution
• Prompt Version Registry: Stores prompt/system-prompt versions with aggregated quality scores; gates promotion of new versions
• Alerting & On-Call: Triggers alerts/paging on error spikes, quality regressions, drift
• inference.trace: One span of an inference request: latency, tokens, model/prompt version, retrieval hits, tool calls
• llm.completion.logged: Prompt/response pair captured for offline & online quality scoring
• user.feedback: End-user feedback on a completion
• external.call.failed: External API failure: timeout, 5xx, rate-limit, circuit-breaker trip

Sequence

Participants: Request Queue, Inference Orchestrator, Context Assembler, RAG / Vector Store, Guardrails, LLM Endpoint, Tool / Action Layer, Session Memory, Metrics & Feedback

• Request Queue→Inference OrchestratorDequeue accepted request
• Inference Orchestrator→Session MemoryFetch session historyLoad prior turns / agent scratchpad for this session
• Session Memory→Inference OrchestratorConversation state
• Inference Orchestrator→Context AssemblerBuild context
• Context Assembler→RAG / Vector StoreRetrieve grounding docsEmbed query, retrieve top-k chunks, rerank
• RAG / Vector Store→Context AssemblerRanked context chunks
• Context Assembler→Inference OrchestratorAssembled prompt payloadSystem prompt + retrieved context + history + user query
• Inference Orchestrator→GuardrailsPre-flight input checkPrompt-injection / policy screen on final payload
• Guardrails→Inference OrchestratorCleared
• Inference Orchestrator→LLM EndpointInvoke model (reasoning step)Model may return a final answer or request a tool call
• LLM Endpoint→Inference OrchestratorTool call requested
• Inference Orchestrator→Tool / Action LayerExecute toolExecute function/API/retrieval the model asked for
• Tool / Action Layer→Inference OrchestratorTool result
• Inference Orchestrator→Session MemoryPersist intermediate stateAppend observation to scratchpad for next reasoning step
• Inference Orchestrator→LLM EndpointRe-invoke with tool resultLoop continues until model returns a final answer
• LLM Endpoint→Inference OrchestratorFinal completion
• Inference Orchestrator→GuardrailsPost-generation checkOutput moderation, PII redaction, safety policies
• Guardrails→Inference OrchestratorApproved / redacted
• Inference Orchestrator→Session MemoryPersist final turn
• Inference Orchestrator→Metrics & FeedbackEmit traces & metricsLatency, tokens, retrieval hit rate, tool usage, user feedback
• Inference Orchestrator→Request QueueStream response to client

Observability, Prompt Versioning, Quality & Fault Tolerance

Observability, Prompt Versioning, Quality & Fault Tolerance

Cross-cutting design for the inference path. Pairs with the event-flow diagram (telemetry producers → channels → consumers) and the sequence diagram (agentic loop).

LLM Observability

Every request carries a traceId (correlation ID assigned at the API Gateway) propagated through orchestrator → assembler → RAG → model → tools, so a single trace reconstructs the full agentic loop.

Capture per span:

Latency — end-to-end and per hop (retrieval, each model call, each tool call).

Tokens — input/output counts per call, aggregated per request and per tenant.

Retrieval quality signals — chunks retrieved, rerank scores, hit rate.

Agentic shape — number of reasoning iterations, tool calls per request, loop-termination reason.

Model & prompt identity — modelId + promptVersion on every span, so metrics slice by version.

Emitted as inference.trace to the telemetry stream, fanning out to the observability platform (dashboards, distributed tracing) and cost/latency analytics (token spend + SLO tracking per tenant/model). Use OpenTelemetry semantics so traces, metrics, and logs share the same IDs.

Prompt & System-Prompt Versioning

Every prompt and system prompt is content-addressed: store a systemPromptHash and a human-readable promptVersion (e.g. semver or name@rev).

The Prompt Version Registry holds each version plus its aggregated quality scores. Versions are immutable once published; a change is a new version, never an in-place edit.

Every inference.trace and llm.completion.logged event stamps the promptVersion used, so quality and cost are always attributable to an exact prompt revision.

Promotion is eval-gated: a new version must beat the incumbent on the eval suite before it can be marked production. Rollout can be staged (canary %). Rollback = re-point the alias to the previous version — instant, no redeploy.

LLM Call Quality

Two loops:

Online — sampled llm.completion.logged pairs scored in near-real-time by the Quality Eval Service (LLM-as-judge + heuristics: groundedness against retrieved context, relevance, format compliance, toxicity/safety). User up/down feedback is joined on traceId to correlate automated scores with real satisfaction.

Offline — a curated eval set is run against each candidate prompt/model version in CI before promotion (the eval harness in the architecture diagram).

Drift detection — response and embedding distributions compared against a baseline; statistically significant drift raises a signal even when individual completions look fine.

Regression or drift signals route to alerting/on-call and feed back into the retrain/curation loop.

Fault Tolerance for External API Calls

External dependencies (LLM endpoint, vector store, embedding model, third-party tools) fail independently. Each call site applies a layered strategy; every failure emits external.call.failed for observability.

Per-call resilience

Timeouts — aggressive, tuned per dependency (a slow tool must not stall the whole request). Always bounded; never an unbounded await.

Retries — only on transient/idempotent failures (timeout, 5xx, rate_limit), with exponential backoff + jitter and a hard attempt cap. No retries on 4xx validation errors.

Circuit breaker — per dependency. After a failure-rate threshold the breaker opens, fails fast for a cool-down window, then half-opens to probe recovery. Prevents retry storms and cascading failure.

Bulkheads — separate connection/concurrency pools per dependency so a saturated tool API can't exhaust capacity needed for model calls.

Rate-limit handling — honor Retry-After; token-bucket client-side throttling to stay under provider quotas.

Graceful degradation (fallback ladder)

LLM endpoint — fail over to a secondary model / region / provider; degrade to a smaller/cheaper model under load.

Vector store — on retrieval failure, proceed with reduced context (model answers from parametric knowledge) rather than failing the request; flag lower-confidence in the trace.

Tools — a failed tool returns a structured error observation into the agent loop so the model can adapt or answer without it, instead of aborting.

Guardrails — fail closed (block) on the safety path; never fail open.

Request integrity

Idempotency keys on inbound requests so client retries don't double-charge tokens or duplicate side effects.

Loop bounds — max reasoning iterations and a total wall-clock budget on the agentic loop to prevent runaway cost/latency; exceeding the budget returns the best answer so far or a graceful timeout.

Dead-letter queue — requests that exhaust retries land in a DLQ for inspection/replay rather than being silently dropped.

Design decisions to revisit

LLM-as-judge adds cost/latency — decide sample rate (e.g. 100% offline, 1–5% online).

Multi-provider failover implies prompt portability — keep prompts model-agnostic or maintain per-model variants.

Circuit-breaker thresholds and timeout budgets are workload-specific; start conservative and tune from external.call.failed data.