Systems that carry real work.
AI automation, embedded Rust, and the reliability layer that keeps both honest when production pushes back.
architecture
Hover a stage, click it for detail.
01 · BACKEND FOUNDATION FastAPI, Pydantic, AsyncIO, Pytest. The service skeleton every later stage plugs into.
02 · LLM API & TOOL CALLING Structured outputs and streaming. Every tool call is schema-checked before it is allowed to run.
03 · AGENT ORCHESTRATION LangGraph state machine. Explicit transitions, so a run can be replayed and compared step by step.
04 · API & BUSINESS INTEGRATIONS REST, webhooks, OAuth 2.0, reconciliation. The layer where the agent meets the systems of record.
05 · RAG ENGINEERING Hybrid search, reciprocal rank fusion, reranking. Retrieval quality before prompt quality.
06 · RELIABILITY & RESILIENCE Circuit breakers and idempotency keys. An upstream failure degrades the run instead of cascading.
07 · SECURITY & AUTHORIZATION Deterministic RBAC and a sandboxed tool runtime. Permissions decided in code, never by the model.
08 · EVALUATION & REGRESSION Trajectory tests over golden datasets. Every prompt or model change is measured before it ships.
09 · SQL & STATE PERSISTENCE Postgres, Redis, checkpointing. Agent state survives a crash, a deploy and a restart.
10 · OBSERVABILITY Tracing and OpenTelemetry. Every run is inspectable from first tool call to final answer.
01 · BACKEND FOUNDATION
My backend is a single async service, and every stage below is a module it loads.
- FastAPI
- Pydantic
- AsyncIO
- Pytest
Key aspects
- Async all the way. I keep blocking calls off the event loop and push CPU work to a thread or a queue.
- Types at the edge. I let Pydantic validate in and out, so a bad payload fails before it reaches a stage.
- One error path. I use custom exceptions and FastAPI handlers instead of try/except in every route.
Pro tips and best practices
- I offload CPU work.
asyncio.to_thread, or Celery and ARQ for parsing and vector maths. - I stream the answer. SSE keeps time-to-first-token low, so a long tool call reads as progress rather than a freeze.
- I configure once.
pydantic-settingsloads.envand validates types at container start.
Pitfalls
- Sync drivers in async code.
psycopg2andrequestsblock the whole thread; I useasyncpgandhttpx. - Unwatched `asyncio.gather`. One failed task leaves the rest running as orphans, so I use
asyncio.TaskGroup.
Production checklist
- Every network client is an
httpx.AsyncClient. - Global handlers for
HTTPExceptionand for internal 500s. -
pytest-asynciofixtures with one database session per test.
02 · LLM API & TOOL CALLING
I only accept a model response I can validate; text I cannot parse is a bug, not an answer.
- Structured Outputs
- Streaming
Key aspects
- Structured output. Every schema comes from a Pydantic model, never from hand-written JSON.
- Pruned context. I count tokens and trim the history, keeping the system prompt and the last message.
- Cost on the meter. Prompt and completion tokens are counted per call and the spend is tracked as it happens.
Pro tips and best practices
- `strict=True` on every tool. The model then cannot emit arguments outside the schema.
- I tag untrusted data. User input and RAG context go inside
<user_context>and<documents>. - I fail over on 5xx. OpenAI hands over to Claude or Gemini instead of surfacing the outage.
Pitfalls
- Trusting argument types. I run tool arguments through Pydantic before the call, because the model returns
"123"where I expected123. - History that only grows. Untrimmed context eventually overruns the window, and the bill with it.
Production checklist
- Every tool schema generated from a Pydantic model.
- Rate-limit handling with exponential backoff.
- Token usage logged per request.
03 · AGENT ORCHESTRATION
My agent is a state graph, not a loop that I hope terminates.
- LangGraph
- State Machine
Key aspects
- State is the source of truth. Reducers update it, and no node mutates it in place.
- Routing is code. Pure functions pick the next node from the state, not from what the model says it intends.
- I checkpoint every step. Each transition is persisted, so a run resumes instead of starting over.
Pro tips and best practices
- I use subgraphs for planning. Router → fetch subgraph → synthesis, instead of free-form wandering.
- I interrupt before I act.
interrupt_beforeon money moves and on outbound email. - I cap the recursion.
recursion_limit=10..20stops a confused agent from burning the budget in a loop.
Pitfalls
- Mutating state in place.
state['key'] = valdesynchronises checkpointing and misbehaves under async. - Unknown tool, no `KeyError`. An invalid tool routes the graph to a self-correction node instead of crashing it.
Production checklist
- A max-steps fuse on every loop.
- Graph state persisted, e.g. with
PostgresSaver. - Human approval on the dangerous actions.
04 · API & BUSINESS INTEGRATIONS
This is where my agent touches systems that charge money and send mail, so the rules here are defensive.
- REST
- Webhooks
- OAuth 2.0
- Reconciliation
Key aspects
- OAuth lifecycle. Tokens are encrypted, validated and refreshed in the background, not under pressure.
- Idempotency everywhere. Every POST, PATCH and DELETE carries an
Idempotency-Key. - Webhooks verified. The HMAC signature is checked before the body is even parsed.
Pro tips and best practices
- I write two phases. Record
PENDING, make the call, then move it toCOMPLETEDonly on success. - I back off with jitter. Randomised delays stop my retries from hitting the provider as a herd.
- I reconcile on a schedule. Compare what I believe with what the system of record says and repair the difference.
Pitfalls
- Retrying without a key. A timeout may have landed already, so a blind retry duplicates the charge.
- Secrets in the database. API keys and refresh tokens are AES-256 (GCM) encrypted at the cell level.
Production checklist
- An
Idempotency-Keyon every mutating call. - Inbound webhooks checked against the provider signature.
- A background job refreshing OAuth tokens before they expire.
05 · RAG ENGINEERING
Retrieval is a pipeline with permissions inside it, not a vector search with a prompt wrapped around it.
- Hybrid Search
- RRF
- Rerank
Key aspects
- Hybrid retrieval. BM25 holds the exact identifier, embeddings hold the paraphrase, and RRF fuses the two.
- The ACL lives in the metadata. Access lists sit on every chunk, so a filter cannot be forgotten on one code path.
- Rerank and ground. A cross-encoder orders the top, and every claim in the answer has to carry a source id.
Pro tips and best practices
- I filter before, not after. The ACL goes into the query; post-filtering burns the context window and can return nothing.
- Small chunks, big context. I index 200-token chunks and hand the model the 1000-token parent section.
- Citations the model has to earn. It cites
[Doc-12], and a post-validator checks the document really said it.
Pitfalls
- Vectors only. Exact part numbers, error codes and proper nouns need lexical search, and vectors lose them.
- Cross-tenant reads. Without index isolation, one user’s query returns another user’s documents.
Production checklist
- An
aclfield on every vector. - A cross-encoder reranking stage.
- RRF merging the BM25 and dense result lists.
06 · RELIABILITY & RESILIENCE
Failures are a given; a run that dies on the first one is not.
- Circuit Breakers
- Idempotency
Key aspects
- Circuit breaker. I open the circuit to a dead provider instead of rediscovering the outage on every run.
- Backoff with jitter. Retries grow slowly and not in lockstep.
- Graceful degradation. A partial answer beats a 500.
Pro tips and best practices
- Timeouts on every hop. Connect and read, and for LLMs a timeout on time-to-first-token.
- I cache semantically. Redis for deterministic repeats, a vector cache for near-identical questions.
- Saga for multi-step. Every step has a compensating action in case the last one fails.
Pitfalls
- Retrying a fatal error. A 401 or a 400 will never succeed; retrying only burns the budget.
- No isolation. One slow external API must not stall every worker I own.
Production checklist
- Timeouts wrapped around every external call.
- A fallback provider configured.
-
tenacityorpybreakerdriving the retries.
07 · SECURITY & AUTHORIZATION
The model proposes and my code decides; nothing runs that the backend has not authorised.
- Deterministic RBAC
- Sandbox
Key aspects
- Authorisation in code. RBAC and token scope are checked in Python, and never asked of the model.
- Injection defence. User text and fetched pages are data, and my system prompt says so.
- Sandboxed tools. Generated code runs in gVisor, Docker sandboxes or WASM.
Pro tips and best practices
- Least privilege keys. Each tool gets a read-only or session-scoped credential and nothing wider.
- I clamp the arguments.
min(limit, 50), so a single call cannot dump the database. - Two guardrail layers. A fast local validator screens the request and the answer before either reaches the client.
Pitfalls
- Rights in the system prompt. "Never share another department's data" is not a control; the model can be talked out of it.
- `eval()` on generated code. Run inside the backend process, that is full server compromise rather than a bug.
Production checklist
- RBAC checked before every external tool call.
- User data isolated in XML tags and stripped of special characters.
- Any dynamic code execution inside a sandbox.
08 · EVALUATION & REGRESSION
Every prompt change is a measurement, and until it is one it is a guess.
- Trajectory Tests
- Golden Datasets
Key aspects
- Golden dataset. Question, expected tool call, context and ideal answer, all versioned.
- Trajectory tests. I assert the path and not just the answer, so a lucky run still fails.
- Automated metrics. Faithfulness, context precision and recall measured without a human in the loop.
Pro tips and best practices
- I evaluate in CI.
promptfooorpyteston every pull request and every prompt change. - The judge gets rubrics. Binary criteria, the reasoning first and the score second.
- I classify the failures. Retrieval, tool choice, hallucination and format, counted separately.
Pitfalls
- Vibe checking. A prompt edit without a golden-dataset run breaks the edge cases, and finds out in production.
- Similarity as truth. Cosine distance passes answers that are factually wrong.
Production checklist
- Fifty to a hundred versioned golden cases.
- CI checking the tool-call trajectory.
- Hit rate and context recall measured.
09 · SQL & STATE PERSISTENCE
A conversation that vanishes when a process restarts is a bug, so state lives in Postgres and only the hot path lives in Redis.
- Postgres
- Redis
- Checkpointing
Key aspects
- Durable checkpoints. Graph snapshots written in ACID transactions, never left in memory.
- pgvector alongside Postgres. Records and their vectors in one store, with HNSW indexes.
- Distributed locks. Redis serialises the concurrent requests of one user.
Pro tips and best practices
- Transactional outbox. The state and the outbound event go in one transaction, and a worker delivers it.
- Proper HNSW settings.
m=16,ef_construction=64, and the right operator class for the distance. - Partitioned history. Old messages archived by date keep the live conversation tables small.
Pitfalls
- Redis as the only store. Restart it and every long conversation is gone.
- Lost updates. Two quick messages read the same state and overwrite each other’s work.
Production checklist
- A checkpoint schema in PostgreSQL.
- HNSW indexes on the vector columns.
- Alembic managing the migrations.
10 · OBSERVABILITY
When a run goes wrong I need to know which request, which node and which call produced it.
- Tracing
- OpenTelemetry
- Alerting
Key aspects
- Distributed tracing. A trace id travels the whole stack, so a long chain can be reconstructed exactly.
- LLM metrics. TTFT, token usage, cost per request, and the p95 and p99 of the model itself.
- Structured logs. JSON carrying the trace id, the user, the tenant and the node.
Pro tips and best practices
- One OpenTelemetry collector. Traces go out to Langfuse, Arize Phoenix, Datadog or Jaeger.
- I log the payloads, and redact. A sanitizer middleware strips personal data and tokens before they are stored.
- Alerts on anomalies. A spike in token spend or a tool error rate over 2% reaches me immediately.
Pitfalls
- No link back to the request. Without it every incident is a guess about what triggered it.
- Logging everything. No rotation and no sampling fills the disk and then the invoice.
Production checklist
- One OTel collector across the whole stack.
- TTFT, token cost and tool error rate charted.
- PII masked in both traces and logs.