root@amaltsev:~$ cat about.md
I build AI systems that survive production.
What I do
AI Architecture
Forward Deployment Engineering
Area of expertise
Hover a stage, click it for detail.
BACKEND FOUNDATION FastAPI, Pydantic, AsyncIO, Pytest. The service skeleton every later stage plugs into.
LLM API & TOOL CALLING Structured outputs and streaming. Every tool call is schema-checked before it is allowed to run.
AGENT ORCHESTRATION LangGraph state machine. Explicit transitions, so a run can be replayed and compared step by step.
API & BUSINESS INTEGRATIONS REST, webhooks, OAuth 2.0, reconciliation. The layer where the agent meets the systems of record.
RAG ENGINEERING Hybrid search, reciprocal rank fusion, reranking. Retrieval quality before prompt quality.
RELIABILITY & RESILIENCE Circuit breakers and idempotency keys. An upstream failure degrades the run instead of cascading.
SECURITY & AUTHORIZATION Deterministic RBAC and a sandboxed tool runtime. Permissions decided in code, never by the model.
EVALUATION & REGRESSION Trajectory tests over golden datasets. Every prompt or model change is measured before it ships.
SQL & STATE PERSISTENCE Postgres, Redis, checkpointing. Agent state survives a crash, a deploy and a restart.
OBSERVABILITY Tracing and OpenTelemetry. Every run is inspectable from first tool call to final answer.
- FastAPI
- Pydantic
- AsyncIO
- Pytest
Key aspects
- Async all the way. I keep blocking calls off the event loop and push CPU work to a thread or a queue.
- Types at the edge. I let Pydantic validate in and out, so a bad payload fails before it reaches a stage.
- One error path. I use custom exceptions and FastAPI handlers instead of try/except in every route.
Pro tips and best practices
- I offload CPU work.
asyncio.to_thread, or Celery and ARQ for parsing and vector maths. - I stream the answer. SSE keeps time-to-first-token low, so a long tool call reads as progress rather than a freeze.
- I configure once.
pydantic-settingsloads.envand validates types at container start.
Production checklist
- Every network client is an
httpx.AsyncClient. - Global handlers for
HTTPExceptionand for internal 500s. -
pytest-asynciofixtures with one database session per test.
- Structured Outputs
- Streaming
Key aspects
- Structured output. Every schema comes from a Pydantic model, never from hand-written JSON.
- Pruned context. I count tokens and trim the history, keeping the system prompt and the last message.
- Cost on the meter. Prompt and completion tokens are counted per call and the spend is tracked as it happens.
Pro tips and best practices
- `strict=True` on every tool. The model then cannot emit arguments outside the schema.
- I tag untrusted data. User input and RAG context go inside
<user_context>and<documents>. - I fail over on 5xx. OpenAI hands over to Claude or Gemini instead of surfacing the outage.
Production checklist
- Every tool schema generated from a Pydantic model.
- Rate-limit handling with exponential backoff.
- Token usage logged per request.
- LangGraph
- State Machine
Key aspects
- State is the source of truth. Reducers update it, and no node mutates it in place.
- Routing is code. Pure functions pick the next node from the state, not from what the model says it intends.
- I checkpoint every step. Each transition is persisted, so a run resumes instead of starting over.
Pro tips and best practices
- I use subgraphs for planning. Router → fetch subgraph → synthesis, instead of free-form wandering.
- I interrupt before I act.
interrupt_beforeon money moves and on outbound email. - I cap the recursion.
recursion_limit=10..20stops a confused agent from burning the budget in a loop.
Production checklist
- A max-steps fuse on every loop.
- Graph state persisted, e.g. with
PostgresSaver. - Human approval on the dangerous actions.
- REST
- Webhooks
- OAuth 2.0
- Reconciliation
Key aspects
- OAuth lifecycle. Tokens are encrypted, validated and refreshed in the background, not under pressure.
- Idempotency everywhere. Every POST, PATCH and DELETE carries an
Idempotency-Key. - Webhooks verified. The HMAC signature is checked before the body is even parsed.
Pro tips and best practices
- I write two phases. Record
PENDING, make the call, then move it toCOMPLETEDonly on success. - I back off with jitter. Randomized delays stop my retries from hitting the provider as a herd.
- I reconcile on a schedule. Compare what I believe with what the system of record says and repair the difference.
Production checklist
- An
Idempotency-Keyon every mutating call. - Inbound webhooks checked against the provider signature.
- A background job refreshing OAuth tokens before they expire.
- Hybrid Search
- RRF
- Rerank
Key aspects
- Hybrid retrieval. BM25 holds the exact identifier, embeddings hold the paraphrase, and RRF fuses the two.
- The ACL lives in the metadata. Access lists sit on every chunk, so a filter cannot be forgotten on one code path.
- Rerank and ground. A cross-encoder orders the top, and every claim in the answer has to carry a source id.
Pro tips and best practices
- I filter before, not after. The ACL goes into the query; post-filtering burns the context window and can return nothing.
- Small chunks, big context. I index 200-token chunks and hand the model the 1000-token parent section.
- Citations the model has to earn. It cites
[Doc-12], and a post-validator checks the document really said it.
Production checklist
- An
aclfield on every vector. - A cross-encoder reranking stage.
- RRF merging the BM25 and dense result lists.
- Circuit Breakers
- Idempotency
Key aspects
- Circuit breaker. I open the circuit to a dead provider instead of rediscovering the outage on every run.
- Backoff with jitter. Retries grow slowly and not in lockstep.
- Graceful degradation. A partial answer beats a 500.
Pro tips and best practices
- Timeouts on every hop. Connect and read, and for LLMs a timeout on time-to-first-token.
- I cache semantically. Redis for deterministic repeats, a vector cache for near-identical questions.
- Saga for multi-step. Every step has a compensating action in case the last one fails.
Production checklist
- Timeouts wrapped around every external call.
- A fallback provider configured.
-
tenacityorpybreakerdriving the retries.
- Deterministic RBAC
- Sandbox
Key aspects
- Authorization in code. RBAC and token scope are checked in Python, and never asked of the model.
- Injection defense. User text and fetched pages are data, and my system prompt says so.
- Sandboxed tools. Generated code runs in gVisor, Docker sandboxes or WASM.
Pro tips and best practices
- Least privilege keys. Each tool gets a read-only or session-scoped credential and nothing wider.
- I clamp the arguments.
min(limit, 50), so a single call cannot dump the database. - Two guardrail layers. A fast local validator screens the request and the answer before either reaches the client.
Production checklist
- RBAC checked before every external tool call.
- User data isolated in XML tags and stripped of special characters.
- Any dynamic code execution inside a sandbox.
- Trajectory Tests
- Golden Datasets
Key aspects
- Golden dataset. Question, expected tool call, context and ideal answer, all versioned.
- Trajectory tests. I assert the path and not just the answer, so a lucky run still fails.
- Automated metrics. Faithfulness, context precision and recall measured without a human in the loop.
Pro tips and best practices
- I evaluate in CI.
promptfooorpyteston every pull request and every prompt change. - The judge gets rubrics. Binary criteria, the reasoning first and the score second.
- I classify the failures. Retrieval, tool choice, hallucination and format, counted separately.
Production checklist
- Fifty to a hundred versioned golden cases.
- CI checking the tool-call trajectory.
- Hit rate and context recall measured.
- Postgres
- Redis
- Checkpointing
Key aspects
- Durable checkpoints. Graph snapshots written in ACID transactions, never left in memory.
- pgvector alongside Postgres. Records and their vectors in one store, with HNSW indexes.
- Distributed locks. Redis serializes the concurrent requests of one user.
Pro tips and best practices
- Transactional outbox. The state and the outbound event go in one transaction, and a worker delivers it.
- Proper HNSW settings.
m=16,ef_construction=64, and the right operator class for the distance. - Partitioned history. Old messages archived by date keep the live conversation tables small.
Production checklist
- A checkpoint schema in PostgreSQL.
- HNSW indexes on the vector columns.
- Alembic managing the migrations.
- Tracing
- OpenTelemetry
- Alerting
Key aspects
- Distributed tracing. A trace id travels the whole stack, so a long chain can be reconstructed exactly.
- LLM metrics. TTFT, token usage, cost per request, and the p95 and p99 of the model itself.
- Structured logs. JSON carrying the trace id, the user, the tenant and the node.
Pro tips and best practices
- One OpenTelemetry collector. Traces go out to Langfuse, Arize Phoenix, Datadog or Jaeger.
- I log the payloads, and redact. A sanitizer middleware strips personal data and tokens before they are stored.
- Alerts on anomalies. A spike in token spend or a tool error rate over 2% reaches me immediately.
Production checklist
- One OTel collector across the whole stack.
- TTFT, token cost and tool error rate charted.
- PII masked in both traces and logs.