Production sizing

How many fahmek instances a deployment needs depends on fahmek calls per second at peak, not on the number of users. Detection itself is cheap (about 0.1 ms per chat message); each HTTP call costs more than the regex work it carries. The integration kit makes one call per message of the conversation on every turn, so long conversations multiply the load. Raise the default rate limit (100 inspections per second per instance) before going live, run at least two instances, and confirm the result with scripts/load_test.py on the customer's own hardware.

The demo stack (one machine, docker-compose.demo.yml) is not a production deployment: one LiteLLM process, one fahmek instance, no TLS, and a trace that can be switched on.

1. Estimate the load#

turns per second  = users chatting at peak × messages each sends per minute ÷ 60
fahmek calls/s    = turns per second × calls per turn

Calls per turn depend on how LiteLLM calls fahmek:

With the kit, a conversation of 10 messages cost 14 calls per turn in the load test (10 inspections, 4 restorations), and the calls run one after another: fahmek adds about 5 ms per message to the turn (measured p50 46 ms for 10 messages at low load).

2. Measured capacity#

Measured on 2026-10-16 with scripts/load_test.py, fahmek 0.1.0, policy bank-pilot, in a 4-vCPU sandbox where the load generator shared the CPUs with the services. Read these as conservative orders of magnitude, not as the customer's numbers.

Rule of thumb from these runs: about 150 fahmek calls per second per worker, one worker per core. Past saturation, turns queue (latency grows to seconds), then a few are refused because their 15 ms inspection budget expires: they come back as 408 or BLOCKED, never as a pass-through.

3. Worked example: 10,000 users chatting at once#

Assumptions: 10,000 users in conversation at the same time, each sending a message every 30 seconds, so 333 turns per second. Plan for the measured 150 calls/s per worker with 50% headroom.

fahmek is rarely the limit. At the same rate:

• LiteLLM needed one fully busy process per 15 turns/s in the demo (with the kit and auto-gad running inside it): plan for tens of LiteLLM processes or instances, following LiteLLM's own scaling guidance, and measure with --mode gateway.
• The model provider's rate limits (requests and tokens per minute) and latency (seconds per answer) usually bind first.
• auto-gad: LanguageTool used 1.3 cores at 15 turns/s. At 333 turns/s that is about 30 cores. Either switch auto-gad off in production, or run several auto-gad + LanguageTool pairs behind a load balancer. Under overload auto-gad returns text unchanged (busy) and never blocks.
• Redis holds a few kilobytes per turn for the policy TTL (120 s for bank-pilot): under 100 MB and a few thousand operations per second here, far below one Redis instance's capacity.

4. Settings#

Set these in the .env next to the compose file; docker-compose.services.yml and docker-compose.demo.yml pass them to fahmek. On Kubernetes they are the Helm chart values fahmek.workers, fahmek.inspectRps, fahmek.inspectBurst and fahmek.replicas or fahmek.autoscaling (deploy/helm/fahmek).

The rate limit is per worker and per instance: total admission = FAHMEK_INSPECT_RPS × workers × instances. It protects fahmek from a runaway caller; it is not a capacity setting. Exceeding it returns 429 (BLOCKED on the web-UI path), never a pass-through.

Other points:

• No stickiness. The masked values live in Redis, so any instance can restore what another masked. Round-robin is fine.
• Health checks. Route on /healthz (open). /readyz (with X-Fahmek-Key) also checks that Redis is reachable with persistence off; it needs Redis's CONFIG GET, which some managed Redis services disable.
• One Redis endpoint. fahmek takes a single FAHMEK_REDIS_URL; Redis Sentinel and Cluster are not supported. A restart loses only in-flight mappings: those answers keep their «…» labels. Under a fail_open: false policy, requests are refused while Redis is down.
• Failure mode. With bank-pilot, an unavailable fahmek stops the chat (requests are refused, nothing leaks). Size for peak plus headroom, and keep at least two instances so one can restart.
• Transport. fahmek speaks plain HTTP and the shared key travels in a header: keep LiteLLM, fahmek and Redis on a private network, with TLS terminated by the customer's proxy or service mesh where links cross hosts.
• Metrics. Scrape /metrics on every instance (counters and histograms, no personal data).

5. Run the load test#

scripts/load_test.py sends synthetic conversations at fixed rates and prints, per rate, the turns completed per second, fahmek calls per second, the share of turns that succeeded and the p50/p95/p99 latency. The output holds counts and timings only.

Run it against a test instance configured like production (same FAHMEK_WORKERS, rate limit, Redis), not against production during working hours. On a machine with the offline package, the fahmek image provides Python; no installation is needed:

export FAHMEK_API_KEY=...   # the same key as the instance
docker run --rm --network host -e FAHMEK_API_KEY \
  -v "$PWD/scripts:/scripts:ro" --entrypoint python fahmek:0.1.0 \
  /scripts/load_test.py --mode ui --url http://localhost:8100 \
  --history 10 --stream-rounds 5 --rates 100,200,400 --duration 60 --max-p99-ms 100

Use --mode gateway only with the demo stub model or a test model: with a real model each turn is a paid request, and the model's latency and limits dominate.

Reading the table:

• The capacity is the highest rate where ok/s equals the target, ok share is 1.0 and p99 meets the target. Plan the peak at no more than two thirds of it.
• rate_limited: raise FAHMEK_INSPECT_RPS / FAHMEK_INSPECT_BURST.
• budget_exceeded, or latency in seconds: the CPUs are saturated; add workers or instances.
• license: the instance has no valid license (fix before testing).

6. Go-live checklist#

• Peak turns per second estimated (section 1), with the customer's real conversation length.
• Load test passed at 1.5 × that peak, on the production hardware.
• At least two fahmek instances; FAHMEK_WORKERS, FAHMEK_INSPECT_RPS and FAHMEK_INSPECT_BURST set.
• Redis without persistence; /readyz returns ready on every instance.
• FAHMEK_TRACE=0; LiteLLM prompt storage off.
• Policy bank-pilot (or stricter) and, on the web-UI path, unreachable_fallback: fail_closed.
• License expiry monitored (GET /v1/license, days_left).
• LiteLLM, the model provider's quotas and (if kept) auto-gad sized for the same peak.