A story of race conditions, Docker containers, RabbitMQ queues, and the architecture decisions that kept our risk management platform alive under pressure.
The Problem Nobody Wants on a Trading Floor
Picture this: it’s 9:15 AM. The market just opened. Orders are flooding in — portfolio checks, risk calculations, compliance validations. And your Risk Management system? It’s processing them one at a time. Sequentially. While traders wait.
That was us.
Our platform — built to protect a fund management firm from compliance breaches — was running as a single Flask container. One instance. One CPU thread doing all the work. As the number of daily orders grew, the queue backed up. Trades sat in “Waiting for risk check” status for minutes. Sometimes longer.
We needed to scale. But scaling a system that shares a database, shared files, and shared state isn’t just “run more containers.” Do it wrong and you get something far worse than slowness — corrupted data and silent failures.
This is the story of how we fixed it.
Understanding What We Had
Before we could fix anything, we needed to understand what we were actually dealing with. Our system had more moving parts than it first appeared:
Single Docker Container
├── app.py → Flask on port 5001 (trade poller)
├── RiskManagementProcess.py → Flask on port 1234 (order processor)
├── 20+ Risk Scripts → Internal ports 1001–1054
│ ├── portfoliocheck :1017
│ ├── liquidityexposure :1007
│ ├── exclusionlist :1002
│ └── … 17 more
└── Redis → Embedded (wrong!)
Every trade followed this chain:
External trigger
→ app.py polls DB for “Waiting RM” orders
→ calls newOrder_transfer.py
→ calls RiskManagementProcess.py
→ runs 20+ risk checks sequentially
→ generates Excel report
→ generates flow diagram image
→ updates DB with result
The whole thing was synchronous, blocking, and single-threaded. Beautiful in its simplicity. Terrifying at scale.
The Race Condition Problem — Why “Just Run 5 Containers” Doesn’t Work
Our first instinct was simple: run five copies of the container. Problem solved, right?
Wrong. Here’s what actually happens when five instances all share the same database and file system:
Scenario 1 — The Lost Update
Instance 1: READ portfolio_101 balance = £1,000,000
Instance 2: READ portfolio_101 balance = £1,000,000
Instance 1: WRITE balance = £980,000 (after BUY order)
Instance 2: WRITE balance = £970,000 (after SELL order)
↑ Instance 1’s update is gone forever
Scenario 2 — The Corrupted File
Instance 3: Opens risk_flow_89291.png → starts writing…
Instance 5: Opens risk_flow_89291.png → starts writing simultaneously…
Result: Corrupted PNG. Neither diagram is valid.
Scenario 3 — The Duplicate Processing
Trade 89290 arrives → Instance 1 starts processing
30 seconds later → Instance 2 also picks up Trade 89290
Both run 20 risk scripts on the same order
DB gets double-written. Reports get overwritten.
We needed a smarter approach!
The Architecture We Built
After a lot of discussion, we landed on this design:
External System / User
│
▼
rm_orchestrator (FastAPI, port 5001)
← the only public-facing service
│
│ publishes task
▼
RabbitMQ (Message Broker)
├── trade_queue
└── order_queue
│
│ ONE message → ONE worker
▼
┌─────────────────────────────────────────┐
│ risk_1 risk_2 risk_3 risk_4 risk_5 │
│ Each running: │
│ • Celery worker (task consumer) │
│ • Gunicorn :8000 (app.py) │
│ • Gunicorn :1234 (RiskManagementProcess)│
└─────────────────────────────────────────┘
│
▼
Shared Redis Shared DB Shared Volumes
(locks+results) (MSSQL) (images, CSVs)
The key insight: a message queue guarantees that each trade is picked up by exactly one worker. RabbitMQ does the coordination work that would otherwise require complex locking logic everywhere.
The Debugging Journey
Getting here wasn’t smooth. We hit every possible failure mode along the way.
Problem 1: RabbitMQ connection refused on startup
Workers were starting before RabbitMQ was ready. Sleep timers weren’t reliable. The fix: use Docker’s service_healthy condition:
YAMLdepends_on: rabbitmq: condition: service_healthy
Combined with a better healthcheck that actually verifies the AMQP port is accepting connections, not just that the OS process is running.
Problem 2: All 5 workers processing the same trade
Celery Beat was dispatching trade 89290 every 30 seconds, and all five workers were picking it up. The Redis distributed lock pattern — with blocking=False — meant workers that couldn’t acquire the lock would immediately skip, and only one would proceed.
Problem 3: Images saving to the wrong path
This one was subtle. The diagrams library — which generates our risk flow PNG files — ignores the directory part of the path you give it. It saves relative to the current working directory of the process. Our gunicorn process started in /app/rm_scripts (lowercase), but the volume was mounted at /app/RM_SCRIPTS (uppercase).
The fix: explicitly chdir before generating the diagram:
# Pythonoriginal_cwd = os.getcwd()os.chdir(RISK_FLOW_DIR) # force CWD to the mounted directorytry: with Diagram(f"risk_flow_{order_id}", show=False, ...): # diagram saves here, in the volume-mounted path ...finally: os.chdir(original_cwd) # always restore
The Hardware We Were Running On
One thing worth mentioning: all of this runs on a single Ubuntu VM with 12 cores and 32GB RAM. No Kubernetes, no cloud orchestration, no managed services. Just Docker Compose.
The resource allocation:
12 cores total:├── 5 × Celery workers (--concurrency=4) ≈ 8 cores for processing├── 5 × Gunicorn :8000 (2 workers each) ≈ 2 cores for Flask├── 5 × Gunicorn :1234 (2 workers each) ≈ 2 cores for order processing└── RabbitMQ + Redis minimal overhead
With 5 instances × 4 concurrency = 20 parallel trade processors on 12 physical cores. More than enough for our trading volume.
Key Takeaways
After weeks of debugging, here’s what we learned:
1. A message queue solves race conditions architecturally RabbitMQ’s guarantee that each message is consumed exactly once eliminates whole classes of race condition before you even write locking code.
2. Never embed infrastructure in your app container Redis was embedded in our original Flask container. Extracting it to a standalone container was the first and most important step. Shared state needs to live outside your application instances.
3. Relative paths are dangerous in multi-process environments ./risk_flow_images/ resolves differently depending on which process calls it and what its working directory is. Always use absolute paths, or explicitly control CWD.
4. service_healthy beats sleep N Sleep timers are guesses. Docker’s healthcheck conditions are guarantees. Always wait for actual readiness, not arbitrary time.
5. Distributed locks need a finally block If your lock isn’t released in a finally block, a crash mid-processing will leave it locked until TTL expiry. Every other worker will skip that trade for the next N minutes.
6. Debug with real data, not assumptions Half our debugging time was saved by running targeted diagnostic commands directly against running containers. docker exec risk_1 python -c "..." is your best friend.
The Full Stack
For reference, here’s everything that made this work:
- Orchestrator: FastAPI + Uvicorn
- Task queue: Celery 5.3.6 + RabbitMQ 3.13
- Result backend: Redis 7
- App servers: Flask + Gunicorn (5 instances)
- Scheduler: Celery Beat (auto-detection of new orders)
- Database: Microsoft SQL Server (pymssql)
- Infrastructure: Docker Compose on Ubuntu 22.04
- Monitoring: RabbitMQ Management UI + Celery Flower
The entire system — orchestrator, 5 app instances, Redis, RabbitMQ, and Celery Beat — runs as 9 containers on a single VM, coordinated by a shared external Docker network.
What’s Next
The system is stable and processing trades reliably in parallel. But there’s always more to do:
- Flower dashboard for real-time worker monitoring
- Dead letter queues in RabbitMQ for failed task recovery
- Prometheus + Grafana for metrics
- Automatic scaling based on queue depth
But those are problems for another day. For now, trades are being processed in parallel, risk checks are running correctly, diagrams are generating to the right place, and the traders aren’t waiting.
That’s a good day.
If you’re building something similar — a multi-instance Python application with shared state — feel free to reach out. The patterns here (queue-based coordination, distributed locking, careful path management) apply well beyond trading systems.
