Five things every developer should know about building mission-critical systems - Loek Duys

In this NDC Copenhagen talk, Loek Duys shares five essential concepts for building mission-critical systems, based on his experience in banking and utilities. He begins with a simple payment system and progressively introduces patterns to make it resilient. First, he explains the importance of retries for transient errors, but warns about the risk of double bookings, leading to the need for idempotency keys. Next, he introduces the circuit breaker pattern to avoid overwhelming a failing downstream service and provide fast feedback to users. To decouple services, he presents the outbox pattern, which ensures guaranteed delivery of messages by storing them in a database and processing them asynchronously. He then discusses observability, emphasizing the need for logs, traces, and metrics, and demonstrates using OpenTelemetry to trace a payment end-to-end. He addresses challenges of background processing, such as concurrency and ordering, and solutions like leader election and partitioning. Finally, he combines outbox and inbox patterns to handle downstream recovery and ensure exactly-once execution. The talk includes live demos and practical advice for applying these patterns in real-world systems.