Kafka Native Delay Subsystem for Distributed Message Processing Preventing Data Loss and Solving Data Residency

In modern distributed systems, particularly those leveraging asynchronous message processing, the ability to introduce controlled delays for messages is crucial for various functionalities, including retry mechanisms, scheduled deliveries, and rate throttling. This white paper presents the design and operational principles of a novel delay subsystem built entirely on Apache Kafka. By strategically utilizing a set of Kafka topics that represent discrete delay durations, this design eliminates the need for external schedulers, databases, or additional services, thereby minimizing architectural complexity and coupling. It also solves the problem associated with data residency due to various legal concerns in the banking and fintech industry. The paper details the "denomination" approach to accumulating arbitrary delays, analogous to dispensing currency change. It elucidates the inherent advantages of this Kafka-native approach, such as natural sorting, Kafka's robust write performance, and preventing data loss by retaining data temporarily. Furthermore, it provides a comprehensive walkthrough of the message flow, worker behavior, and the critical role of message headers in maintaining