Integrating Kafka Streams, Blockchain, and AI: Solutions for Mission-Critical and Real-Time Microservices

Kafka Streams is a powerful library for building applications and microservices where input and output data are processed through Kafka clusters. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the robustness of Kafka’s server-side cluster technology. Kafka Streams allows developers to focus on “what to do” rather than “how to do it,” promoting a declarative programming style, which differs from traditional Kafka clients.

Dividing software into asynchronous streams reflects the inherently asynchronous nature of modern distributed systems, enabling the resolution of complex problems in a modular way. In this context, microservices operate independently, each responsible for a subset of state transitions, while interacting through event chains.

Event Collaboration Pattern

An example of this is the Event Collaboration Pattern, where the order service has no knowledge of the delivery service. It simply generates an event indicating that it has completed its work (the order was created), and the delivery service decides whether to participate in the interaction. This model, known as receiver-driven routing, decentralizes routing logic, reducing the coupling between services and increasing the system’s flexibility.

In traditional systems, the database is the “source of truth,” which can be suitable for internal application processes. However, when considering interactions between microservices, the shared view is what really matters. In this context, having the event as the source of truth makes more sense, leading us to the CQRS pattern.

Command Query Responsibility Segregation (CQRS)

CQRS (Command Query Responsibility Segregation) separates the write path from the read path and links them to an asynchronous channel.
This idea is not limited to application design — it also appears in several other fields. Databases, for instance, implement the concept of a write-ahead log. Inserts and updates are immediately recorded sequentially on disk as soon as they arrive. This makes them durable, allowing the database to respond to the user, knowing the data is safe, without having to wait for the slower process of updating various data structures, such as tables, indexes, and so on. The point is that if something goes wrong, the internal state of the database can be recovered from the log, and writes and reads can be optimized independently.

When we apply CQRS in our applications, we do so for very similar reasons. Entries are logged in Kafka, which is much faster than writing to a database. Segregating the read model and updating it asynchronously means that the expensive maintenance of data update structures, such as indexes, can be batched to be more efficient. This means that CQRS will have better overall read and write performance when compared to an equivalent, more traditional CRUD-based system.

Join Operations in Kafka Streams

Kafka Streams offers several join options between streams and tables, allowing the combination of datasets.

Stream-Stream Stream-stream joins combine two streams of events into a new stream. The streams are joined based on a common key, so keys are required. You define a time window, and records on both sides of the join must arrive within the defined window. Kafka Streams uses a state store to buffer records, so when a record arrives, it can search the state store of the other stream and find records by key that fit within the time window, based on the timestamp.

Inner Joins If both sides are available during the window, a join is performed. So, if the left side has a record but the right side does not, nothing will be emitted.

Outer Joins With an outer join, both sides always produce an output record. Within the join window, if both the left and right sides are available, a join of the two is returned. If only the left side is available, the join will have the left side’s value and a null for the right side. The reverse is true: if only the right side is available, the join will include the right side’s value and a null for the left side.

Left-Outer Joins Left-outer joins also always produce output records. If both sides are available, the join will consist of both sides. Otherwise, the left side will be returned with a null for the right side.

Stream-Table As you learned above, stream-stream joins are windowed joins. However, the types available for Stream-Table joins are not windowed. You can join a KStream with a KTable and a KStream with a GlobalKTable. In a stream-table join, the stream is always the primary side; it drives the join. Only new records arriving in the stream result in output, while new records arriving in the table do not (this is the case for KStream-KTable and KStream-GlobalKTable joins).

Inner An inner join is triggered only if both sides are available.

Left Outer The KStream always produces a record, either a combination of left and right values, or the left value and a null representing the right side.

GlobalKTable-Stream Join With a GlobalKTable, you get full replication of the underlying topic in all instances instead of sharding. GlobalKTable provides a mechanism whereby, when you perform a join with a KStream, the stream key does not need to match. You get a KeyValueMapper when defining the join and can derive the key to match the GlobalKTable key using the stream key and/or value.

Another difference between a KTable join and a GlobalKTable join is the fact that a KTable uses timestamps. With a GlobalKTable, when there is an update to the underlying topic, the update is automatically applied. It is completely separate from the time mechanism within Kafka Streams. (On the other hand, with a KTable, timestamps are part of the event stream processing.) This means that a GlobalKTable is mainly suitable for static lookup data. For example, you can use it to maintain static user information and compare it with transactions.

Table-Table You can also join one KTable with another KTable. Note that you can only join a GlobalKTable with a KStream.

Stateful Operations

Reduce With reduce, you have a Reducer interface, a single abstract method that takes a value type as a parameter and applies an operation. Reduce expects you to return the same type.

Aggregation Aggregation is a form of reduce, but with aggregation, you can return a different type.

Stateful operations do not emit results immediately. Kafka Streams has an internal buffer mechanism that caches results. Two factors control when the cache emits records: records are emitted when the cache is full (set equally per instance, which is 10 MB), and by default, Kafka Streams commits every 30 seconds (you don’t call the commit yourself). At this point, you will see an update. To see all updates coming through your aggregation, you can set the cache size to zero (which is also useful for debugging).

Even with the cache, you will get multiple results, so for a single, final stateful result, you need to perform an aggregation/reduce operation.

Windowing Windowing allows you to bucket stateful operations by time, without which your aggregations would accumulate infinitely. A window provides a snapshot of an aggregate within a given time period and can be defined as hopping, tumbling, session, or sliding.

Hopping A hopping window is time-limited: you define the window size, but it advances in increments smaller than the window size, so you end up with overlapping windows. You might have a window size of 30 seconds with a 5-second advance size. Data points can belong to more than one window.

Tumbling A tumbling window is a special type of hopping window. It’s a hopping window with an advance size that is the same as the window size. So, you just define a window size of 30 seconds. When 30 seconds ends, you get a new 30-second window. Thus, you don’t get duplicate results as you do with overlapping hopping windows.

Session Session windows are different from the previous two types because they are not defined by time but rather by user activity. So, with session windows, you define an inactivityGap. Essentially, as long as a record arrives within the inactivityGap, your session continues to grow. Theoretically, if you are tracking something with a very active key, your session will keep growing.

Sliding A sliding window is time-limited, but its endpoints are determined by user activity. You create your stream and define a maximum time difference between two records that will allow them to be included in the first window. The window does not advance continuously, like in a hopping window, but advances based on user activity.

Grace Periods Except for session windows, which are behavior-driven, all windows have the concept of grace periods. A grace period is an extension of the window size. Specifically, it allows events with timestamps later than the window end (but earlier than the window end plus the grace period) to be included in the window calculation.

Time Concepts Timestamps are a critical component of Apache Kafka® and similarly drive the behavior of Kafka Streams. You can configure timestamps to follow the event time (the default) or the log-append time.

Timestamps drive action in Kafka Streams The windowing operations you learned in the Windowing module are driven by record timestamps, not wall-clock time. In Kafka Streams, the oldest timestamp in all partitions is chosen first for processing, and Kafka Streams uses the TimeStampExtractor interface to retrieve the timestamp of the current record.

The default behavior is to use the timestamp of a ConsumerRecord, which has a timestamp set by the producer or broker. The default implementation of TimeStampExtractor is FailOnInvalidTimestamp, meaning that if you get a timestamp less than zero, it throws an exception. If you want to use a timestamp embedded in the record key or value itself, you can provide a custom TimeStampExtractor.

Blockchain and Its Applications in Microservices

Blockchain integrates with microservices by providing a secure, immutable, and decentralized record of events. Examples of its application include:

• Order Tracking: Blockchain allows for efficient tracking of orders, with data accessible in real-time.
• Smart Bills of Lading: Eliminating physical documents and using smart contracts reduces costs and increases reliability.
• Payments via Cryptocurrencies: Enables secure and decentralized payments, improving transaction efficiency.

Benefits of Blockchain in Transportation

• Cost Reduction: By eliminating intermediaries and automating processes.
• Security and Efficiency: The decentralized and encrypted nature of blockchain enhances security against fraud and accelerates communication between stakeholders.

Artificial Intelligence (AI) and Automation

AI plays a crucial role in optimizing real-time operations. It can predict delays and potential issues in deliveries based on traffic, weather conditions, and supplier performance. Additionally, AI automates repetitive tasks, such as generating documents and updating systems, reducing errors and increasing efficiency.

In the context of product recommendations and fraud detection, AI analyzes historical data to provide more accurate insights and enhance customer experience.

Final Considerations

The combination of Kafka Streams, Blockchain, and Artificial Intelligence provides a robust foundation for creating mission-critical, highly scalable, and efficient microservices. By integrating these technologies, companies can build innovative solutions to tackle the challenges of real-time data processing and automation of critical tasks.

References:

• Stopford, Ben. Designing Event-Driven Systems, Concepts and Patterns for Streaming Services with Apache Kafka.