How can we process the messages in order using Streams API?

We need maintain sequence for messages. We have decided to send all messages from a particular source to a partition, so that sequence of messages will be maintained that means one source will push the messages to a single parttion not multiple parttions and also a partion can have messases from multiple sources and we will be able to identify each source with their key.

Now, we need to consume those messages and do some processing. We do multiple independent operations on the messages what we have consumed like storing them in the database, forwarding them . In this scenario which one we should use Kafka Streams API or Consumer API?

our limitation is we can not have single topic from single resource as this will increase the number of topics .even we can group messages by key to identify the source, for using Streams, we want to order the messages for a key.

Our base requirement is to commit those messages into a database in order and forward those messages in order.

So how can we process the messages in order using Streams API?

Tagged:

Comments

  • Kafka guarantees order within a single partition, by the order they arrived at the broker, not by any other field or by time. All Kafka clients (Consumer, Streams, third-party libraries) respect this fact.

    Generally, though, if you included the message timestamp as part of your database insert event, then it would be possible to group by key, order by timestamp. That will depend on the database, though.

    If you're already partitioning by a known source key, why don't you just filter by that in the streams app? Otherwise, you will have to use Consumer API because it allows you to assign specific partitions (the Processor API, might, but haven't used it).

Sign In or Register to comment.