What are the pro's and con's when putting data first in Kafka and then in a database ?

edited December 2018 in Kafka

What are the pro's and con's when putting data first in Kafka and then in a database, or the other way? Example: A user does a REST (POST) call to store products. Normally i would pick up this call in the backend and save the body to a database.Is it a best practice to pick up this call and store the data in Kafka fist and then save it to the database Or is it better to save it in the database first, then send it to kafka?

Tagged:

Comments

  • lets take example of both scenarios with your use case, api call for storing a product lets say object:

    you database: product_table(product_id , product_name,product_info)

    API pseudo code:

    valiadteProductInfo
    save - either first in kafka or in DB
    APPROACH 1 -

    saving to kafka first means that you might see that result in the DB some time later, you will return the objec id to the user and if user wants to populate the object id its not visible. generally this is not the right approach as you will then need to handle many things on the UI side for such delay.

    APPROACH 2 - saving to db first and kafka second there are two scenarios: 1. kafka push is sync in code- in this case in sending to kafka fails , which in your business case if very critical as other microservice is dependent. it is not the right approach , but if its ok that for minute % of the time if push fails and then you delete the product from DB and return exception to the user. I think its completely ok on this.

    kafka push is by polling the db for changes and put changes to kafka (read about EventSourcing for this) : in this case you will get 100% guarantee , but some small delay . this also you can use

Sign In or Register to comment.