How to know the right value of log.retention.hours ? according to the disk size or other?
Cluster include 3 kafka machines and each kafka have disk with 5 T.we found that all kafka disks was with 100% size , so kafka disk was full and this is the reason that all kafka brokers was failed
df -h /kafka
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 5T 5T 13M 100% /var/kafka
After investigation we saw that log.retention.hours=10 days
So seems that purging is after 10 days and maybe this is the reason that kafka disks are full with 100% even if they are huge 5T
We want to know how to avoid full used capacity on kafka disks
What we need to set in Kafka config in order to purge the kafka disk according to the disk size \96 is it possible ?
And how to know the right value of log.retention.hours ? according to the disk size or other?