Problems with shelf life on Kafka's offset topic

Question

We encountered a problem with Kafka. We have a topic with only a section and only one consumer in a consumer group. The consumer was stopped for a month. In the meantime, manufacturers send messages to the topic.

When we start the consumer again, it cannot consume messages. I assume that the previously recorded bias was lost, so the consumer has no idea when he wakes up.

When we stop and start the consumer again, he can pick up new messages, but all previously sent messages have not been used.

Was the bias damaged? Does kafka's internal offsets storage period mean that the last recorded offsets were deleted?

Answer 1

The retention value can be configured in the kafka broker using:

offsets.retention.minutes 

The default is 24 hours. See: Kafka consumer group reimbursed retention

Answer 2

we also faced the same problem and I studied it a bit. I have the steps below that he decided.

We have a shelf life of 14 days, so we changed the shelf life of the Kafka displacement theme to 14 days.

We changed cleanup.policy from compact to uninstall by running below

 ./kafka-configs.sh --alter --zookeeper localhost:2181 --entity-type topics --entity-name __consumer_offsets --add-config cleanup.policy=delete 

Updated config for topic: "__consumer_offsets".

Answer 3

The default message storage period (offset) in kafka is one week, i.e.) 7 days.

After the expiration of the storage period, all existing offsets will be deleted from the intermediary in order to avoid space overflow.

therefore, the previous offset will be changed to the last offset, as kafka / zookeeper deleted the previous offset.

If you want your message to be long, specify the value - retention-period when creating a topic in kafka.