Is it possible to get the last available offset for the Kafka section without getting all the messages?

Question

Looking at the latest version (v0.10) of the Kafka Consumer documentation :

"Position The consumer gives the offset of the next record to be issued. It will be greater than the largest offset that the consumer has seen in this section. It automatically promotes every time during which the consumer receives a data call poll (long) and receives messages.

Is there a way to request the largest offset available for a server-side section without to receive all messages?

The logic I'm trying to implement is as follows:

• request every second for the amount (A) of pending messages in the subject
• if A> threshold, start a processor that will search for all messages and process them
• otherwise do nothing (sleep 1)

The motivation is that I need to do batch processing, but I want the processor to wake up only when there is enough data (and I do not want to retrieve the data twice).

Answer 1

Consumer.seekToEnd(), Consumer.poll(0), , , Consumer.position(), ( ) . . , , .

, serejja, , , .

Answer 2

, , 0.10.

, Kafka (, , JVM, ).

, , : FetchResponse ( " " ) HighwaterMarkOffset, (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse). , FetchRequest, (, ) , HighwaterMarkOffset.

FetchRequest :

• MaxWaitTime 0, " , MinBytes".
• MinBytes 0, " , ".
• FetchOffset , , , , , .
• MaxBytes 0, " 0 ", . .

, , . highwatermark, , .

, .

Answer 3

public OffsetAndMetadata committed(TopicPartition partition) API,

https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html