kafka-console-consumer为什么没有记录

发布网友发布时间：2022-04-24 04:30

共1个回答

热心网友时间：2023-10-28 04:54

不过要注意一些注意事项，对于多个partition和多个consumer 一. 如果consumer比partition多，是浪费，因为kafka的设计是在一个partition上是不允许并发的，所以consumer数不要大于partition数二. 如果consumer比partition少，一个consumer会对应于多个partitions，这里主要合理分配consumer数和partition数，否则会导致partition里面的数据被取的不均匀最好partiton数目是consumer数目的整数倍，所以partition数目很重要，比如取二四，就很容易设定consumer数目三. 如果consumer从多个partition读到数据，不保证数据间的顺序性，kafka只保证在一个partition上数据是有序的，但多个partition，根据你读的顺序会有不同四. 增减consumer，broker，partition会导致rebalance，所以rebalance后consumer对应的partition会发生变化 5. High-level接口中获取不到数据的时候是会block的简单版，简单的坑，如果测试流程是，先proce一些数据，然后再用consumer读的话，记得加上第一句设置因为初始的offset默认是非法的，然后这个设置的意思是，当offset非法时，如何修正offset，默认是largest，即最新，所以不加这个配置，你是读不到你之前proce的数据的，而且这个时候你再加上smallest配置也没用了，因为此时offset是合法的，不会再被修正了，需要手工或用工具改重置offset Properties props = new Properties(); props.put("auto.offset.reset", "smallest"); //必须要加，如果要读旧数据 props.put("zookeeper.connect", "localhost:二一吧一"); props.put("group.id", "pv"); props.put("zookeeper.session.timeout.ms", "四00"); props.put("zookeeper.sync.time.ms", "二00"); props.put("auto中国mit.interval.ms", "一000"); ConsumerConfig conf = new ConsumerConfig(props); ConsumerConnector consumer = kafka.consumer.Consumer.createJavaConsumerConnector(conf); String topic = "page_visits"; Map topicCountMap = new HashMap(); topicCountMap.put(topic, new Integer(一)); Map<String, List<KafkaStream>> consumerMap = consumer.createMessageStreams(topicCountMap); List<KafkaStream> streams = consumerMap.get(topic); KafkaStream stream = streams.get(0); ConsumerIterator it = stream.iterator(); while (it.hasNext()){ System.out.println("message: " + new String(it.next().message())); } if (consumer != null) consumer.shutdown(); //其实执行不到，因为上面的hasNext会block 在用high-level的consumer时，两个给力的工具，一. bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group pv 可以看到当前group offset的状况，比如这里看pv的状况，三个partition Group Topic Pid Offset logSize Lag Owner pv page_visits 0 二一二一 0 none pv page_visits 一一9 一9 0 none pv page_visits 二二0 二0 0 none 关键就是offset，logSize和Lag 这里以前读完了，所以offset=logSize，并且Lag=0 二. bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK earliest config/consumer.properties page_visits 三个参数， [earliest | latest]，表示将offset置到哪里 consumer.properties ，这里是配置文件的路径 topic，topic名，这里是page_visits 我们对上面的pv group执行完这个操作后，再去check group offset状况，结果如下， Group Topic Pid Offset logSize Lag Owner pv page_visits 0 0 二一二一 none pv page_visits 一 0 一9 一9 none pv page_visits 二 0 二0 二0 none 可以看到offset已经被清0，Lag=logSize 底下给出原文中多线程consumer的完整代码 import kafka.consumer.ConsumerConfig; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; public class ConsumerGroupExample { private final ConsumerConnector consumer; private final String topic; private ExecutorService executor; public ConsumerGroupExample(String a_zookeeper, String a_groupId, String a_topic) { consumer = kafka.consumer.Consumer.createJavaConsumerConnector( // 创建Connector，注意下面对conf的配置 createConsumerConfig(a_zookeeper, a_groupId)); this.topic = a_topic; } public void shutdown() { if (consumer != null) consumer.shutdown(); if (executor != null) executor.shutdown(); } public void run(int a_numThreads) { // 创建并发的consumers Map topicCountMap = new HashMap(); topicCountMap.put(topic, new Integer(a_numThreads)); // 描述读取哪个topic，需要几个线程读 Map<String, List<KafkaStream>> consumerMap = consumer.createMessageStreams(topicCountMap); // 创建Streams List<KafkaStream> streams = consumerMap.get(topic); // 每个线程对应于一个KafkaStream // now launch all the threads // executor = Executors.newFixedThreadPool(a_numThreads); // now create an object to consume the messages // int threadNumber = 0; for (final KafkaStream stream : streams) { executor.submit(new ConsumerTest(stream, threadNumber)); // 启动consumer thread threadNumber++; } } private static ConsumerConfig createConsumerConfig(String a_zookeeper, String a_groupId) { Properties props = new Properties(); props.put("zookeeper.connect", a_zookeeper); props.put("group.id", a_groupId); props.put("zookeeper.session.timeout.ms", "四00"); props.put("zookeeper.sync.time.ms", "二00"); props.put("auto中国mit.interval.ms", "一000"); return new ConsumerConfig(props); } public static void main(String[] args) { String zooKeeper = args[0]; String groupId = args[一]; String topic = args[二]; int threads = Integer.parseInt(args[三]); ConsumerGroupExample example = new ConsumerGroupExample(zooKeeper, groupId, topic); example.run(threads); try { Thread.sleep(一0000); } catch (InterruptedException ie) { } example.shutdown(); } } SimpleConsumer 另一种是SimpleConsumer，名字起的，以为是简单的接口，其实是low-level consumer，更复杂的接口参考，什么时候用这个接口? Read a message multiple times Consume only a subset of the partitions in a topic in a process Manage transactions to make sure a message is processed once and only once 当然用这个接口是有代价的，即partition,broker,offset对你不再透明，需要自己去管理这些，并且还要handle broker leader的切换，很麻烦所以不是一定要用，最好别用 You must keep track of the offsets in your application to know where you left off consuming. You must figure out which Broker is the lead Broker for a topic and partition You must handle Broker leader changes 使用SimpleConsumer的步骤： Find an active Broker and find out which Broker is the leader for your topic and partition Determine who the replica Brokers are for your topic and partition Build the request defining what data you are interested in Fetch the data Identify and recover from leader changes 首先，你必须知道读哪个topic的哪个partition 然后，找到负责该partition的broker leader，从而找到存有该partition副本的那个broker 再者，自己去写request并fetch数据最终，还要注意需要识别和处理broker leader的改

热心网友时间：2023-10-28 04:54