Consumer group
How to make multi-worker in same consumer-group all recieve same topic’s message?
In kafka the topic only send one worker in consumer-group per partition(randomly?).
It’s better set the partition number multiple of consumer group’s number
So if you need pulish/subscribe mode, make sure consumer in different group id.
How do I choose the number of partitions for a topic?
The partition count determines the maximum consumer parallelism and so
you should set a partition count based on the maximum consumer parallelism you would expect to need
Partition
- A partition is basically a directory of log files.
- Each partition must fit entirely on one machine.
- Each partition is totally ordered.
- Each partition is not consumed by more than one consumer thread/process in each consumer group.
- Many partitions can be consumed by a single process(randomly?)
- Another way to say the above is that the partition count is a bound on the maximum consumer parallelism.
parallelism mean different group can consume different partitions - More partitions will mean more files and more machine restrict.
- Each partition corresponds to several znodes in zookeeper.
- More partitions means longer leader fail-over time.Synchronize thousands of znodes is not a easy job.
- More partitions the more expensive the position checkpoint is.
- Expand more partitions need manually synchronize data from old partitions
https://www.oschina.net/question/2558468_2145935
https://blog.csdn.net/gezilan/article/details/80412490
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowmanytopicscanIhave?