Kafka quickstart in windows

Posted on | Edited on | In

Download

download kafka package from http://kafka.apache.org/downloads

un-tar package kafka_2.11-1.1.0.tgz

Start

copy config file in bin\windows
Start zookeeper first, because kafka is a cluster type middleware.

1
zookeeper-server-start.bat config/zookeeper.properties

If alert Error: can't find or can't load main class ...,
modify bin\windows\kafka-run-class.bat

add Double quotation mark to %CLASSPATH% like this : "%CLASSPATH%"

Now start the Kafka server:

1
kafka-server-start.sh config/server.properties

Create topic

1
2
3
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

kafka-topics.bat --list --zookeeper localhost:2181

Send messages[Producer]

1
2
3
kafka-console-producer.bat --broker-list localhost:9092 --topic test
> Hello world
> Hello topic

Start a consumer

1
2
3
4
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Hello world
Hello topic
_

Multi-broker cluster

copy server.properties

1
2
copy config/server.properties config/server-1.properties
copy config/server.properties config/server-2.properties

modify configuration:
In same machine, node’s id/ port must be unique, log file should be seperated.

1
2
3
4
5
6
7
8
9
config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dir=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dir=/tmp/kafka-logs-2

Now start other two broker

1
2
kafka-server-start.sh config/server-1.properties
kafka-server-start.sh config/server-2.properties

Create new topic

1
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

Show detail

1
kafka-topics.bat --describe --zookeeper localhost:2181 --topic my-replicated-topic

1
2
3
kafka-topics.bat --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic       PartitionCount:1        ReplicationFactor:3 Configs:
        Topic: my-replicated-topic      Partition: 0    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1
  • “leader” is the node responsible for all reads and writes for the given partition.
    Each node will be the leader for a randomly selected portion of the partitions.
  • “replicas” is the list of nodes that replicate the log for this partition
    regardless of whether they are the leader or even if they are currently alive.
  • “isr” is the set of “in-sync” replicas. This is the subset of the replicas list
    that is currently alive and caught-up to the leader.

send some message

1
2
3
4
kafka-console-producer.bat --broker-list localhost:9092 --topic my-replicated-topic
> test message 1
> test message 2
>

consume these messages

1
2
3
4
kafka-console-consumer.bat --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
test message 1
test message 2
_

now the leader broker id is 2, kill it[ ctrl+c ]

1
2
3
4
> wmic process where "caption = 'java.exe' and commandline like '%server-2.properties%'" get processid
ProcessId
6016
> taskkill /pid 6016 /f

1
2
3
kafka-topics.bat --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic       PartitionCount:1        ReplicationFactor:3 Configs:
        Topic: my-replicated-topic      Partition: 0    Leader: 0       Replicas: 2,0,1 Isr: 0,1

now only 0,1 broker ‘in-sync’, the leader is ‘0’ broker
send message still available.

import / export data

make a test file

1
2
> echo foo> test.txt
> echo bar>> test.txt

1
connect-standalone.bat config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
1
2
3
> type test.sink.txt
foo
bar
1
2
3
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}

If add more string append to test.txt, topic will load more data, and test.sink.txt will in-sync too.

http://kafka.apache.org/quickstart
https://blog.csdn.net/cx2932350/article/details/78870135