Error creating Kafka Topics- Replication factor larger than available brokers
Apache KafkaApache Kafka Problem Overview
I am trying to create topics in Kafka by following the guide on Apache Kafka website through command line. While running the command:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
I get the error as follows:
Error while executing topic command replication factor: 1 larger than available
brokers: 0
kafka.admin.AdminOperationException: replication factor: 1 larger than available
brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171)
at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:93)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:55)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
[2015-10-13 14:25:38,607] INFO Processed session termination for sessionid: 0x15
06056566d0004 (org.apache.zookeeper.server.PrepRequestProcessor)
[2015-10-13 14:25:38,608] INFO Closed socket connection for client /127.0.0.1:33
391 which had sessionid 0x1506056566d0004 (org.apache.zookeeper.server.NIOServer
Cnxn)
How can I resolve the issue? Any good documentation to get started with this? Thanks!!
Apache Kafka Solutions
Solution 1 - Apache Kafka
Seems like your server not started , check logs for issue
Kill process
sudo fuser -k 2181/tcp
run zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Run Kafka
bin/kafka-server-start.sh config/server.properties '
Solution 2 - Apache Kafka
If I were you I would try this code:
> bin/kafka-topics.sh --create --zookeeper localhost:2181/kafka > --replication-factor 1 --partitions 1 --topic test
If you are using a cluster where zookeeper be distributed in 3 nodes you should substitue localhost:2181/kafka
for direccion1:2181,direction2:2181,direction3:2181/kafka
Solution 3 - Apache Kafka
For those who are sure that the brokers are up and running, check your server.properties
configuration.
In my case the znodes for kafka is /kafka
as shown below.
zookeeper.connect=zk1.beta:2181,zk2.beta:2181,zk3.beta:2181/kafka
So, try the following command while creating topic:
bin/kafka-topics.sh --create --zookeeper zk1.beta:2181/kafka --replication-factor 2 --partitions 2 --topic test2
Solution 4 - Apache Kafka
It clearly says 0 broker is available . Start a broker by
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
base on the Apache Kafka tutorial then try
Solution 5 - Apache Kafka
I have had the same problem.I solve the problem.I think the reason for this is because there is no background start-up. so,we neet start-up background
bin/kafka-server-start.sh config/server.properties &
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test &
Solution 6 - Apache Kafka
I had the same problem but finally I tried the following steps which worked for me:
1)Create the chroot in the Zookeeper with the following commands:
zkCli.sh -server localhost:2181
create /kafka1 []
2)For the kafka brokers config , modify zookeeper.connect and add chroot path : For eg.
In server.properties :
zookeeper.connect=localhost:2181/kafka1
3)Start Zookeeper
4)Start Kafka Server
- Create a topic as below: bin/kafka-topic.sh --create -zookeeper localhost:2181/kafka1 --replication-factor 1 --partitions 1 --topic data
6)Confirm the created topic: bin/kafka-topics.sh --list --zookeeper localhost:2181/kafka1
I followed below for reference: https://community.hortonworks.com/questions/102132/run-multiple-broker-versions-in-the-same-cluster.html
Solution 7 - Apache Kafka
Your Zookeeper is unable to see any Kafka brokers.Try logging in to your Zookeepr console : bin/zkCli.sh -server localhost:2181
Run [zk: localhost:2181(CONNECTED) 2] ls /brokers/ids
on the console. It must be not be showing up any brokers. You need to restart all your Kafka brokers, and check for the kafka process log. Sometimes, even when the process shows as running (ps -ef | grep kafka
), it may not have really started completely, because of which Zookeeper is unable to see it. I had the same issue with Apache Kafka distribution 0.11.0.
Solution 8 - Apache Kafka
The same scenario I faced after deleting the only topic I had created.
The way to solve it was put Kafka and Zookeeper down, updated Kafka settings (server.properties file) where I add the following setting
delete.topic.enable=true
saved file, turn Zookeeper and Kafka up and everything run as usual.
Solution 9 - Apache Kafka
make sure the kafka server has started properly. If you are using -dameon
parameter to start kafka server as daemon. Try to remove it and see if there are any errors during the startup.
The issue I ran into turned out to be a file access issue, where the user runs kafka doesn't have access to the log directory I configured. After I granted the access, it just works!
Solution 10 - Apache Kafka
The issue is because of logs in data/kafkalogs & data/zookeeperlogs folder.
- Keep these folder as is, just delete the logs inside them.
- Kill the previous processes, start fresh
- Start Zookeeper
- Start Kafka- wait to see if it restarting or throwing any error.
- Create a topic ( if both zookeeper & kafka cli throws no error, this should work)
Solution 11 - Apache Kafka
Try adding in your serverwith:
delete.topic.enable=true
Solution 12 - Apache Kafka
Probably there is leftover, runaway zookeeper on your server from one of the previous deployments. List the pods, containers, processes, etc. and check their age/start time to spot it.
Solution 13 - Apache Kafka
In my case the error was caused by the fact that I started creating topics before the zookeeper became available. Since I did everything in a script, introducing some latency before creating topics helped. If your zookeeper client port is 2181
, you can check the availability of your brokers by:
echo dump | nc zookeeper 2181 | grep brokers
Solution 14 - Apache Kafka
-
Check if all the brokers are registered to the Zookeeper:
zookeeper-shell.sh localhost:2181 ls /brokers/ids
-
Most probably some ID's are missing in the brackets
[]
-
Login to the machine where this ID resides and restart the broker:
kafka-server-stop.sh nohup kafka-server-start.sh ../config/server.properties > /dev/null 2>&1 &
-
Check again if the Zookeeper can now see all your brokers properly.
-
Retry your create topic command.
Hope it helps.
Solution 15 - Apache Kafka
Delete the logs and consumer offsets folder which is getting created in the below directory.
C:\Kafka\kafka_2.12-2.5.0\data\zookeeper
PS: You may loose all the previous topic details.
Solution 16 - Apache Kafka
HDFS/namenode happens to be in safe-mode
>hdfs dfsadmin -safemode get >hdfs dfsadmin -safemode leave
1 - Start zookeeper
>bin/zookeeper-server-start.sh config/zookeeper.properties
You can verify it on other terminal
>telnet localhost 2181
//Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
stat
2 - Start kafka broker/server
>bin/kafka-server-start.sh config/server.properties
Now create Topics >kafka-topics.sh --create --topic t1 --zookeeper localhost:2181 --replication-factor 1 --partitions 1
>kafka-topics.sh --list --zookeeper localhost:2181
Solution 17 - Apache Kafka
Ensure your config/server.properties file has valid (and uncommented) values for the listeners and advertised.listeners properties.
Solution 18 - Apache Kafka
Looks like your kafka server not running well Just make sure this
zookeeper.connect={{your zk ip/localhost}}:2181
at server.properties is correct.
Solution 19 - Apache Kafka
I have had the same problem. You have to follow this part of the guide: http://kafka.apache.org/documentation.html#quickstart_multibroker
I didn't have any created broker. Create 2 new brokers and you will fix the problem. Follow this steps:
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
Once you create that files you have to configure it with this some changes:
In "server-1.properties":
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
host.name=localhost
(Port number must be the port number you have in your server.properties + 1. I had 9092 so I have to write 9093)
In "server-2.properties":
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
host.name=localhost
(Port number must be the next to server-1. In my case 9094)