kafka 0.72, minimum number of brokers

Question

I'm trying to create a kafka producer that sends messages to kafka brokers (and not to zoo keeper).

I know that the better practice is working with zk, but for the moment I would like to send messages directly to a broker.

To do that, I'm setting the property "broker.list" as described in the documentation. The thing is that it appears that in order for it to work it requires minimum of 3 brokers (else I get an exception).

In the source code of kafka I can see:

if(brokerInfo.size < 3) throw new InvalidConfigException("broker.list has invalid value")

This is weird cause in my data center I hold only 2 kafka nodes (and 3 zk), what can I do in this case? Is there a way go around this?

Answer 1

The brokerInfo is obtained by splitting the individual broker info and NOT the number of brokers .. if you checked the source code more carefully you would see some thing like

// check if each individual broker info is valid => (brokerId: brokerHost: brokerPort)

and then they split this info as below

   brokerInfoList.foreach { bInfo =>
       val brokerInfo = bInfo.split(":")
       if(brokerInfo.size < 3) throw new InvalidConfigException("broker.list has invalid value")
   }

so every single broker expected to have an id with host name and port separated by the : delimiter
basically regarding the number of broker it just do this

    val brokerInfoList = config.brokerList.split(",")
    if(brokerInfoList.size == 0) throw new InvalidConfigException("broker.list is empty")

So you should be fine with that I guess, just try to pass a single broker and it should work. Let us know how it goes

Answer 2

Apparently when writing

props.put("broker.list", "0:" + <host:port>);

It works (I added the "0:" to the original string). I have found it in section 9 of the quick start guide.

I'm not sure I'm getting it, maybe this zero is the partition number(?) maybe something else (could be nice if someone can shed some light here).