Specify number of partitions on Kafka producer

Question

I have the following code from the web page https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example

What seems to be missing is how to configure the number of partitions. I want to specify 4 partitions but am always ending up with the default of 2 partitions. How do I change the code to have 4 partitions(without changing the default).

    Properties props = new Properties();
     
    props.put("metadata.broker.list", "localhost:9092,broker2:9092");
    props.put("serializer.class", "kafka.serializer.StringEncoder");
    props.put("partitioner.class", "com.gnip.kafka.SimplePartitioner");
    props.put("request.required.acks", "1");
    props.put("num.partitions", 4);
     
    ProducerConfig config = new ProducerConfig(props);
    Producer<String, String> producer = new Producer<String, String>(config);
    
    Random rnd = new Random();
    for (long nEvents = 0; nEvents < 1000; nEvents++) { 
        long runtime = new Date().getTime();
        String ip = "192.168.2." + rnd.nextInt(255);
        String msg = runtime + ",www.example.com," + ip;
        KeyedMessage<String, String> data = new KeyedMessage<String, String>("page_visits2", ip, msg);
        producer.send(data);
    }

    producer.close();

Answer 1

The Kafka producer api does not allow you to create custom partition, if you try to produce some data to a topic which does not exists it will first create the topic if the auto.create.topics.enable property in the BrokerConfig is set to TRUE and start publishing data on the same but the number of partitions created for this topic will based on the num.partitions parameter defined in the configuration files (by default it is set to one).

Increasing partition count for an existing topic can be done, but it'll not move any existing data into those partitions.

To create a topic with different number of partition you need to create the topic first and the same can be done with the console script that shipped along with the Kafka distribution. The following command will allow you to create a topic with 2 partition (as specified by the --partition flag)

    bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 3 --partition 2 --topic my-custom-topic

Unfortunately as far as my understanding goes currently there is no direct alternative to achieve this.

Answer 2

The number of partitions is a broker property and will not have any effect for the producer, see here. As the producer example page shows, you can use a custom partitioner to route messages as you prefer but new partitions will not be created if not defined in the broker properties.