Multicast Enable for Logstash - Elasticsearch

StackOverflow https://stackoverflow.com//questions/24055417

Question

I'm trying to configure logstash to join my elasticsearch cluster as a node using multicast to avoid the configuration of a specific host on logstash configuration.

The configuration I have on elasticsearch is basically:

transport.tcp.port: 9300
http.port: 9200
cluster.name: myclustername
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.multicast.group: 239.193.200.01
discovery.zen.ping.multicast.port: 54328

On logstash side, I have this configuration:

output {
        elasticsearch {
                host => "239.193.200.01"
                cluster => "myclustername"
                protocol => "node"
        }
}

My elasticsearch cluster is being discovered successfully using multicast meaning the multicast IP is working as expected, but from that configuration I get the following log output:

log4j, [2014-06-05T05:51:44.001]  WARN: org.elasticsearch.transport.netty: [logstash-aruba-30825-2014] exception caught on transport layer [[id: 0xe33ea7dd]], closing connection
java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:465)
    at sun.nio.ch.Net.connect(Net.java:457)

If I remove the host key from the configuration I receive this output log:

log4j, [2014-06-05T06:07:45.500]  WARN: org.elasticsearch.discovery: [logstash-aruba-31431-2014] waited for 30s and no initial state was set by the discovery
imeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180)
    at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:615)

What am I doing wrong here? I suppose my logstash configuration is wrong but I'm sure what.

Was it helpful?

Solution

As per Logstash 1.4.1 documentation (http://logstash.net/docs/1.4.1/outputs/elasticsearch) you could create an elasticsearch.yml file in the $PWD dir of the Logstash process to ensure it is configured with the same multicast details.

I assume the Elasticsearch cluster can see each other successfully using multicast and there isn't some network issue preventing that. Check at http://your-es-host:9200/_cluster/health?pretty=true make sure the number of nodes is what you expect.

Setting elasticsearch variable to the jvm using $JAVA_OPTS is another possibility:

export JAVA_OPTS="-Des.discovery.zen.ping.multicast.group=224.2.2.4 \
                  -Des.discovery.zen.ping.multicast.port=54328 \
                  -Des.discovery.zen.ping.multicast.enabled=true"

OTHER TIPS

Other option is use of elasticsearch_http, I've had the same problem, and now working good.

Resource here

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top