Cluster Not forming with UDP multicasting

StackOverflow https://stackoverflow.com/questions/19711559

Question

Hi I ran 2 instances of infinispan server 6.0.0.CR1 using default Jgroups-UDP config and i dont see cluster formation. I updated the Default Jgroups UDP config interface to point to machine IP instead of Loop back Address.

 <interface name="management">
            <inet-address value="${jboss.bind.address.management:10.51.11.218}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:10.51.11.218}"/>
        </interface>

Logs from Machine1:

=========================================================================

  JBoss Bootstrap Environment



  JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

=========================================================================

09:53:33,218 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1
09:53:33,501 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA
09:53:33,616 INFO  [org.jboss.as] (MSC service thread 1-2) JBAS015899: JBoss Infinispan Server 6.0.0.CR1 (AS 7.2.0.Final) starting
09:53:35,570 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.7.GA
09:53:35,582 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.7.GA
09:53:35,592 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.14.GA
09:53:35,572 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
09:53:35,749 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.
09:53:35,769 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.
09:53:35,807 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)
09:53:36,014 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem
09:53:36,153 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem
09:53:36,288 INFO  [org.jboss.as.security] (MSC service thread 1-4) JBAS013170: Current PicketBox version=4.0.15.Final
09:53:36,308 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]
09:53:36,367 INFO  [org.jboss.as.naming] (MSC service thread 1-3) JBAS011802: Starting Naming Service
09:53:36,650 WARN  [org.infinispan.configuration.global.TransportConfigurationBuilder] (MSC service thread 1-1) ISPN000254: The transport element's 'strictPeerToPeer' attribute is no longer in use.
09:53:36,970 INFO  [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/10.51.11.215:8080
09:53:37,111 INFO  [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003000: Coyote HTTP/1.1 starting on: http-/10.51.11.215:8080
09:53:37,124 INFO  [org.apache.coyote.ajp] (MSC service thread 1-2) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/10.51.11.215:8089
09:53:37,355 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: HotRodServer starting
09:53:37,356 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) JDGS010000: REST starting
09:53:37,431 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: HotRodServer listening on 10.51.11.215:11222
09:53:37,431 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: WebSocketServer starting
09:53:37,432 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010001: WebSocketServer listening on 10.51.11.215:8181
09:53:37,836 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-2) ISPN000128: Infinispan version: Infinispan 'Infinium' 6.0.0.CR1
09:53:38,067 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-4) JDGS010002: REST mapped to /rest
09:53:38,089 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups Channel
09:53:38,145 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-3) JGRP000015: the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
09:53:38,146 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:53:38,146 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-3) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
09:53:38,147 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-3) JGRP000015: the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
09:53:38,147 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-3) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
09:53:38,151 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:53:38,155 INFO  [stdout] (MSC service thread 1-3)
09:53:38,155 INFO  [stdout] (MSC service thread 1-3) -------------------------------------------------------------------
09:53:38,155 INFO  [stdout] (MSC service thread 1-3) GMS: address=lpv-ecisdev215/clustered, cluster=clustered, physical address=10.51.11.215:55200
09:53:38,156 INFO  [stdout] (MSC service thread 1-3) -------------------------------------------------------------------
09:53:38,156 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started jboss-web-policy cache from security container
09:53:38,157 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started other cache from security container
09:53:38,259 INFO  [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final
09:53:38,279 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364
09:53:43,187 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view: [lpv-ecisdev215/clustered|0] [lpv-ecisdev215/clustered]
09:53:43,319 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Cache local address is lpv-ecisdev215/clustered, physical addresses are [10.51.11.215:55200]
09:53:43,373 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:53:43,390 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:53:43,395 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started memcachedCache cache from clustered container
09:53:43,398 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:53:43,401 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started namedCache cache from clustered container
09:53:43,405 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started default cache from clustered container
09:53:43,425 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-2) JDGS010000: MemcachedServer starting
09:53:43,426 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-2) JDGS010001: MemcachedServer listening on 10.51.11.215:11211
09:53:43,634 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 10.51.11.215:9999
09:53:43,645 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 10.51.11.215:4447
09:53:44,403 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:53:44,433 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started ___hotRodTopologyCache cache from clustered container
09:53:44,781 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://10.51.11.215:9990/management
09:53:44,784 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://10.51.11.215:9990
09:53:44,784 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.CR1 (AS 7.2.0.Final) started in 11981ms - Started 93 of 142 services (49 services are passive or on-demand)
09:53:54,327 WARN  [org.jgroups.protocols.UDP] (TransferQueueBundler,shared=udp) null: no physical address for 71cc339a-27ad-278a-919e-b6c2e17e01db, dropping message
09:54:30,596 WARN  [org.jgroups.protocols.UDP] (TransferQueueBundler,shared=udp) null: no physical address for 71cc339a-27ad-278a-919e-b6c2e17e01db, dropping message

09:55:25,051 WARN  [org.jgroups.protocols.UDP] (TransferQueueBundler,shared=udp) null: no physical address for 71cc339a-27ad-278a-919e-b6c2e17e01db, dropping message

Logs From Machine 2:

./clustered.sh -b 10.51.11.218
=========================================================================

  JBoss Bootstrap Environment



  JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

=========================================================================

09:48:25,167 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1
09:48:25,448 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA
09:48:25,551 INFO  [org.jboss.as] (MSC service thread 1-2) JBAS015899: JBoss Infinispan Server 6.0.0.CR1 (AS 7.2.0.Final) starting
09:48:27,469 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
09:48:27,555 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.
09:48:27,570 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]
09:48:27,591 INFO  [org.xnio] (MSC service thread 1-4) XNIO Version 3.0.7.GA
09:48:27,607 INFO  [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.0.7.GA
09:48:27,621 INFO  [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 3.2.14.GA
09:48:27,665 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.
09:48:27,695 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem
09:48:27,924 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem
09:48:27,972 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013170: Current PicketBox version=4.0.15.Final
09:48:28,224 INFO  [org.apache.coyote.ajp] (MSC service thread 1-4) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/10.51.11.218:8089
09:48:28,302 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)
09:48:28,385 INFO  [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/10.51.11.218:8080
09:48:28,446 INFO  [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003000: Coyote HTTP/1.1 starting on: http-/10.51.11.218:8080
09:48:28,452 INFO  [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final
09:48:28,538 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364
09:48:28,628 INFO  [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service
09:48:28,745 WARN  [org.infinispan.configuration.global.TransportConfigurationBuilder] (MSC service thread 1-2) ISPN000254: The transport element's 'strictPeerToPeer' attribute is no longer in use.
09:48:28,963 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on 10.51.11.218:9999
09:48:28,973 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on 10.51.11.218:4447
09:48:29,339 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: WebSocketServer starting
09:48:29,339 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010001: WebSocketServer listening on 10.51.11.218:8181
09:48:29,355 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: REST starting
09:48:29,563 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan 'Infinium' 6.0.0.CR1
09:48:29,801 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010002: REST mapped to /rest
09:48:29,803 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: HotRodServer starting
09:48:29,803 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: HotRodServer listening on 10.51.11.218:11222
09:48:29,829 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups Channel
09:48:29,893 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 4.19MB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
09:48:29,893 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 4.19MB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
09:48:29,913 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:48:29,924 INFO  [stdout] (MSC service thread 1-1)
09:48:29,925 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------
09:48:29,925 INFO  [stdout] (MSC service thread 1-1) GMS: address=lpv-ecisdev218/clustered, cluster=clustered, physical address=10.51.11.218:55200
09:48:29,925 INFO  [stdout] (MSC service thread 1-1) -------------------------------------------------------------------
09:48:29,920 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:48:29,931 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started other cache from security container
09:48:29,931 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started jboss-web-policy cache from security container
09:48:34,981 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view: [lpv-ecisdev218/clustered|0] [lpv-ecisdev218/clustered]
09:48:35,133 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Cache local address is lpv-ecisdev218/clustered, physical addresses are [10.51.11.218:55200]
09:48:35,214 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:48:35,255 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started namedCache cache from clustered container
09:48:35,262 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:48:35,264 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started memcachedCache cache from clustered container
09:48:35,264 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-2) JDGS010000: MemcachedServer starting
09:48:35,265 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-2) JDGS010001: MemcachedServer listening on 10.51.11.218:11211
09:48:35,269 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:48:35,277 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010281: Started default cache from clustered container
09:48:36,360 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.
09:48:36,391 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started ___hotRodTopologyCache cache from clustered container
09:48:36,580 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://10.51.11.218:9990/management
09:48:36,580 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://10.51.11.218:9990
09:48:36,597 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.CR1 (AS 7.2.0.Final) started in 11856ms - Started 93 of 142 services (49 services are passive or on-demand)

Ifconfig from first machine:

command not found

/sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr 00:50:56:9D:50:89
          inet addr:10.51.11.215  Bcast:10.51.11.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:341548 errors:0 dropped:0 overruns:0 frame:0
          TX packets:474190 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:49619673 (47.3 MiB)  TX bytes:108671222 (103.6 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:562 errors:0 dropped:0 overruns:0 frame:0
          TX packets:562 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:176331 (172.1 KiB)  TX bytes:176331 (172.1 KiB)

Ifconfig from second machine:

eth0      Link encap:Ethernet  HWaddr 00:50:56:9D:71:E3
          inet addr:10.51.11.218  Bcast:10.51.11.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe9d:71e3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6202451 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1164337 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1805709659 (1.6 GiB)  TX bytes:140887855 (134.3 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:45811520 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45811520 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15389551121 (14.3 GiB)  TX bytes:15389551121 (14.3 GiB)
Was it helpful?

Solution

Check firewall settings in these machines, they could be blocking UDP traffic. You can use Wireshark to verify that UDP packets are actually leaving the machine, or enable TRACE logging in org.jgroups package. Also, router/switch could be configured to not allow UDP traffic. Finally, if you can't get it to work, change to TCP as main transport protocol, and use TCPPING for discovery of nodes, passing in the IP addresses of the two nodes forming the cluster. Switching to TCP is easy, just pass in -Djboss.default.jgroups.stack=tcp, but you need to switch MPING configuration for TCPPING In standalone/configuration/clustered.xml:

<protocol type="TCPPING">
  <property name="initial_hosts">10.51.11.215[7600],10.51.11.218[7600]</property>
</protocol>
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top