Astyanax Cassandra driver removes host unexpectedly during batch inserting

Question

Astyanax 1.56.37 connecting to Cassandra 1.2.6 running on Debian:

When performing a number of inserts in quick succession to a Cassandra cluster containing only one node located at 10.10.1.141, at seemingly random points, I will see the following in the console:

- AddHost: 127.0.0.1
- RemoveHost: 10.10.1.141

Every attempt to connect to this keyspace after I get this fails with the same message.

Here is my configuration:

AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
        .forCluster("Titan Cluster")
        .forKeyspace(keyspaceName)
        .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
            .setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE)
            .setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE)
            .setTargetCassandraVersion("1.2")
        )
        .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
            .setPort(9160)
            .setMaxConnsPerHost(50)
            .setSeeds("10.10.1.141:9160")
            .setConnectTimeout(2000)
            .setSocketTimeout(30000)
            .setMaxTimeoutWhenExhausted(10000)
            .setMaxTimeoutCount(3)
            .setTimeoutWindow(10000)
            .setLatencyAwareBadnessThreshold(10)
            .setLatencyAwareUpdateInterval(1000)
            .setLatencyAwareResetInterval(10000)
            .setLatencyAwareWindowSize(100)
        )
        .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
        .buildKeyspace(ThriftFamilyFactory.getInstance());

    context.start();

The connection fails on subsequent attempts at context.start()

Answer 1

I too faced the same issue where I had my Cassandra and application (Cassandra client) running on different machines.

AddHost: 10.10.1.141  
AddHost: 127.0.0.1  
RemoveHost: 10.10.1.141  

When I checked my Cassandra ring status, I noticed that the Cassandra was running with the address 127.0.0.1, instead of 10.10.1.141

root@10.10.1.141:/opt/dsc-cassandra$ **bin/nodetool ring**

Address    Rack        Status State   Load               Owns                        Token                                       
127.0.0.1  rack1       Up     Normal  169.87 KB       100.00%             -9217929600007243236                        
127.0.0.1  rack1       Up     Normal  169.87 KB       100.00%             -9140762708880451456                        
127.0.0.1  rack1       Up     Normal  169.87 KB       100.00%             -8952943573583903866                        
127.0.0.1  rack1       Up     Normal  169.87 KB       100.00%             -8891950316930533160* 

In conf/cassandra.yaml, I had specified the hostname instead of IP address for listen_address. The cassandra resoved the hostname to localhost (127.0.0.1) instead of the actual IP (10.10.1.141). After changing the listen_address to the actual IP, the client established connection successfully.

listen_address: 10.10.1.141

Answer 2

I was running Cassandra on VirtualBox on Windows, so the IP was something like 168.192.0.14, and for me, using NodeDiscoveryType.NONE prevented disconnections:

    AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
        .forCluster(clusterName)
        .forKeyspace(keyspaceName)
        .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
            .setDiscoveryType(NodeDiscoveryType.NONE)
        )
        .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
            .setPort(9160)
            .setMaxConnsPerHost(3)
            .setSeeds("192.168.0.14:9160")
        )
        .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
        .buildKeyspace(ThriftFamilyFactory.getInstance());

    context.start();
    Keyspace keyspace = context.getClient();