Cassandra/Hadoop WITH COMPACT STORAGE option. Why is it needed, is it possible to add it to existing tables/cf

Question

I'm working on a Hadoop / Cassandra integration I have a couple of questions I was hoping someone could help me with.

First, I seem to require the source table/cf to have been created with the option WITH COMPACT STORAGE otherwise I get an error can't read keyspace in map/reduce code.

I was wondering if this is just how it needs to be?

And if this is the case, my second question was, is it possible/how do I add the WITH COMPACT STORAGE option on to a pre-exsting table? .. or am I going to have to re-create them and move data around.

I am using Cassandra 1.2.6

thanks in advance Gerry

Answer 1

I'm assuming you are using job.setInputFormatClass(ColumnFamilyInputFormat.class);

Instead, try using job.setInputFormatClass(CqlPagingInputFormat.class);

The Mapper input for this is Map<String, ByteBuffer>, Map<String,ByteBuffer>

Similarly, if you need to write out to Cassandra us CqlPagingOutputFormat and the appropriate output types.

See http://www.datastax.com/dev/blog/cql3-table-support-in-hadoop-pig-and-hive for more info.

Answer 2

@Gerry

The "WITH COMPACT STORAGE" thing is CQL3 syntax to create tables structure compatible with Thrift clients and legacy column families.

Essentially, when using this option, the table, or should I say column family, is created without using any Composite.

You should know that CQL3 tables heavily rely on composites to work.

Now to answer your questions:

I was wondering if this is just how it needs to be?

Probably because your map/reduce code cannot deal with Composites. But I believe in version 1.2.6 of Cassandra, you have all the necessary code to deal with CQL3 tables. Look at classes in package org.apache.cassandra.hadoop.

is it possible/how do I add the WITH COMPACT STORAGE option on to a pre-exsting table?

No, it's not possible to modify/change table structure once created. You'll need some kind of migrations.