Can you append data to an existing Avro data file?
-
15-04-2021 - |
Question
It seems like there isn't any way to append data to an existing Avro serialized file. I'd like to have multiple processes writing to a single avro file, but it looks like each time I open it, I start over from scratch. I don't want to read in all the data and then write it back out again.
Using the ruby example code I have tried "ab" and "ab+" as various settings, but no joy.
file = File.open('data.avr', 'wb')
schema = Avro::Schema.parse(SCHEMA)
writer = Avro::IO::DatumWriter.new(schema)
dw = Avro::DataFile::Writer.new(file, writer, schema)
dw << {"username" => "john", "age" => 25, "verified" => true}
dw << {"username" => "ryan", "age" => 23, "verified" => false}
dw.close
Solution
don't pass the schema to append to the file
OTHER TIPS
I did figure out how to do it in Java using the appendTo method:
DatumWriter writer = new ReflectDatumWriter(Record.class);
DataFileWriter file = new DataFileWriter(writer);
file.setMeta("version", 1);
file.setMeta("creator", "ThinkBigAnalytics");
file.setCodec(CodecFactory.deflateCodec(5));
//file.create(schema, new File("/tmp/records"));
file.appendTo(new File("/tmp/records"));
However, I'd love to do it from Ruby.
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow