How to create risk-free avro files using apache avro?

Question

I am using Apache avro to serialize data. Since the data has a fixed schema, I do not want the schema to be part of serialized data. In the following example, the schema is part of the avro file "users.avro".

User user1 = new User(); user1.setName("Alyssa"); user1.setFavoriteNumber(256); User user2 = new User("Ben", 7, "red"); User user3 = User.newBuilder() .setName("Charlie") .setFavoriteColor("blue") .setFavoriteNumber(null) .build(); // Serialize user1 and user2 to disk File file = new File("users.avro"); DatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class); DataFileWriter<User> dataFileWriter = new DataFileWriter<User (userDatumWriter); dataFileWriter.create(user1.getSchema(), new File("users.avro")); dataFileWriter.append(user1); dataFileWriter.append(user2); dataFileWriter.append(user3); dataFileWriter.close(); 

Can anyone tell me how to store avro files without a built-in circuit?

Answer 1

Here you will find a comprehensive description of how I will explain how to achieve serialization without schema without using Apache Avro. A related services test campaign displays some of the performance data you might expect.

The code is on GitHub : an example and test classes show how to use Data Reader and Writer with the Stub class created by Avro itself.

Answer 2

Must be doable.

Given the encoder, you can use DatumWriter to write data directly to ByteArrayOutputStream (which can then be written to java.io.File).

Here's how to get started with Scala (from Salat-Avro ):

 val baos = new ByteArrayOutputStream val encoder = EncoderFactory.get().binaryEncoder(baos, null) encoder.write(myRecord, encoder)