How do you upgrade an existing avro scheme using apache avro SchemaBuilder?

Question

I am testing a new schema registry that downloads and retrieves various avro schemas. In the testing process, I need to create a bunch of different types of avro circuits. Since this involves a lot of permutations, I decided to create the circuit programmatically. For this I use apache avro SchemaBuilder .

I created avro using:

Schema oldSchema = SchemaBuilder .record("abc") .aliases("records") .fields() .name("field_null") .type("null") .noDefault() .endRecord(); 

It worked. The created avro looks like this:

 { "type" : "record", "name" : "abc", "fields" : [ { "name" : "field_null", "type" : "null" } ], "aliases" : [ "records" ] } 

Now I want to create a new version of the schema using the apro apache libraries, for example:

 { "type" : "record", "name" : "abc", "fields" : [ { "name" : "field_null", "type" : "null" }, { "name" : "new_field", "type" : "int", "default" : 10 } ], "aliases" : [ "records" ] } 

For this, I tried:

 Schema.Field field = new Schema.Field("new_field", SchemaBuilder.builder().intType(), "NewField", 10); List<Schema.Field> fields = new ArrayList<>(); fields.add(field); fields.addAll(oldSchema.getFields()); Schema record = Schema.createRecord(oldSchema.getName(), "Changes", oldSchema.getNamespace(), false, fields); 

I get:

 org.apache.avro.AvroRuntimeException: Field already used: field_null type:NULL pos:0 at org.apache.avro.Schema$RecordSchema.setFields(Schema.java:647) at org.apache.avro.Schema$RecordSchema.<init>(Schema.java:618) at org.apache.avro.Schema.createRecord(Schema.java:167) 

My problem:

• How to add new versions of a schema using existing libraries?
• Should I use avro schemaBuilder to create the schema or rather create my own POJOs to create the schema / save the avsc files in the data directory.

Answer 1

You can try this to create fields, maybe this is inconvenient:

 Schema.Field field = new Schema.Field("new_field",SchemaBuilder.builder().intType(), "NewField", 10); List<Schema.Field> fields = new ArrayList<>(); for (Schema.Field f : oldSchema.getFields()) { Schema.Field _field = new Schema.Field(f.name(), f.schema(), f.doc(), f.defaultValue()); fields.add(_field); } 

Answer 2

To copy fields from the old scheme to the new, you need to make a deep copy of each field as suggested by @xiping xing.

This is because the Schema class checks that the field is added only once to the scheme, and in your case, these fields have already been added to the old scheme.

You can see how they use the flag in this snippet from Avro 1.7.7 :

 @Override public void setFields(List<Field> fields) { if (this.fields != null) { throw new AvroRuntimeException("Fields are already set"); } int i = 0; fieldMap = new HashMap<String, Field>(); LockableArrayList ff = new LockableArrayList(); for (Field f : fields) { if (f.position != -1) throw new AvroRuntimeException("Field already used: " + f); f.position = i++; final Field existingField = fieldMap.put(f.name(), f); if (existingField != null) { throw new AvroRuntimeException(String.format( "Duplicate field %s in record %s: %s and %s.", f.name(), name, f, existingField)); } ff.add(f); } this.fields = ff.lock(); this.hashCode = NO_HASHCODE; }