The problem of creating a pig udf circuit

Trying to parse xml and I am having problems returning the UDF tuple. Following the example http://verboselogging.com/2010/03/31/writing-user-defined-functions-for-pig

pigs script

titles = FOREACH programs GENERATE (px.pig.udf.PARSE_KEYWORDS(program))
    AS (root_id:chararray, keyword:chararray);

here is the output circuit code:

 override def outputSchema(input: Schema): Schema = {
    try {
      val s: Schema = new Schema
      s.add(new Schema.FieldSchema("root_id", DataType.CHARARRAY))
      s.add(new Schema.FieldSchema("keyword", DataType.CHARARRAY))
      return s
    }
    catch {
      case e: Exception => {
        return null
      }
    }
  }

I get this error

pig script failed to validate: org.apache.pig.impl.logicalLayer.FrontendException: 
ERROR 0: Given UDF returns an improper Schema. 
Schema should only contain one field of a Tuple, Bag, or a single type. 
Returns: {root_id: chararray,keyword: chararray}

Update final solution:

In java

public Schema outputSchema(Schema input) {
    try {
        Schema tupleSchema = new Schema();
        tupleSchema.add(input.getField(1));
        tupleSchema.add(input.getField(0));
        return new Schema(new Schema.FieldSchema(getSchemaName(this.getClass().getName().toLowerCase(),  input),tupleSchema, DataType.TUPLE));
    } catch (Exception e) {
        return null;
    }
}
+4
source share
1 answer

You need to add your instance variable sto another Schema object.

Try returning new Schema(new FieldSchema(..., input), s, DataType.TUPLE));as in the template below:

Here is my answer in Java (fill in the variable names):

@Override
    public Schema outputSchema(Schema input) {
        Schema tupleSchema = new Schema();
        try {

            tupleSchema.add(new FieldSchema("root_id", DataType.CHARARRAY));
            tupleSchema.add(new FieldSchema("keyword", DataType.CHARARRAY));

            return new Schema(new FieldSchema(getSchemaName(this.getClass().getName().toLowerCase(), input), tupleSchema, DataType.TUPLE));
        } catch (FrontendException e) {
            e.printStackTrace();
            return null;
        }
    }

Would you try:

titles = FOREACH programs GENERATE (px.pig.udf.PARSE_KEYWORDS(program));

If this is not an error, try:

titles = FOREACH TITLES GENERATE
    $0 AS root_id
    ,$1 AS keyword
;

And tell me the mistake?

+5
source

Source: https://habr.com/ru/post/1537024/


All Articles