How to parse a JSON string from a column using Pig

Question

I have tsv log files where the column is populated with a json string.

I want to parse this column with JsonLoader in a Pig script. I have seen many examples where JsonLoader is used in cases where each line is only a json line. I have other columns that I want to skip, and I don't know how to do this.

The file is as follows:

foo    bar    {"version":1; "type":"an event"; "count": 1}
foo    bar    {"version":1; "type":"another event"; "count": 1}

How can i do this?

Answer 1

UDF JsonStringToMap, Elephant Bird: https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform

:

foo     bar     {"version":1, "type":"an event", "count": 1}
foo     bar     {"version":1, "type":"another event", "count": 1}

Script:

REGISTER /path/to/elephant-bird.jar;
DEFINE JsonStringToMap com.twitter.elephantbird.pig.piggybank.JsonStringToMap();
raw = LOAD '/tmp/file.tsv' USING PigStorage('\t') AS (col1:chararray,col2:chararray,json_string:chararray);
parsed = FOREACH raw GENERATE col1,col2,JsonStringToMap(json_string);
ILLUSTRATE parsed; -- Just to show the output

(JSON chararray/string):

-------------------------------------------------------------------------------------------------------
| raw     | col1:chararray    | col2:chararray    | json_string:chararray                             | 
-------------------------------------------------------------------------------------------------------
|         | foo               | bar               | {"version":1, "type":"another event", "count": 1} | 

(JSON as map):

---

-------------------------------------------------------------------------------------------------
| parsed     | col1:chararray    | col2:chararray    | json:map(:chararray)                     | 
-------------------------------------------------------------------------------------------------
|            | foo               | bar               | {count=1, type=another event, version=1} | 
-------------------------------------------------------------------------------------------------

, : JSON Pig, ?

Answer 2

Elephantbird ( Twitter Hadoop) - UDF JsonStringToMap, , ( ).