Go - Decode JSON as it is still streaming through net / http

Question

In the past, I used go to decode JSON from an API endpoint as follows.

client := &http.Client{}

req, err := http.NewRequest("GET", "https://some/api/endpoint", nil)
res, err := client.Do(req)
defer res.Body.Close()

buf, _ := ioutil.ReadAll(res.Body)

// ... Do some error checking etc ...

err = json.Unmarshal(buf, &response)

In the near future I will work on an endpoint that could send me several megabytes of JSON data in the following format.

{
    "somefield": "value",
    "items": [
        { LARGE OBJECT },
        { LARGE OBJECT },
        { LARGE OBJECT },
        { LARGE OBJECT },
        ...
    ]
}

JSON at some point will contain an array of large, arbitrary lengths of objects. I want to take each of these objects and put them separately in the message queue. I do not need to decode the objects themselves.

If I used my usual method, this would have to load the entire response into memory before decrypting it.

Is there a good way to split all the LARGE OBJECT elements as the response still works and sends it to the queue? I do this so as not to store as much data as possible in memory.

Thank!

Answer 1

JSON json.Decoder.

Decoder.Decode() () , . , - "" JSON, JSON, , Decoder.Decode() JSON ( ).

, , , JSON. Decoder.Token(), () JSON . , .

, "" ( ) " ", , JSON, .

, .

JSON:

{
    "somefield": "value",
    "otherfield": "othervalue",
    "items": [
        { "id": "1", "data": "data1" },
        { "id": "2", "data": "data2" },
        { "id": "3", "data": "data3" },
        { "id": "4", "data": "data4" }
    ]
}

items, " ", :

type LargeObject struct {
    Id   string 'json:"id"'
    Data string 'json:"data"'
}

JSON, / .

:

he := func(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

. Go string. HTTP, , json.Decoder:

dec := json.NewDecoder(res.Body)

, :

dec := json.NewDecoder(strings.NewReader(jsonStream))
// We expect an object
t, err := dec.Token()
he(err)
if delim, ok := t.(json.Delim); !ok || delim != '{' {
    log.Fatal("Expected object")
}

// Read props
for dec.More() {
    t, err = dec.Token()
    he(err)
    prop := t.(string)
    if t != "items" {
        var v interface{}
        he(dec.Decode(&v))
        log.Printf("Property '%s' = %v", prop, v)
        continue
    }

    // It the "items". We expect it to be an array
    t, err := dec.Token()
    he(err)
    if delim, ok := t.(json.Delim); !ok || delim != '[' {
        log.Fatal("Expected array")
    }
    // Read items (large objects)
    for dec.More() {
        // Read next item (large object)
        lo := LargeObject{}
        he(dec.Decode(&lo))
        fmt.Printf("Item: %+v\n", lo)
    }
    // Array closing delim
    t, err = dec.Token()
    he(err)
    if delim, ok := t.(json.Delim); !ok || delim != ']' {
        log.Fatal("Expected array closing")
    }
}

// Object closing delim
t, err = dec.Token()
he(err)
if delim, ok := t.(json.Delim); !ok || delim != '}' {
    log.Fatal("Expected object closing")
}

:

2009/11/10 23:00:00 Property 'somefield' = value
2009/11/10 23:00:00 Property 'otherfield' = othervalue
Item: {Id:1 Data:data1}
Item: {Id:2 Data:data2}
Item: {Id:3 Data:data3}
Item: {Id:4 Data:data4}

Go Playground.

Answer 2

, - , mailru/easyjson :

r := bufio.NewReader(stream)
for err == nil {
    pair, _ := r.ReadBytes(',')
    x := jlexer.Lexer{
        Data: pair,
    }
    fmt.Printf("%q = ", x.String())
    x.WantColon()
    fmt.Printf("%d\n", x.Int())
}

, . : https://play.golang.org/p/kk-7aEotqFd