Export JSON from Spark and enter R

Question

I wrote some data from Spark to a JSON file, and I'm trying to import it into R.

I cannot import it using any traditional JSON packages in R:

library("jsonlite")
bids <- fromJSON("win_rate_sample.json")

I get the following error:

Error in feed_push_parser(readBin(con, raw(), n), reset = TRUE) : 
 parse error: trailing garbage 
      X","domain":"ifunny_premium"}{"win":0,"bid_price":0.75,"size
                 (right here) ------^

How can I get this file in R?

Answer 1

It turns out that Spark exports stream json files similar to those discussed in the following question: Error parsing a JSON file using the jsonlite package

The solution is to use the jsonlite streaming function:

library(jsonlite)
json_file <- stream_in(file("win_rate_sample.json"))