Update json file

Question

I have a json file with some data and would like to update this file sometimes.

I read the file:

with open('index.json', 'rb') as f: idx = json.load(f) 

then check for a key from potentially new data, and if the key is missing, update the file:

 with open('index.json', mode='a+') as f: json.dump(new_data, f, indent=4) 

However, this procedure simply creates a new json object (python dict) and adds it as a new object to the json output file, making the file an invalid json file.

Is there an easy way to add new data to a json file without overwriting the whole file by updating the original dict?

Answer 1

One way to do what you need is to write one JSON object per line in the file. I use this approach and it works very well.

A good advantage is that you can read the file more efficiently (from memory) because you can read one line at a time. If you need everything, there is no problem assembling the list in Python, but if you do not, you will work much faster, and you can also add.

So, to initially write all of your objects, you would do something like this:

 with open(json_file_path, "w") as json_file: for data in data_iterable: json_file.write("{}\n".format(json.dumps(data))) 

Then, to read efficiently (will consume little memory, regardless of file size):

 with open(json_file_path, "r") as json_file: for line in json_file: data = json.loads(line) process_data(data) 

To update / add:

 with open(json_file_path, "a") as json_file: json_file.write("{}\n".format(json.dumps(new_data))) 

Hope this helps :)