Problem removing specific nested JSON objects in python

Question

I am trying to iterate over a list of nested JSON objects (returned from twitter API via tweepy.api.search) and delete specific objects. I have a list of objects that need to be saved. I want to indicate which dictionary objects to save, not delete, because different tweets have different keys. All of them have keys such as text, created_at, etc., but there are other keys that only have certain tweets.

I ran into two problems.

1) I cannot delete a dictionary item during iteration through a dictionary

2) Many of the dictionary objects contain nested lists and dictionaries with which I have problems accessing

A small part of the JSON file through which I execute:

{ "statuses": [ { "contributors": null, "coordinates": null, "created_at": "Thu Nov 12 01:28:07 +0000 2015", "entities": { "hashtags": [], "symbols": [], "urls": [ { "display_url": "twitter.com/thehill/status\u2026", "expanded_url": "https://twitter.com/thehill/status/664581138975989761", "indices": [ 139, 140 ], "url": "https://t.co/9zfkg2FixZ" } ], "user_mentions": [ { "id": 2517854953, "id_str": "2517854953", "indices": [ 3, 19 ], "name": "It'sAlwaysPolitical", "screen_name": "politicspodcast" } ] }, "favorite_count": 0, "favorited": false, "geo": null } ] } 

Each item in the status list is one tweet, and 100 tweets are returned for each call.

List of items I want to keep:

 keepers_list = [tweetlist["statuses"][i]["coordinates"], tweetlist["statuses"][i]["created_at"], tweetlist["statuses"][i]["entities"]["urls"] ] 

I am trying to do:

 for item in tweetlist: if item not in keepers_list: del item 

I tried this exact code and other variations of its / different methods than I can remember, but can't make it work. I looked through numerous table exchange posts on this topic, but could not adapt them to my purpose.

I tried using

 for key in dict.iterkeys(): ... for value in dict.itervalues(): ... for key, value in dict.iteritems(): 

but I cannot get any of them to work for what I want to do.

Any help or just a push in the right direction is welcome.

Answer 1

Never delete items in a list, iterate over it, you can

Make a copy of the list to iterate over:

 for item in tweetlist[:]: ... 

Save the desired results in a different list:

 keep = [] for item in tweetlist: if item in keepers_list: keep.append(item) 

Answer 2

My general rule in Python is, if I'm in a loop, to look for a different approach. In this case, to use a dictionary understanding based on the original entry:

 keep = {key:tweet_list[key] for key in tweet_list.keys() if key in keepers_list} 

If the original data set is so large that it needs to be processed in place, the understanding is usually quick and, if relatively short, self-documenting enough to be easy to understand.

Answer 3

If you want to filter out a dict, you can do something like:

 for k in dict.keys(): if k not in keepers_list: del(dict[k]) print dict