All geek questions in one place

Python - finding duplicates in a dictionary list and grouping them

I am not a programmer, and also new to python, I have a list of dicts coming from a json file:

# JSON file (film.json) [{"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["19,00"]}, {"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fiction"], "price": ["20,00"]}, {"year": ["2003"], "director": ["Tarantino"], "film": ["Kill Bill vol.1"], "price": ["10,00"]}, {"year": ["2003"], "director": ["Wachowski"], "film": ["The Matrix Reloaded"], "price": ["9,99"]}, {"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fyction"], "price": ["15,00"]}, {"year": ["1994"], "director": ["E. de Souza"], "film": ["Street Fighter"], "price": ["2,00"]}, {"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["20,00"]}, {"year": ["1982"], "director": ["Ridley Scott"], "film": ["Blade Runner"], "price": ["19,99"]}]

I can import json file with:

 import json json_file = open('film.json') f = json.load(json_file)

but after that I can’t find occurrences in f and divide them into groups according to the name of the movie. This is what I am looking to achieve:

 ## result grouped by 'film' #group 1 {"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["19,00"]} {"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["20,00"]} #group 2 {"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fiction"], "price": ["20,00"]} {"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fyction"], "price": ["15,00"]} #group X ...

Or better:

 new_dict = { 'group1':[[],[],...] , 'group2':[[],[],...] , 'groupX':[...] }

I am currently testing with a nested for , but no luck.

Thanks.

note: "pulp fyction" is the desired error for a future implementation with fuzzy string matching, at the moment I only need a "duplicate grouper"

note2: with python 2.x

+6
source share
2 answers

Since your data is not sorted, use the collections.defaultdict() object to materialize the list for new keys, then click on the movie name:

 from collections import defaultdict grouped = defaultdict(list) for film in f: grouped[film['film'][0]].append(film)

The value of film['film'][0] used to group films. You will need to create a canonical version of this key if you want to use a more complex grouping of names.

Demo:

 >>> from collections import defaultdict >>> import json >>> with open('film.json') as film_file: ... f = json.load(film_file) ... >>> grouped = defaultdict(list) >>> for film in f: ... grouped[film['film'][0]].append(film) ... >>> grouped defaultdict(<type 'list'>, {u'Street Fighter': [{u'director': [u'E. de Souza'], u'price': [u'2,00'], u'film': [u'Street Fighter'], u'year': [u'1994']}], u'Pulp Fiction': [{u'director': [u'Tarantino'], u'price': [u'20,00'], u'film': [u'Pulp Fiction'], u'year': [u'1994']}], u'Pulp Fyction': [{u'director': [u'Tarantino'], u'price': [u'15,00'], u'film': [u'Pulp Fyction'], u'year': [u'1994']}], u'The Matrix': [{u'director': [u'Wachowski'], u'price': [u'19,00'], u'film': [u'The Matrix'], u'year': [u'1999']}, {u'director': [u'Wachowski'], u'price': [u'20,00'], u'film': [u'The Matrix'], u'year': [u'1999']}], u'Blade Runner': [{u'director': [u'Ridley Scott'], u'price': [u'19,99'], u'film': [u'Blade Runner'], u'year': [u'1982']}], u'Kill Bill vol.1': [{u'director': [u'Tarantino'], u'price': [u'10,00'], u'film': [u'Kill Bill vol.1'], u'year': [u'2003']}], u'The Matrix Reloaded': [{u'director': [u'Wachowski'], u'price': [u'9,99'], u'film': [u'The Matrix Reloaded'], u'year': [u'2003']}]}) >>> from pprint import pprint >>> pprint(dict(grouped)) {u'Blade Runner': [{u'director': [u'Ridley Scott'], u'film': [u'Blade Runner'], u'price': [u'19,99'], u'year': [u'1982']}], u'Kill Bill vol.1': [{u'director': [u'Tarantino'], u'film': [u'Kill Bill vol.1'], u'price': [u'10,00'], u'year': [u'2003']}], u'Pulp Fiction': [{u'director': [u'Tarantino'], u'film': [u'Pulp Fiction'], u'price': [u'20,00'], u'year': [u'1994']}], u'Pulp Fyction': [{u'director': [u'Tarantino'], u'film': [u'Pulp Fyction'], u'price': [u'15,00'], u'year': [u'1994']}], u'Street Fighter': [{u'director': [u'E. de Souza'], u'film': [u'Street Fighter'], u'price': [u'2,00'], u'year': [u'1994']}], u'The Matrix': [{u'director': [u'Wachowski'], u'film': [u'The Matrix'], u'price': [u'19,00'], u'year': [u'1999']}, {u'director': [u'Wachowski'], u'film': [u'The Matrix'], u'price': [u'20,00'], u'year': [u'1999']}], u'The Matrix Reloaded': [{u'director': [u'Wachowski'], u'film': [u'The Matrix Reloaded'], u'price': [u'9,99'], u'year': [u'2003']}]}

Using SoundEx to group movies will be as simple as:

 from itertools import groupby, islice, ifilter _codes = ('bfpv', 'cgjkqsxz', 'dt', 'l', 'mn', 'r') _sounds = {c: str(i) for i, code in enumerate(_codes, 1) for c in code} _sounds.update(dict.fromkeys('aeiouy')) def soundex(word, _sounds=_sounds): grouped = groupby(_sounds[c] for c in word.lower() if c in _sounds) if _sounds.get(word[0].lower()): next(grouped) # remove first group. sdx = ''.join([k for k, g in islice((g for g in grouped if g[0]), 3)]) return word[0].upper() + format(sdx, '<03') grouped_by_soundex = defaultdict(list) for film in f: grouped_by_soundex[soundex(film['film'][0])].append(film)

as a result of:

 >>> pprint(dict(grouped_by_soundex)) {u'B436': [{u'director': [u'Ridley Scott'], u'film': [u'Blade Runner'], u'price': [u'19,99'], u'year': [u'1982']}], u'K414': [{u'director': [u'Tarantino'], u'film': [u'Kill Bill vol.1'], u'price': [u'10,00'], u'year': [u'2003']}], u'P412': [{u'director': [u'Tarantino'], u'film': [u'Pulp Fiction'], u'price': [u'20,00'], u'year': [u'1994']}, {u'director': [u'Tarantino'], u'film': [u'Pulp Fyction'], u'price': [u'15,00'], u'year': [u'1994']}], u'S363': [{u'director': [u'E. de Souza'], u'film': [u'Street Fighter'], u'price': [u'2,00'], u'year': [u'1994']}], u'T536': [{u'director': [u'Wachowski'], u'film': [u'The Matrix'], u'price': [u'19,00'], u'year': [u'1999']}, {u'director': [u'Wachowski'], u'film': [u'The Matrix Reloaded'], u'price': [u'9,99'], u'year': [u'2003']}, {u'director': [u'Wachowski'], u'film': [u'The Matrix'], u'price': [u'20,00'], u'year': [u'1999']}]}
+8
source

If it was one, and I was in a hurry, I would do it like that. Assuming in this example that your dictionary list is accessible, and that the movie title will only ever be a one-item list

 new_dict = {k:[d for d in lod if d.get('film')[0] == k] for k in set(d.get('film')[0] for d in l)}

To make it more readable and explain what it is doing, the same thing broke, again a list of dictionaries: lod:

 #get all the unique film names # note: the [0] is because its a list for the title, and set doesn't work with lists, #so we're just taking the first one for this example. films = set(d.get('film')[0] for d in lod) #create a dictionary new_dict = {} #iterate over the unique film names for k in films: #make a list of all the films that match the name we're on filmswiththisname = [d for d in lod if d.get('film')[0] == k] #add the list of films to the new dictionary with the film name as the key. new_dict[k] = filmswiththisname
0
source

Source: https://habr.com/ru/post/954618/

More articles:


All Articles