The cleanest way to remove common list items from multiple lists in python

Question

I have n lists of numbers. I want each list to contain unique elements in this particular list. That is, there are no “common” duplicates for any of the others.
This is very easy to do with two lists, but a bit more complicated with n lists.

eg mylist = [ [1, 2, 3, 4], [2, 5, 6, 7], [4, 2, 8, 9] ] 

becomes:

 mylist = [ [1, 3], [5, 6, 7], [8, 9] ] 

Answer 1

 from collections import Counter from itertools import chain mylist = [ [1,2,3,4], [2,5,6,7,7], [4,2,8,9] ] counts = Counter(chain(*map(set,mylist))) [[i for i in sublist if counts[i]==1] for sublist in mylist] #[[1, 3], [5, 6, 7, 7], [8, 9]] 

Answer 2

This is done in linear time, 2 passes. I assume that you want to keep duplicates in the list; if not, this can be simplified a bit:

 >>> import collections, itertools >>> counts = collections.defaultdict(int) >>> for i in itertools.chain.from_iterable(set(l) for l in mylist): ... counts[i] += 1 ... >>> for l in mylist: ... l[:] = (i for i in l if counts[i] == 1) ... >>> mylist [[1, 3], [5, 6, 7], [8, 9]] 

Answer 3

Since you don’t care about order, you can easily remove duplicates by subtracting and converting to a list. Here he is in a one-line monster:

 >>> mylist = [ ... [1, 2, 3, 4], ... [2, 5, 6, 7], ... [4, 2, 8, 9] ... ] >>> mynewlist = [list(set(thislist) - set(element for sublist in mylist for element in sublist if sublist is not thislist)) for thislist in mylist] >>> mynewlist [[1, 3], [5, 6, 7], [8, 9]] 

Note: This is not very effective because duplicates are recounted for each row. Whether this will be a problem or not depends on your data size.

Answer 4

set () is the right approach. although you do not need to use list comprehension.

No additional import:

 mylist = [ [1, 2, 3, 4], [2, 5, 6, 7], [4, 2, 8, 9] ] >>> result_list = [] >>> for test_list in mylist: ... result_set = set(test_list) ... for compare_list in mylist: ... if test_list != compare_list: ... result_set = result_set - set(compare_list) ... result_list.append(result_set) ... >>> result_list [set([1, 3]), set([5, 6, 7]), set([8, 9])] 

Answer 5

This is my solution, using Counter to build a set of all common numbers, and then it just sets the difference in settings:

 from collections import Counter def disjoin(lsts): c = Counter(num for lst in lsts for num in lst) common = set(x for x,v in c.items() if v > 1) result = [] for lst in lsts: result.append(set(lst) - common) return result 

Example:

 >>> remove_common(mylist) [set([1, 3]), set([5, 6, 7]), set([8, 9])]