Filtering / iterating through very large lists in python

Question

If I have a list with 10 million objects, how to quickly filter the list. It takes about 4-5 seconds for a complete iteration through list comprehension. Are there any efficient data structures or libraries for this in python? Or python is not suitable for large datasets?

Answer 1

Itertools is designed for an efficient cycle. In particular, you may find that ifiltermeets your goal. Iterating through large data structures is always expensive, but if you only need some data in time, then a lazy evaluation can help.

, , , ( ) , .

Answer 2

, ( python), Numpy.

Answer 3

.

>>> l=[1]*10000000
>>> s=filter(lambda x:True,l)

, , Numpy generators / itertools