Python: quick display and search between two lists

Question

I am currently working on a high-performance python 2.7 project that lists tens of thousands of elements. Obviously, each operation should be performed as quickly as possible.

So, I have two lists: one of them is a list of unique arbitrary numbers, let it be called A, and the other is a linear list starting with 1 and with the same length as the first list, with the name B, which represents the indices in (starting from 1)

Something like an enumeration starting at 1.

For example:

A = [500, 300, 400, 200, 100] # The order here is arbitrary, they can be any integers, but every integer can only exist once
B = [  1,   2,   3,   4,   5] # This is fixed, starting from 1, with exactly as many elements as A

If I have an element B (called e_B) and the corresponding element in A is required, I can just do correspond_e_A = A[e_B - 1]. No problems.

But now I have a huge list of random, not unique integers, and I want to know the indices of the integers that are in A, and what are the corresponding elements in B.

I think I have a reasonable solution for the first question:

indices_of_existing = numpy.nonzero(numpy.in1d(random_list, A))[0]

What's good about this approach is that there is no need to match () single operations, numpy in1d just returns a list, for example [True, True, False, True, ...]. Using nonzero (), I can get the indexes of the elements in random_list that exist in A. Ideally, I think.

But for the second question, I'm at a standstill. I tried something like:

corresponding_e_B = map(lambda x: numpy.where(A==x)[0][0] + 1, random_list))

, , (), -, , , , numpy.where() , (, A ), , , .

bisect, , bisect , , , map() ( , ? )

Python, , - ? , , ?

Answer 1

, - numpy.in1d, O(n log n) .

>>> A = [500, 300, 400, 200, 100]
>>> index = { k:i for i,k in enumerate(A, 1) }
>>> random_list = [200, 100, 50]
>>> [i for i,x in enumerate(random_list) if x in index]
[0, 1]
>>> map(index.get, random_list)
[4, 5, None]
>>> filter(None, map(index.get, random_list))
[4, 5]

Python 2, Python 3 map filter , list .

, ( , CPython). , .

, PyPy, Python JIT.

:

import sys
is_pypy = '__pypy__' in sys.builtin_module_names

import timeit
import random
if not is_pypy:
  import numpy
import bisect

n = 10000
m = 10000
q = 100

A = set()
while len(A) < n: A.add(random.randint(0,2*n))
A = list(A)

queries = set()
while len(queries) < m: queries.add(random.randint(0,2*n))
queries = list(queries)

# these two solve question one (find indices of queries that exist in A)
if not is_pypy:
  def fun11():
    for _ in range(q):
      numpy.nonzero(numpy.in1d(queries, A))[0]

def fun12():
  index = set(A)
  for _ in range(q):
    [i for i,x in enumerate(queries) if x in index]

# these three solve question two (find according entries of B)
def fun21():
  index = { k:i for i,k in enumerate(A, 1) }
  for _ in range(q):
    [index[i] for i in queries if i in index]

def fun22():
  index = { k:i for i,k in enumerate(A, 1) }
  for _ in range(q):
    list(filter(None, map(index.get, queries)))

def findit(keys, values, key):
  i = bisect.bisect(keys, key)
  if i == len(keys) or keys[i] != key:
    return None
  return values[i]

def fun23():
  keys, values = zip(*sorted((k,i) for i,k in enumerate(A,1)))
  for _ in range(q):
    list(filter(None, [findit(keys, values, x) for x in queries]))

if not is_pypy:
  # note this does not filter out nonexisting elements
  def fun24():
    I = numpy.argsort(A)
    ss = numpy.searchsorted
    maxi = len(I)
    for _ in range(q):   
      a = ss(A, queries, sorter=I)
      I[a[a<maxi]]

tests = ("fun12", "fun21", "fun22", "fun23")
if not is_pypy: tests = ("fun11",) + tests + ("fun24",)

if is_pypy:
  # warmup
  for f in tests:
    timeit.timeit("%s()" % f, setup = "from __main__ import %s" % f, number=20)

# actual timing
for f in tests:
  print("%s: %.3f" % (f, timeit.timeit("%s()" % f, setup = "from __main__ import %s" % f, number=3)))

:

$ python2 -V
Python 2.7.6
$ python3 -V
Python 3.3.3
$ pypy -V
Python 2.7.3 (87aa9de10f9ca71da9ab4a3d53e0ba176b67d086, Dec 04 2013, 12:50:47)
[PyPy 2.2.1 with GCC 4.8.2]
$ python2 test.py
fun11: 1.016
fun12: 0.349
fun21: 0.302
fun22: 0.276
fun23: 2.432
fun24: 0.897
$ python3 test.py
fun11: 0.973
fun12: 0.382
fun21: 0.423
fun22: 0.341
fun23: 3.650
fun24: 0.894
$ pypy ~/tmp/test.py
fun12: 0.087
fun21: 0.073
fun22: 0.128
fun23: 1.131

n ( A), m ( random_list) q ( ). , comps , fun22 , fun21 ( ~ 10% Python 2 ~ 25% Python 3, 75% PyPy). . , , fun22 Python 2. , ( fun23).

Answer 2

def numpy_optimized(index, values):
    I = np.argsort(values)
    Q = np.searchsorted(values, index, sorter=I)
    return I[Q]

, OP, , , , . , OP ; , pypy, .

, , , :

def numpy_optimized_filtered(index, values):
    I = np.argsort(values)
    Q = np.searchsorted(values, index, sorter=I)
    Z = I[Q]
    return Z[values[Z]==index]