Why hash is so slow

Question

I wonder why whoosh is a little slower with the following code. In particular, fixing takes quite a lot of time.

I tried to use limitmb = 2048 with a record, not 128 by default, but it practically does not matter. As suggested, I tried procs = 3 for the author, which makes indexing a little faster, but commit is even slower. Also commit (merge = False) does not help here, since the index is empty.

I get the following results:

index_documents 12.41 seconds
commit 22.79 seconds
run 35.34 seconds

What for such a small circuit and about 45,000 objects seems a bit.

I tested with whoosh 2.5.7 and Python 2.7.

Is this normal and am I just expecting too much, or am I doing something wrong?

I also profiled a little, and it seems that he writes, and then reads a lot of pickles. It seems to be related to how transactions are handled.

from contextlib import contextmanager
from whoosh import fields
from whoosh.analysis import NgramWordAnalyzer
from whoosh.index import create_in
import functools
import itertools
import tempfile
import shutil
import time


def timecall(f):
    @functools.wraps(f)
    def wrapper(*args, **kw):
        start = time.time()
        result = f(*args, **kw)
        end = time.time()
        print "%s %.2f seconds" % (f.__name__, end - start)
        return result
    return wrapper


def schema():
    return fields.Schema(
        path=fields.ID(stored=True, unique=True),
        text=fields.TEXT(analyzer=NgramWordAnalyzer(2, 4), stored=False, phrase=False))


@contextmanager
def create_index():
    directory = tempfile.mkdtemp()
    try:
        yield create_in(directory, schema())
    finally:
        shutil.rmtree(directory)


def iter_documents():
    for root in ('egg', 'ham', 'spam'):
        for i in range(1000, 16000):
            yield {
                u"path": u"/%s/%s" % (root, i),
                u"text": u"%s %s" % (root, i)}


@timecall
def index_documents(writer):
    start = time.time()
    counter = itertools.count()
    for doc in iter_documents():
        count = counter.next()
        current = time.time()
        if (current - start) > 1:
            print count
            start = current
        writer.add_document(**doc)


@timecall
def commit(writer):
    writer.commit()


@timecall
def run():
    with create_index() as ix:
        writer = ix.writer()
        index_documents(writer)
        commit(writer)


if __name__ == '__main__':
    run()