Os.walk () caching / acceleration

Question

I have a server prototype [0] that does os.walk()[1] for every request that client [0] makes.

I am currently studying ways:

• caching this data in memory,
• query acceleration and
• We hope to further expand the storage of metadata and data storage.

I find SQL tricky for tree structures, so I thought I would get some tips before committing SQLite

Are there any cross-platform, embeddable, or package related databases other than SQL that can handle this type of data?

• I have a small (10k-100k file) list.
• I have a very small number of connections (maybe 10-20).
• I want to be scalable to handle metadata.

[0] the server and the client are actually the same software, this is a P2P application designed to share files through a local trusted network with the main server using zeroconffor detecting and twisting for almost everything else

[1] The request time is currently 1.2 s os.walk()per 10,000 files

Here is a related function in my Python code that does walking:

def populate(self, string):
    for name, sharedir in self.sharedirs.items():
        for root, dirs, files, in os.walk(sharedir):
            for dir in dirs:
                if fnmatch.fnmatch(dir, string):
                    yield os.path.join(name, *os.path.join(root, dir)[len(sharedir):].split("/"))
            for file in files:
                if fnmatch.fnmatch(file, string): 
                    yield os.path.join(name, *os.path.join(root, ile)[len(sharedir):].split("/"))

Answer 1

- , , ?

, :

i   X    result of os.path.join for X

X, , , ( ), ( ), , os.path.join(name, *os.path.join(root, & c.

, , SQL!

, if fnmatch.fnmatch ( string) , os.path.join cursor.executemany, enumerate ( , ). , populate a:

select result from thetable where X LIKE '%foo%' order by i

string - foo.

Answer 2

, , ( , ). , , . . os.walk(), , .

cache = {}
def os_walk_cache( dir ):
   if dir in cache:
      for x in cache[ dir ]:
         yield x
   else:
      cache[ dir ]    = []
      for x in os.walk( dir ):
         cache[ dir ].append( x )
         yield x
   raise StopIteration()

, cache.

Answer 3

MongoDB? mod_python? mod_python os.walk() Python, script .