Share a dict with multiple Python scripts

Question

I would like a unique database dict(key / value) to be available from several Python scripts running at the same time.

If it script1.pyupdates d[2839], then you script2.pyshould see the changed value at the request d[2839]a few seconds after.

• I was thinking about using SQLite, but it seems that writing / reading from multiple processes at once is not the power of SQLite (say, script1.pyjust changed d[2839]how the script2.pySQLite connection knows should it reload this particular part of the database? )

• I also thought about locking the file when I want to discard the changes (but it’s rather difficult to do ) and use json.dumpfor serialization, then, trying to detect the changes, use json.loadto reboot if there is any modification, etc .... oh no, I reinvent the wheel and reinvent the particularly inefficient key / value database!

• redis looked like a solution, but does not officially support Windows , the same applies to leveldb .

• several scenarios might want to be written exactly at the same time (even if this is a very rare event), is there a way the database system handles this (thanks to the lock parameter? It seems that SQLite cannot do this by default, because that SQLite supports an unlimited number of concurrent readers, but that will only allow one writer at a time. )

What would Pythonic be for this?

Note. I am on Windows, and the dict should have a maximum of 1M elements (the key and value of both integers).

Answer 1

MOS , SQLite, , , SQLite, :

import time
import sqlite3
import os
import random
import sys
import multiprocessing


class Store():

    def __init__(self, filename='kv.db'):
        self.conn = sqlite3.connect(filename, timeout=60)
        self.conn.execute('pragma journal_mode=wal')
        self.conn.execute('create table if not exists "kv" (key integer primary key, value integer) without rowid')
        self.conn.commit()

    def get(self, key):
        item = self.conn.execute('select value from "kv" where key=?', (key,))
        if item:
            return next(item)[0]

    def set(self, key, value):
        self.conn.execute('replace into "kv" (key, value) values (?,?)', (key, value))
        self.conn.commit()


def worker(n):
    d = [random.randint(0, 1<<31) for _ in range(n)]
    s = Store()
    for i in d:
        s.set(i, i)
    random.shuffle(d)
    for i in d:
        s.get(i)


def test(c):
    n = 5000
    start = time.time()
    ps = []
    for _ in range(c):
        p = multiprocessing.Process(target=worker, args=(n,))
        p.start()
        ps.append(p)
    while any(p.is_alive() for p in ps):
        time.sleep(0.01)
    cost = time.time() - start
    print(f'{c:<10d}\t{cost:<7.2f}\t{n/cost:<20.2f}\t{n*c/cost:<14.2f}')


def main():
    print(f'concurrency\ttime(s)\tpre process TPS(r/s)\ttotal TPS(r/s)')
    for c in range(1, 9):
        test(c)


if __name__ == '__main__':
    main()

4- macOS, SSD:

concurrency time(s) pre process TPS(r/s)    total TPS(r/s)
1           0.65    7638.43                 7638.43
2           1.30    3854.69                 7709.38
3           1.83    2729.32                 8187.97
4           2.43    2055.25                 8221.01
5           3.07    1629.35                 8146.74
6           3.87    1290.63                 7743.78
7           4.80    1041.73                 7292.13
8           5.37    931.27                  7450.15

8 2012 , SSD:

concurrency     time(s) pre process TPS(r/s)    total TPS(r/s)
1               4.12    1212.14                 1212.14
2               7.87    634.93                  1269.87
3               14.06   355.56                  1066.69
4               15.84   315.59                  1262.35
5               20.19   247.68                  1238.41
6               24.52   203.96                  1223.73
7               29.94   167.02                  1169.12
8               34.98   142.92                  1143.39

, concurrency, SQLite , macOS, , .

---

SQLite - , TPS, :

class MultiDBStore():

    def __init__(self, buckets=5):
        self.buckets = buckets
        self.conns = []
        for n in range(buckets):
            conn = sqlite3.connect(f'kv_{n}.db', timeout=60)
            conn.execute('pragma journal_mode=wal')
            conn.execute('create table if not exists "kv" (key integer primary key, value integer) without rowid')
            conn.commit()
            self.conns.append(conn)

    def _get_conn(self, key):
        assert isinstance(key, int)
        return self.conns[key % self.buckets]

    def get(self, key):
        item = self._get_conn(key).execute('select value from "kv" where key=?', (key,))
        if item:
            return next(item)[0]

    def set(self, key, value):
        conn = self._get_conn(key)
        conn.execute('replace into "kv" (key, value) values (?,?)', (key, value))
        conn.commit()

mac 20 :

concurrency time(s) pre process TPS(r/s)    total TPS(r/s)
1           2.07    4837.17                 4837.17
2           2.51    3980.58                 7961.17
3           3.28    3047.68                 9143.03
4           4.02    2486.76                 9947.04
5           4.44    2249.94                 11249.71
6           4.76    2101.26                 12607.58
7           5.25    1903.69                 13325.82
8           5.71    1752.46                 14019.70

TPS , .

Answer 2

, redis, Memcached ( Windows).   . https://realpython.com/blog/python/python-memcache-efficient-caching/

Answer 3

2 ,

SQlite

,

BerkeleyDB

Berkeley DB (BDB) - , /

BDB , 256 , 3 , Unix- Windows-, .

, .

redis/memcached/ , sysops IMO, ,

Answer 4

python .

script G, . G script1.py script2.py , python . , , , . , script1.py script2.py . . x .

script1.py

class SCRIPT1:

    def __init__(self, dictionary):
        self.dictionary = dictionary
        self.dictionary.update({"a":"a"})
        print("SCRIPT1 : ", self.dictionary)

    def update(self):
        self.dictionary.update({"c":"c"})          

script2.py

class SCRIPT2:
    def __init__(self, dictionary):
        self.dictionary = dictionary
        self.dictionary.update({"b":"b"})
        print("SCRIPT 2 : " , self.dictionary)

main_script.py

import script1
import script2

x = {}

obj1 = script1.SCRIPT1(x) # output: SCRIPT1 :  {'a': 'a'}
obj2 = script2.SCRIPT2(x) # output: SCRIPT 2 :  {'a': 'a', 'b': 'b'}
obj1.update()
print("SCRIPT 1 dict: ", obj1.dictionary) # output: SCRIPT 1 dict:  {'c': 'c', 'a': 'a', 'b': 'b'}

print("SCRIPT 2 dict: ", obj2.dictionary) # output: SCRIPT 2 dict:  {'c': 'c', 'a': 'a', 'b': 'b'}

_ init _.py , .

:

Redis

Answer 5

. , , , API .

MongoDB Python, . API Python , ( ) , .

Answer 6

/sub websocket-framework, Autobahn/Python script "", , , , Overkill.

Answer 7

, Apache Arrow (pyarrow python).

Apache Arrow - . ​​ , . .

Python NumPy, pandas Python.

Arrow .

, (Arrow), .

Answer 8

CodernintyDB , .

http://labs.codernity.com/codernitydb/

: http://labs.codernity.com/codernitydb/server.html

Answer 9

, .

redis , MongoDB.

https://docs.mongodb.com/manual/tutorial/install-mongodb-on-windows/

MongoDB works great with python and can function like redis. Here are the installation documents for PyMongo: http://api.mongodb.com/python/current/installation.html?_ga=2.78008212.1422709185.1517530606-587126476.1517530605

In addition, many people have raised SQlite. I think you were worried that he allows only one writer at a time, but that is not a problem for you. I think this suggests that if there are two authors, the second will be blocked until the first is completed. This is probably good for your situation.