Multiprocessing bulk insert using peewee

Question

I am working on a simple html scraper in Python 3.4, using peewee as ORM (great btw ORM!). My script takes a bunch of sites, extracts the necessary data and saves them in the database, however, each site is cleaned in a separate process to improve performance, and the stored data must be unique. Data can be duplicated not only between sites, but also on a specific site, so I want to store them only once.

Example: Message and category - many-to-many relationship. During curettage, the same category appears several times in different messages. For the first time I want to save this category in a database (create a new row). If the same category is displayed in different messages, I want to associate this message with an already created line in db.

My question is: do I need to use atomic updates / inserts (insert one record, save, get_or_create categories, save, insert new rows in the many-to-many table, save), or can I use bulk insertion somehow? What is the quick fix to this problem? Maybe some temporary tables are shared between processes that will be filled with mass at the end of the work? I am using MySQL db.

thanks for the answers and your time

Answer 1

You can rely on a database to provide unique constraints by adding unique=Trueto fields or unique multi-column indexes. You can also check documents for get / create and bulk inserts:

• http://docs.peewee-orm.com/en/latest/peewee/models.html#indexes-and-unique-constraints
• http://docs.peewee-orm.com/en/latest/peewee/querying.html#get-or-create
• http://docs.peewee-orm.com/en/latest/peewee/querying.html#bulk-inserts