Insert database with error without error

Question

I work with scrapy and dataset ( https://dataset.readthedocs.io/en/latest/quickstart.html#storing-data ), which is a layer on top of sqlalchemy, trying to load data into a sqllite table as a continuation of Sqlalchemy: dynamically create a table from point Scrapy .

using the dataset package I have:

class DynamicSQLlitePipeline(object): def __init__(self,table_name): db_path = "sqlite:///"+settings.SETTINGS_PATH+"\\data.db" db = dataset.connect(db_path) self.table = db[table_name].table def process_item(self, item, spider): try: print('TEST DATASET..') self.table.insert(dict(name='John Doe', age=46, country='China')) print('INSERTED') except IntegrityError: print('THIS IS A DUP') return item 

after starting my spider, I see that the printed statements are printed in the try except block, without errors, but after completion I look at the table and see a screenshot. There is no data in the table. What am I doing wrong?

Answer 1

The code you posted does not work, as for me:

 TypeError: __init__() takes exactly 2 arguments (1 given) 

This is because the __init__ method expects an argument to table_name that is not passed. You need to implement the from_crawler class from_crawler in the pipeline object, for example:

 @classmethod def from_crawler(cls, crawler): return cls(table_name=crawler.spider.name) 

This will create a pipeline object using the name spider as the name of the table, you can, of course, use whatever name you want.

In addition, the line self.table = db[table_name].table should be replaced by self.table = db[table_name] ( https://dataset.readthedocs.io/en/latest/quickstart.html#storing-data )

After that, the data is saved:

Answer 2

Perhaps some problems with connecting Db. Put your this snippet in an attempt, except to check the problem.

 try: db_path = "sqlite:///"+settings.SETTINGS_PATH+"\\data.db" db = dataset.connect(db_path) self.table = db[table_name].table except Exception: traceback.exec_print()