Python is slow when iterating over a cursor

I have the following code that runs very slowly (6.5 sec for iterating over 57.390 lines):

import mysql.connector

cnx=mysql.connector.connect(host=***, ***)
cursorSQL = cnx.cursor()

for dt in date_vect:
        cursorSQL.execute(query2, ((dt[0] + '-00:00:00'), (dt[0] + '-23:59:59'))) 
          #Takes about 0.25sec per iteration
        usr_list = list(cursorSQL.fetchall())  
          #takes about 6.20sec per iteration

As recommended here: https://stackoverflow.com/a/2776262/2326321 , I tried:

  • usr_list= cursorSQL.fetchall()
  • usr_list= list(cursorSQL.fetchall())

And as @postoronnim suggested, I also tried:

  • usr_list= cursorSQL.fetchmany(57390)

Without success.

However, there is some caching effect, since the same iteration takes only 0.5 seconds, when the iteration starts a second time, and then slows down to 6.5 seconds.

  • Any idea where this might come from?
  • Could you confirm that it has nothing to do with my database, since all import from MySQL is performed in a row cursor.execute, and fetchall()slowness is only due to list processing?
  • Could you explain why there is a caching effect?

Thank.

: Python 3.5 | 8- i7 | 16go Ram

+4

Source: https://habr.com/ru/post/1672386/


All Articles