When using multiprocessing to access MySQL, the following error always occurs, how to fix it?

Question

When I used Python multiprocessing to access the MySQL database, I always got errors:

OperationalError: (2006, "MySQL server is gone")

Lost connection to MySQL server during query

I would really appreciate it if someone could explain this to me.

Here is my code:

class MetricSource:
  def __init__(self, task):
    self.task = task

  def get_next_task_time(self):
    try:
      task_id = self.task.id
      next_task = Task.objects.get(id=task_id)
      next_time = next_task.last_success_time
    except Task.DoesNotExist as e:
      print 'Find Exception: %d' % self.task.id

def access_queue(input_queue, output_queue):
  while True:
    try:
      metric_source = input_queue.get(timeout=0.5)
      metric_source.get_next_task_time()
      output_queue.put(metric_source)
    except Queue.Empty:
      print "Queue Empty Error"
      continue

class Command(BaseCommand):

  def handle(self, *args, **options):
    self.manager = multiprocessing.Manager()
    self.input_queue = self.manager.Queue()
    self.output_queue = self.manager.Queue()

    self.init_data()

    for i in range(PROCESS_NUM):
      Process(target=access_queue, args=(self.input_queue, self.output_queue)).start()

  def init_data(self):
    for i in range(200):
      try:
        task = Task.objects.get(id=i+1)
        self.input_queue.put(MetricSource(task))
      except Exception as e:
        print 'find task_id %d' % i
        continue
      except IOError as e:
        print "find IOError: %r" % e
        continue

And I doubt that this is my problem with MySQL configuration, but I think it is not a problem. Here is my.cnf:

[mysqld]
default-character-set=utf8
collation_server = utf8_general_ci
character_set_server = utf8
max_allowed_packet = 100M

datadir=/home/work/data1/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
slow_query_log
slow_query_log_file=/home/work/data1/mysql/mysql-slow.log
max_allowed_packet=100M

log-error=/home/work/data1/mysql/error.log
general_log
general_log_file=/home/work/data1/mysql/mysql.log

tmp_table_size=2G
max_heap_table_size=2G

wait_timeout=2880000
interactive_timeout=2880000
innodb_data_home_dir=/home/work/data1/mysql/ibdata/

[mysqld_safe]
default-character-set=utf8
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Answer 1

I found the cause of the db connection failure, shared some points here.

"init_data" "fork subProcess":

self.init_data()

for i in range(PROCESS_NUM):
  Process(target=access_queue, args=(self.input_queue, self.output_queue)).start()

, init_data(), db , fork subProcess, . , , .

:

for i in range(PROCESS_NUM):
  Process(target=access_queue, args=(self.input_queue, self.output_queue)).start()

self.init_data()

sleep :

def access_queue(id, input_queue, output_queue):
  time.sleep(5)
  while True:
    ...

. , , db.

:

/ db orm?

- ?

: Django