Python Hanging Threads

Question

Python Hanging Threads

I have the following code:

    final = []
    with futures.ThreadPoolExecutor(max_workers=self.number_threads) as executor:
        _futures = [executor.submit(self.get_attribute, listing,
                                    self.proxies[listings.index(listing) % len(self.proxies)]) for listing
                    in listings]
        for result in futures.as_completed(_futures):
            try:
                listing = result.result()
                final.append(listing)
            except Exception as e:
                print traceback.format_exc()
    return final

The self.get_attribute function, which is presented to the executor, enters the dictionary and proxies as input and makes one or two HTTP requests to receive some data and return using an editable dictionary. The problem is that work / threads hang at the end of all tasks. If I send 400 dictionaries, it will complete ~ 380 tasks, and then they will meet. If I send 600, it will complete ~ 570-580. However, if I send 25, he will fill them all. I’m not sure what is the threshold on which he will go until the end to the end.

I also tried using the queue and thread system:

  def _get_attribute_thread(self):
     while self.q.not_empty:
         job = self.q.get()
         listing = job['listing']
         proxy = job['proxy']
         self.threaded_results.put(self.get_attribute(listing, proxy))
         self.q.task_done()

  def _get_attributes_threaded_with_proxies(self, listings):

       for listing in listings:
           self.q.put({'listing': listing, 'proxy': self.proxies[listings.index(listing) % len(self.proxies)]})

       for _ in xrange(self.number_threads):
           thread = threading.Thread(target=self._get_attribute_thread)
           thread.daemon = True
           thread.start()

       self.q.join()

       final = []
       while self.threaded_results.not_empty:
           final.append(self.threaded_results.get())

       return final