Python how to set flow restriction?

Question

I was wondering how I can limit something like this, use only 10 threads at a time

with open("data.txt") as f: for line in f: lines = line.rstrip("\n\r") t1 = Thread(target=Checker, args=("company")) t1.start() 

Answer 1

Use Python ThreadPoolExecutor with argument max_workers set to 10.

Something like this: `

 pool = ThreadPoolExecutor(max_workers=10) with open("data.txt") as f: for line in f: lines = line.rstrip("\n\r") pool.submit(Checker,"company") pool.shutdown(wait=True) 

pool will automatically allocate threads as needed, limiting the maximum number of allocation to 10. The first argument to pool.submit() is the name of the function, the arguments are simply passed as comma-separated values.

pool.shutdown(wait=True) waits for all threads to complete.

Answer 2

Use ThreadPoolExecutor and say you need 10 threads.

 def your_function_processing_one_line(line): pass # your computations with concurrent.futures.ThreadPoolExecutor(10) as executor: result = executor.map(your_function_processing_one_line, [line for line in f]) 

... and you will get all the results in result .

Answer 3

(for both Python 2.6+ and Python 3)

Use the threadPool module from multiprocessing :

 from multiprocessing.pool import ThreadPool 

The only thing is that it is poorly documented ...

Answer 4

I wrote this nested loop to limit the flow of a variable. This code is based on a given array of instructions for processing. I borrowed some elements from other answers to start the stream.

 import os, sys, datetime, logging, thread, threading, time from random import randint # set number of threads threadcount = 20 # alltests is an array of test data numbertests = len(alltests) testcounter = numbertests # run tests for test in alltests: # launch worker thread def worker(): """thread worker function""" os.system(command) return threads = [] t = threading.Thread(target=worker) threads.append(t) t.start() testcounter -= 1 # cap the threads if over limit while threading.active_count() >= threadcount: threads = threading.active_count() string = "Excessive threads, pausing 5 secs - " + str(threads) print (string) logging.info(string) time.sleep(5) # monitor for threads winding down while threading.active_count() != 1: threads = threading.active_count() string = "Active threads running - " + str(threads) print (string) logging.info(string) time.sleep(5)