Python performance - best parallelism approach

Question

I am implementing a Python script that should continue to send 1500+ packets in parallel in less than 5 seconds.

In short, what I need:

def send_pkts(ip): #craft packet while True: #send packet time.sleep(randint(0,3)) for x in list[:1500]: send_pkts(x) time.sleep(randint(1,5)) 

I tried simple single-threaded, multi-threaded, multi-processor and multi-processor + multi-threaded forms and had the following problems:

• Simple single-threaded: “Delay” seems to compromise the “5 seconds” dependency.
• Multithreading: I think I was unable to accomplish what I want, due to Python GIL limitations.
• Multiprocessing: This was the best approach that seemed to work. However, due to the excessive number of processes, the VM where I run the script freezes (of course, 1,500 processes). Thus, it becomes impractical.
• Multiprocessing + Multithreading: In this approach, I created fewer processes, each of which caused some threads (let's suppose: 10 processes that call 150 threads each). It was clear that the VM did not freeze as fast as approach number 3, but the most “simultaneous packet sending" I could achieve was ~ 800. GIL limitations? VM limitations? In this attempt, I also tried using the process pool, but the results are where they are similar.

Is there a better approach I could use to complete this task?

[1] EDIT 1:

  def send_pkt(x): #craft pkt while True: #send pkt gevent.sleep(0) gevent.joinall([gevent.spawn(send_pkt, x) for x in list[:1500]]) 

[2] EDIT 2 (gevent monkey-patching):

 from gevent import monkey; monkey.patch_all() jobs = [gevent.spawn(send_pkt, x) for x in list[:1500]] gevent.wait(jobs) #for send_pkt(x) check [1] 

However, I got the following error: "ValueError: filedescriptor is out of range in select ()" . So I checked my ulimit system (Soft and Hard both maximally: 65536). After that, I checked that this has something to do with select () restrictions for Linux (maximum 1024 fds). Please check: http://man7.org/linux/man-pages/man2/select.2.html (BUGS section) - To overcome this, I have to use poll () ( http://man7.org/linux /man-pages/man2/poll.2.html ). But with poll (), I come back to the same limitations: since polling is a “blocking approach”.

Hello,