Parallel mail requests using multiprocessing and requests in Python

Question

I have a small piece of code as shown below:

import requests import multiprocessing header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId': 'com.my_app', 'Accept-Language': 'en-ID', 'X-PushTokenType': 'GCM', 'X-DeviceToken': 'some_device_token' } BASE_URI = 'https://my_server.com/v2/customers/login' def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URI, headers=header, json=post_data) stuff_got.append(response.json()) return stuff_got tokens = [{"my_token":'EAAOZAe8Q2rKYBAu0XETMiCZC0EYAddz4Muk6Luh300PGwGAMh26Bpw3AA6srcxbPWSTATpTLmvhzkUHuercNlZC1vDfL9Kmw3pyoQfpyP2t7NzPAOMCbmCAH6ftXe4bDc4dXgjizqnudfM0D346rrEQot5H0esW3RHGf8ZBRVfTtX8yR0NppfU5LfzNPqlAem9M5ZC8lbFlzKpZAZBOxsaz'},{"my_token":'EAAOZAe8Q2rKYBAKQetLqFwoTM2maZBOMUZA2w5mLmYQi1GpKFGZAxZCaRjv09IfAxxK1amZBE3ab25KzL4Bo9xvubiTkRriGhuivinYBkZAwQpnMZC99CR2FOqbNMmZBvLjZBW7xv6BwSTu3sledpLSGQvPIZBKmTv3930dBH8lazZCs3q0Q5i9CZC8mf8kYeamV9DED1nsg5PQZDZD'}] pool = multiprocessing.Pool(processes=3) pool_outputs = pool.map(internet_resource_getter, tokens) pool.close() pool.join() 

All I'm trying to do is run parallel POST requests to the endpoint, while each POST will have a different token as it sends the body.

• Will I be able to achieve what I want with the above? I get output, but not sure if my requests are sent in parallel or not.
• I know about grequests. I wanted to achieve real parallel requests (as when using multiple processors on my system), and therefore I chose multiprocessing by grequests (which, as I understand it, uses gevents that are not parallel again, but are multi-threaded). Is my understanding right here?

Answer 1

If you are interested in running multiple POST requests in parallel, I suggest you use asyncio or aiohttp , which both implements the idea of ​​asynchronous tasks that run in parallel.

For example, you can do something like this with asyncio :

 import requests import asyncio header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId': 'com.my_app', 'Accept-Language': 'en-ID', 'X-PushTokenType': 'GCM', 'X-DeviceToken': 'some_device_token' } BASE_URI = 'https://my_server.com/v2/customers/login' def internet_resource_getter(post_data): stuff_got = [] response = requests.post(BASE_URI, headers=header, json=post_data) stuff_got.append(response.json()) print(stuff_got) return stuff_got tokens = [ { "my_token": 'EAAOZAe8Q2rKYBAu0XETMiCZC0EYAddz4Muk6Luh300PGwGAMh26B' 'pw3AA6srcxbPWSTATpTLmvhzkUHuercNlZC1vDfL9Kmw3pyoQfpyP' '2t7NzPAOMCbmCAH6ftXe4bDc4dXgjizqnudfM0D346rrEQot5H0es' 'W3RHGf8ZBRVfTtX8yR0NppfU5LfzNPqlAem9M5ZC8lbFlzKpZAZBO' 'xsaz' }, { "my_token": 'EAAOZAe8Q2rKYBAKQetLqFwoTM2maZBOMUZA2w5mLmYQi1GpKFGZAx' 'ZCaRjv09IfAxxK1amZBE3ab25KzL4Bo9xvubiTkRriGhuivinYBkZA' 'wQpnMZC99CR2FOqbNMmZBvLjZBW7xv6BwSTu3sledpLSGQvPIZBKmT' 'v3930dBH8lazZCs3q0Q5i9CZC8mf8kYeamV9DED1nsg5PQZDZD' } ] loop = asyncio.get_event_loop() for token in tokens: loop.run_in_executor(None, internet_resource_getter, token) 

Please note: they exist only in python 3.x But, in my opinion, it looks much better and concise, and this ensures that they work in parallel.

Answer 2

1) Yes, the above code will make requests for each token. One way to verify that requests are processed correctly is to check the return code:

 for response in pool_outputs: if response.status_code != 200: raise Exception("{} - {}".format(response.status_code, response.text)) 

2) Yes, your understanding sounds. I also use multiprocess + combo request instead of grequests.

Connected:

When running queries in parallel, you don’t need to focus on using multiple cores unless you are making millions of queries. This is due to the fact that the HTTP request is 99% of the Internet response time and 1% of the processor processing time. Your code will send multiple requests at once, which is really important. Alternatively, you might want to look into the dreaded GlobalInterpreterLock to see if this affects your multi-core application: What is Global Interpreter Lock (GIL)?