Network latency can contribute to your unused CPU time in your python client application. If the network delay between the client and server is only 2 milliseconds and you execute 10,000 redis commands, your application should sit idle for at least 20 seconds, regardless of the speed of any other component.
Using multiple python threads may help, but each thread will continue to work when a blocking command is sent to the server. If you have a lot of threads, they will often synchronize, and the whole block is waiting for a response. Since each thread connects to all three servers, the chances of this are reduced, unless all are blocked while waiting for the same server.
Assuming you have a single random distributed server access to serve your requests (by hashing key names to implement scalding or splitting), the likelihood that three random requests will be hashed on the same redis server is inversely proportional to the number of servers. For 1 server, in 100% of cases you will hash on the same server, for 2 - in 50% of cases, for 3 - 33% of the time. What can happen is that 1/3 of the time all your threads are blocked waiting for the same server. Redis is single-threaded when processing data operations, so it must process each request one by one. Your observation that your processor reaches 60% utilization is consistent with the likelihood that your requests will be blocked during network latency on the same server.
Continuing to assume that you are overlaying on the client side by hashing key names, you can resolve the conflict between threads by assigning a single server connection to each thread and evaluating the hash hash before sending the request to the worker thread. This ensures that all threads wait at different network latencies. But using pipelining can be even a better improvement.
You can reduce the impact of network latency by using the redis-py module pipeline function if you do not need an immediate result from the server. This may be viable for you, since you save the data processing results in redis. To implement this with redis-py, periodically retrieve the pipeline descriptor for an existing redis connection object using the .pipeline() method and call several storage commands with this new descriptor in the same way as for the main redis.Redis connection object. Then run .execute() to block the responses. You can receive value improvement orders using pipelining to group dozens or hundreds of teams. Your client thread will not block until you issue the final .execute() method in the pipeline descriptor.
If you apply both changes, and each workflow communicates with only one server, pipelining several commands together (at least 5-10 to see a significant result), you can see more CPU usage in the client (closer to 100%). Cpython GIL still limits the client to one core, but it looks like you are already using other kernels to parse the XML using the multiprocessing module.
The redis.io website has a good record of pipelining .