With the Python multiprocessing library, we can write truly parallel software. When we were using Python threads, we weren’t utilizing multiple CPUs or CPU cores. However, with this library, we will.
Table of contents
Multiprocessing based example
import time import multiprocessing # A CPU heavy calculation, just # as an example. This can be # anything you like def heavy(n, myid): for x in range(1, n): for y in range(1, n): x**y print(myid, "is done") def multiproc(n): processes =  for i in range(n): p = multiprocessing.Process(target=heavy, args=(500,i,)) processes.append(p) p.start() for p in processes: p.join() if __name__ == "__main__": start = time.time() multiproc(80) end = time.time() print("Took: ", end - start)
This code takes about 23 seconds to run, which is half of the threaded version!
As you can see, this looks almost the same as the threaded version, code-wise. The threading and multiprocessing libraries are intentionally made very equivalent. But the 80 invocations of heavy finish roughly twice as fast this time.
My test system (a small desktop computer) has only two CPU cores, so that explains why it’s a factor two. If I run this code on my brand new laptop, with 4 faster CPU cores, it’s more than four times faster. This perfectly demonstrates the linear speed increase multiprocessing offers us in the case of CPU-bound code.
Python multiprocessing pool
We can make the multiprocessing version a little more elegant and slightly faster by using
multiprocessing.Pool(p). This Python multiprocessing helper creates a pool of size
p processes. If you don’t supply a value for
p, it will default to the number of CPU cores in your system, which is actually a sensible choice most of the time.
By using the
Pool.map() method, we can submit work to the pool. This work comes in the form of a simple function call:
import time import multiprocessing # A CPU heavy calculation, just # as an example. This can be # anything you like def heavy(n, myid): for x in range(1, n): for y in range(1, n): x**y print(myid, "is done") def doit(n): heavy(500, n) def pooled(n): # By default, our pool will have # numproc slots with multiprocessing.Pool() as pool: pool.map(doit, range(n)) if __name__ == "__main__": start = time.time() pooled(80) end = time.time() print("Took: ", end - start)
This version’s runtime is roughly the same as the non-pooled version, but it has to create fewer processes, so it is more efficient. After all, instead of creating 80 processes, we created four and reused those each time.
If you want to learn more about Python multiprocessing and working with multiprocessing pools, you can head over to the official documentation. If you work on the command line a lot, you might be interested in my article on Bash multiprocessing as well!