Python’s multiprocessing performance problem
Because Python has limited parallelism when using threads, using worker processes is a common way to take advantage of multiple CPU cores. The
multiprocessing
module is built-in to the standard library, so it’s frequently used for this purpose.But while multiple processes let you take advantage of multiple CPUs, moving data between processes can be very slow. And that can reduce some of the performance benefits of using worker processes.
Let’s see:
- Why processes can have performance problems that threads don’t.
- A number of ways to work around or deal with this performance overhead.
- A bad solution you don’t want to us.
Source: Python’s multiprocessing performance problem, an article by Itamar Turner-Trauring.